Jupyterhub on K8s: How to run a single command as root at container startup?

We have singleuser configuration that mounts ca.crt CA file for accessing Elasticsearch:

singleuser:
  storage:
    extraVolumes:
      - name: elasticsearch-certs
        secret:
          secretName: elasticsearch-master-crt
    extraVolumeMounts:
      - name: elasticsearch-certs
        mountPath: "/usr/local/share/ca-certificates/"
        readOnly: true

This places cert to /usr/local/share/ca-certificates/ca.crt for Elasticsearch.

We now want to run a single sudo update-ca-certificates at pod/container startup.

What is the recommended and cleanest way to do that with singleuser configuration? An additional initContainer, or is there a simpler way? Looking through docker-stacks/start.sh at 5573d9d0374dedaf92de0be50dd90ca70ea2c5e6 · jupyter/docker-stacks · GitHub, I see a reference to starting as root and dropping down to NB_USER; what would that look like in terms of singleuser configuration?

I’m not sure what update-ca-certificates will do to the file system and such, but if it for example produces a simple file to be positioned somewhere, I’d go with an init container that shares access to an empty-directory kind of k8s volume mounted on both contianers container where that file is emitted by the initContainer.

If it isn’t that easy, then you can indeed start the user server container, if it is based on a jupyter/docker-stacks image, via the start.sh script, that will consider various environment variables you set and act based on those.

What do you need to set? Well, you need to start the container as root user, and then you have to use the flags to switch to the use you want again. singleuser.extraEnv is relevant here, and singleuser.uid I think. See the configuration reference at z2jh.jupyter.org to verify what you need.

The downside of starting as root and such is that you can’t lock down the permissions as tight for the user that ends up running if the k8s pod is required to be allowed to first run as root etc.

2 Likes

Gotcha - FYI, update-ca-certificates will look for any certificates with a .crt extension found below /usr/local/share/ca-certificates, and re-generates /etc/ssl/certs/ca-certificates.crt, a single-file version of CA certificates. It also places a symlink to the new cert in /etc/ssl/certs. (source)

My issue with an initContainer is that /etc/ssl/certs/ is not empty to start with. Putting a volumeMount there will clear it out.

So perhaps before-notebook.d is my best option here, will give that a shot.

It doesn’t always have to do that, but perhaps this is only a functional feature of certain volumes, but, you can use subPath in the volume to specify a single file, then mountPath to point to a single file.

When using subPath on configmaps and secret volumes, you won’t receive updates as you otherwise would though.

Hello @consideRatio , I’m returning back to this one and not sure if I’m fully following your suggestion,

you can use subPath in the volume to specify a single file, then mountPath to point to a single file

I have:

singleuser:
  # ...
  initContainers:
    - name: update-ca-certificates
      image: jupyter/base-notebook
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        privileged: true
      command:
        - "update-ca-certificates"
        - "--verbose"
  # ...
  storage:
    type: dynamic
    extraLabels: {}
    extraVolumes:
      - name: elasticsearch-certs
        secret:
          secretName: hulk-es-http-certs-public
          items:
            - key: tls.crt
              path: elasticsearch.crt
              mode: 0644
      - name: ssl-certs
        emptyDir: {}
    extraVolumeMounts:
      - name: elasticsearch-certs
        mountPath: "/usr/local/share/ca-certificates/elasticsearch/"
        readOnly: true
      - name: ssl-certs
        mountPath: /etc/ssl/certs/ca-certificates.crt
        subPath: ca-certificates.crt

When launching a new singleuser pod, the initContainer completes successfully, but then the main container errors out with:

2021-12-16T14:31:38Z [Warning] Error: failed to start container “notebook”: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting “/var/lib/kubelet/pods/abb36795-51f5-4a1d-a8ef-8f9bb9c7309d/volume-subpaths/ssl-certs/notebook/2” to rootfs at “/etc/ssl/certs/ca-certificates.crt” caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Am I misinterpreting your suggestion for mountPath/subPath here?

Note that /etc/ssl/certs/ca-certificates.crt exists to start with on the filesystem, and is then updated by the update-ca-certificates call.

It doesn’t always have to do that, but perhaps this is only a functional feature of certain volumes, but, you can use subPath in the volume to specify a single file, then mountPath to point to a single file.

I think you run into an issue related to trying to mount a specific file from an emptyDir kind of volume. I’m not sure if emptyDir volumes support subPath like I know configmap/secret does. Would it error in this way if it didn’t? Maybe…

One idea that would work I think, would be to make the image have a symlink pointing somewhere, which you then populate with an emptydir volume that is populated within an initContainer. Another idea is to mount the entire folder and let the volumeMount it override things in the main container, where you populate the folder you mount fully from the init container.

Thank you for the reply but you lost me on that one unfortunately :frowning:

One idea that would work I think, would be to make the image have a symlink pointing somewhere, which you then populate with an emptydir volume that is populated within an initContainer.

Okay, I get that this is an ln -s call in the Dockerfile, but from what src to tgt?

Just anywhere, for example /mnt/certs/some-file-name - whats important that is a location you then also mount the emptydir volume you have populated in an initcontainer to hold the new cert.