`extraFiles` makes folder unwritable?

JupyterHub Zero to JupyterHub on Kubernetes
1

I’m trying to map in a jupyter_server_config.py file to restrict the available kernels in our custom image. My config looks like:

singleuser:
  extraFiles:
    custom_config:
      mountPath: /home/user/.jupyter/jupyter_server_config.py
      mode: 0777
      stringData: |
        c.KernelSpecManager.ensure_native_kernel = False
        c.KernelSpecManager.whitelist = {'xpython'}

…but unfortunately this doesn’t work.

I’m using a recent version of the chart:

> helm install jupyterhub `
>>     jupyterhub/jupyterhub `
>>     --namespace jhub `
>>     --version 0.11.1-n361.h166e2157 `
>>     --values jupyterhub-helm.yaml

The container logs show a permission denied error trying to access the ~/.jupyter/lab folder:

[W 2021-04-03 20:46:47.962 SingleUserNotebookApp web:1787] 500 PUT /user/dhirschfeld/lab/api/workspaces/default?1617446807946 (10.200.72.34): [Errno 13] Permission denied: '/home/user/.jupyter/lab'

In the JupyterLab container itself I can see the config is mapped in:

user@jupyter-dhirschfeld:~/.jupyter$ cat jupyter_server_config.py 
c.KernelSpecManager.ensure_native_kernel = False
c.KernelSpecManager.whitelist = {'xpython'}

…but the lab folder is missing and the folder seems to be read-only:

user@jupyter-dhirschfeld:~/.jupyter$ ls
jupyter_server_config.py
user@jupyter-dhirschfeld:~/.jupyter$ touch jupyter_server_config.py
touch: cannot touch 'jupyter_server_config.py': Read-only file system
user@jupyter-dhirschfeld:~/.jupyter$ touch a.txt
touch: cannot touch 'a.txt': Permission denied
user@jupyter-dhirschfeld:~/.jupyter$ mkdir lab
mkdir: cannot create directory ‘lab’: Permission denied

Is this expected? It would seem to make it difficult to use to map in configuration.

2

Saving the config to /usr/local/etc/jupyter/jupyter_server_config.py did the job, but only in combination with changing the cmd:

singleuser:
  cmd: jupyter-labhub
3

Does the ~/.jupyter directory already exist in the container image with the correct permissions before you mount the extra file?

4

Good point. I don’t think it existed previously.

I wonder if I can use an init container to create it or if it’s already too late by that point :thinking:

5

@dhirschfeld I’d like to understand this better, I think no matter what there is an action point for the Helm chart to either document this better or similar.

Background knowledge

  1. extraFiles work by mounting k8s Secret resources.
    1. An entry is added to the pod’s volumes
    2. An entry is added to the container’s volumeMounts
    3. The volumeMounts entry is not having readOnly explicitly set, which make it default to true for a Secret I think.
    4. The volumeMounts entry always have subPath specified, which makes the file only update during startup of the pod.
    5. If it is functional in k8s at all to do this, we can absolutely support setting readOnly to false if it helps you get write permissions even though the changes won’t persist for that user specifically or any user until the next restart.

Questions

  • Can you describe if your volumeMount has readOnly explicitly set by inspection using kubectl get pod <podname> -o yaml?
  • Can you describe output of ls -alh <path to a singleuser.extraFiles file that has been mounted>?
  • Can you verify that it is the file itself that is readOnly, but not an entire folder or similar.
6

I’m no longer trying to map the file into the users home folder - I’m now mapping it into:

/usr/local/etc/jupyterhub/jupyterhub_config.d/jupyterhub_config.py

…which is better since it allows the sysadmin to provide defaults but the user to override them in their own /home/user/.jupyter/jupyter_server_config.py

Can you verify that it is the file itself that is readOnly, but not an entire folder or similar.

I wouldn’t have minded if it was just the mapped-in file itself which was readonly but it definitely appeared that mapping in the file made the whole folder readonly.

Sorry, I’m completely smashed at the moment and won’t realistically have time to investigate the original problem since I’ve now side-stepped it.