The `c.JupyterHub.cleanup_servers = False` parameter is invalid and the corresponding pod is deleted on the jupyterhub+k8s+jupyterlab


My deployment environment is Jupiterhub (1.4.2) + k8s (v1.21.2) + Jupiterhub/singleuser: latest,It works normally.
c.JupyterHub.cleanup_servers = False is configured in the configuration file.
When I click Stop My Server, the corresponding pod of k8s is deleted. The created files are also deleted.

How to solve this problem? I want to keep the pod because it has a lot of data.

Here is my brief configuration:

c.JupyterHub.cleanup_proxy = False
c.JupyterHub.cleanup_servers = False
c.KubeSpawner.start_timeout = 60 * 5
c.KubeSpawner.image = 'jupyterhub/singleuser'
c.KubeSpawner.service_account = 'default'
c.KubeSpawner.storage_pvc_ensure = False
c.KubeSpawner.cmd = ["jupyter-labhub"]

cleanup_servers only refers to whether the singleuser servers are stopped when the hub is stopped: Application configuration — JupyterHub 1.4.2 documentation

Keeping the singleuser pods after they’re stopped is not possible. Kubernetes pods are ephemeral, you can read more about them here:

If you need to maintain some persistent data you can use persistent volumes.

@manics hello!Thank you for your explanation!

I will mount the data volume according to your opinion.

There was a problem mounting the data volume.

Problem description:
After the data volume is mounted successfully, there is a problem in the page display. Check the pod log and find that there are permission problems and the file cannot be found.
I mount /home/jovyan/work/(jovyan permission) in the pod to /opt/jupyterhub/(root permission) in the Linux virtual machine. I modified the /opt/jupyterhub/ permission of Linux virtual machine to jovyan permission, but it had no effect.

Pod log information:

[W 2021-08-17 08:55:28.011 SingleUserLabApp configurable:168] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2021-08-17 08:55:29.103 SingleUserLabApp extension:168] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2021-08-17 08:55:29.104 SingleUserLabApp extension:169] JupyterLab application directory is /opt/conda/share/jupyter/lab
[W 2021-08-17 08:55:29.105 SingleUserLabApp labapp:375] JupyterLab server extension not enabled, manually loading...
[I 2021-08-17 08:55:29.108 SingleUserLabApp extension:168] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2021-08-17 08:55:29.109 SingleUserLabApp extension:169] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2021-08-17 08:55:29.109 SingleUserLabApp singleuser:561] Starting jupyterhub-singleuser server version 1.0.0
[W 2021-08-17 08:55:29.115 SingleUserLabApp _version:56] jupyterhub version 1.4.2 != jupyterhub-singleuser version 1.0.0. This could cause failure to authenticate and result in redirect loops!
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] Serving notebooks from local directory: /home/jovyan/work
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] The Jupyter Notebook is running at:
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] http://(jupyter-limy or
[I 2021-08-17 08:55:29.116 SingleUserLabApp notebookapp:1775] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2021-08-17 08:55:29.119 SingleUserLabApp notebookapp:1796]

    To access the notebook, open this file in a browser:
    Or copy and paste one of these URLs:
        http://(jupyter-limy or
[I 2021-08-17 08:55:29.120 SingleUserLabApp singleuser:542] Updating Hub with activity every 300 seconds
[I 2021-08-17 08:55:29.137 SingleUserLabApp log:174] 302 GET /user/limy/ -> /user/limy/lab? (@ 0.71ms
[I 2021-08-17 08:55:29.204 SingleUserLabApp log:174] 302 GET /user/limy/ -> /user/limy/lab? (@ 1.06ms
[I 2021-08-17 08:55:29.222 SingleUserLabApp log:174] 302 GET /user/limy/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-limy&redirect_uri=%2Fuser%2Flimy%2Foauth_callback&response_type=code&state=[secret] (@ 2.65ms
[I 2021-08-17 08:55:29.315 SingleUserLabApp auth:981] Logged-in user {'kind': 'user', 'name': 'limy', 'admin': False, 'groups': [], 'server': '/user/limy/', 'pending': None, 'created': '2021-08-17T08:55:12.785259Z', 'last_activity': '2021-08-17T08:55:29.315874Z', 'servers': None}
[I 2021-08-17 08:55:29.317 SingleUserLabApp log:174] 302 GET /user/limy/oauth_callback?code=[secret]&state=[secret] -> /user/limy/lab? (@ 47.65ms
[I 2021-08-17 08:55:29.340 SingleUserLabApp log:174] 200 GET /user/limy/lab? (limy@ 7.80ms
[I 2021-08-17 08:55:31.107 SingleUserLabApp log:174] 200 GET /user/limy/api/kernelspecs?1629190539996 (limy@ 16.59ms
[I 2021-08-17 08:55:31.110 SingleUserLabApp log:174] 200 GET /user/limy/api/terminals?1629190539999 (limy@ 1.21ms
[I 2021-08-17 08:55:31.297 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/application-extension:sidebar?1629190540200 (limy@ 3.70ms
[I 2021-08-17 08:55:31.323 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/shortcuts-extension:plugin?1629190540203 (limy@ 23.99ms
[I 2021-08-17 08:55:31.328 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1629190540211 (limy@ 4.05ms
[I 2021-08-17 08:55:31.330 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/apputils-extension:themes?1629190540215 (limy@ 4.81ms
[I 2021-08-17 08:55:31.409 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190540313 (limy@ 2.12ms
[I 2021-08-17 08:55:31.766 SingleUserLabApp log:174] 200 GET /user/limy/api/contents/?content=1&1629190540664 (limy@ 4.37ms
[I 2021-08-17 08:55:31.788 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/extensionmanager-extension:plugin?1629190540689 (limy@ 2.64ms
[I 2021-08-17 08:55:31.820 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/notebook-extension:tracker?1629190540723 (limy@ 3.72ms
[I 2021-08-17 08:55:31.966 SingleUserLabApp log:174] 200 GET /user/limy/api/nbconvert?1629190540727 (limy@ 144.55ms
[I 2021-08-17 08:55:31.970 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1629190540737 (limy@ 2.86ms
[I 2021-08-17 08:55:32.110 SingleUserLabApp log:174] 200 GET /user/limy/api/nbconvert?1629190540875 (limy@ 136.75ms
[I 2021-08-17 08:55:32.114 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/codemirror-extension:commands?1629190540886 (limy@ 3.53ms
[I 2021-08-17 08:55:32.115 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190540904 (limy@ 3.24ms
[I 2021-08-17 08:55:32.338 SingleUserLabApp commands:1484] Node v11.11.0

[I 2021-08-17 08:55:32.342 SingleUserLabApp build_handler:43] Build is up to date
[I 2021-08-17 08:55:32.343 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/build?1629190540656 (limy@ 588.37ms
[W 2021-08-17 08:55:32.358 SingleUserLabApp web:1782] 404 GET /user/limy/lab/api/workspaces/user/limy/lab?1629190541260 ( Workspace 'user/limy/lab' ('userlimylab-2cdd') not found
[W 2021-08-17 08:55:32.358 SingleUserLabApp handlers:620] Workspace 'user/limy/lab' ('userlimylab-2cdd') not found
[W 2021-08-17 08:55:32.359 SingleUserLabApp log:174] 404 GET /user/limy/lab/api/workspaces/user/limy/lab?1629190541260 (limy@ 1.94ms
[I 2021-08-17 08:55:33.555 SingleUserLabApp log:174] 204 PUT /user/limy/lab/api/workspaces/user/limy/lab?1629190542459 (limy@ 1.99ms
[I 2021-08-17 08:55:41.106 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190550009 (limy@ 1.66ms
[I 2021-08-17 08:55:41.134 SingleUserLabApp log:174] 200 GET /user/limy/api/terminals?1629190550012 (limy@ 1.25ms
[I 2021-08-17 08:55:41.775 SingleUserLabApp log:174] 200 GET /user/limy/api/contents/?content=1&1629190550679 (limy@ 2.70ms
# click create python3
[I 2021-08-17 08:58:03.530 SingleUserLabApp handlers:153] Creating new notebook in /
[I 2021-08-17 08:58:03.575 SingleUserLabApp sign:396] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[W 2021-08-17 08:58:03.577 SingleUserLabApp web:1782] 403 POST /user/limy/api/contents/?1629190692424 ( Permission denied: Untitled.ipynb
[W 2021-08-17 08:58:03.578 SingleUserLabApp handlers:620] Permission denied: Untitled.ipynb
[W 2021-08-17 08:58:03.578 SingleUserLabApp log:174] 403 POST /user/limy/api/contents/?1629190692424 (limy@ 49.51ms

Problem screenshot:

My deployment environment information:

  • jupyterhub environment information

    jupyterhub                         1.4.2
    jupyterhub-kubespawner             1.1.0
    jupyterhub-dummyauthenticator      0.3.1
  • Jupyterhub brief configuration information

    c.KubeSpawner.image = 'jupyterhub/singleuser:1.0.0'
    c.KubeSpawner.service_account = 'default'
    c.KubeSpawner.cmd = ["jupyter-labhub","--notebook-dir=/home/jovyan/work/"]
    c.KubeSpawner.image_pull_policy = 'IfNotPresent'
    c.KubeSpawner.namespace = 'jupyterhub-k8s'
    c.KubeSpawner.volumes = [
         'name': 'test-container-{username}--{servername}',
         'persistentVolumeClaim': {
             'claimName': 'test-pvc-jupyter'
    c.KubeSpawner.volume_mounts = [
          'name': 'test-container-{username}--{servername}',
          'mountPath': '/home/jovyan/work/',
          'subPath': 'sub-{username}--{servername}'
    c.KubeSpawner.storage_access_modes = ['ReadWriteMany']
    c.KubeSpawner.storage_capacity = '5G'
    c.KubeSpawner.storage_class = 'normal'
    c.KubeSpawner.storage_extra_labels = {'pvc-label':'pvc-label-value'}
    c.KubeSpawner.storage_pvc_ensure = True
    c.KubeSpawner.delete_pvc = False
    c.KubeSpawner.pvc_name_template = 'test-pvc-jupyter'
  • k8s PersistentVolume information (yaml)

    apiVersion: v1
    kind: PersistentVolume
      name: test-pv-jupyter-nfs  
      namespace: jupyterhub-k8s
        name: test-pv-jupyter-nfs
        storetype: nfs
      storageClassName: normal
        - ReadWriteMany
        storage: 5G
    #  persistentVolumeReclaimPolicy: Retain 
        path: /opt/jupyterhub
  • Linux virtual machine /opt/jupyterhub/ directory information

    drwxr-xr-x. 3 jovyan jovyan  22 Aug 17 16:55 jupyterhub

Can you show us the output of

kubectl get pv,pvc

in your JupyterHub namespace?

pv,pvc status

# kubectl get pvc,pv -n jupyterhub-k8s
NAME                                     STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/test-pvc-jupyter   Bound    test-pv-jupyter-nfs   5G         RWX            normal         19h

NAME                                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                             STORAGECLASS   REASON   AGE
persistentvolume/test-pv-jupyter-nfs   5G         RWX            Retain           Bound    jupyterhub-k8s/test-pvc-jupyter   normal                  19h

It sounds like the permissions on your volume are still incorrect.

Can you run ls -lR /opt/jupyterhub on the VM and and ls -lR /home/jovyan in the pod?

ls -lR /opt/jupyterhub

total 0
drwxr-xr-x. 2 root root 6 Aug 19 09:35 sub-limy

total 0

kubectl exec jupyter-limy -n jupyterhub-k8s -it – /bin/bash

(base) jovyan@jupyter-limy:~$ ls -lR /home/jovyan
total 0
drwxr-xr-x. 2 root root 6 Aug 19 01:35 work

total 0

As you can see your directories under /opt/jupyterhub are still owned by root. One way this can occur is if the parent directory was initially empty, so when the volume is mounted the directory defaults to being root owned.

If this is the case running chmod -R ... to fix the permissions on everything under /opt/jupyterhub should fix things for now. However if you later add more users you may need to reset the permissions on their folder, unless you create the folders in advance with the correct permissions.

Thank you for your explanation

I use NFS to map all user identities to those that do not exist in the container. The reason for this problem is that the file inside the container does not belong to the master file, and other users can access it.

/opt/jupyterhub *(rw,sync,all_squash,anonuid=1002,anongid=1002)

If you want you could try running your container as the UID of the logged in user by overriding pre_spawn_start
See for example

Note the example is a bit out of date, but the general idea still holds.

Ok. Thanks.

I’ll try it later