The `c.JupyterHub.cleanup_servers = False` parameter is invalid and the corresponding pod is deleted on the jupyterhub+k8s+jupyterlab

meng-yu-github · August 16, 2021, 7:38am

Hi!

My deployment environment is Jupiterhub (1.4.2) + k8s (v1.21.2) + Jupiterhub/singleuser: latest,It works normally.
c.JupyterHub.cleanup_servers = False is configured in the configuration file.
When I click Stop My Server, the corresponding pod of k8s is deleted. The created files are also deleted.

How to solve this problem? I want to keep the pod because it has a lot of data.

Here is my brief configuration：

c.JupyterHub.cleanup_proxy = False
c.JupyterHub.cleanup_servers = False
c.KubeSpawner.start_timeout = 60 * 5
c.KubeSpawner.image = 'jupyterhub/singleuser'
c.KubeSpawner.service_account = 'default'
c.KubeSpawner.storage_pvc_ensure = False
c.KubeSpawner.cmd = ["jupyter-labhub"]

manics · August 16, 2021, 8:33pm

cleanup_servers only refers to whether the singleuser servers are stopped when the hub is stopped: Application configuration — JupyterHub 1.4.2 documentation

Keeping the singleuser pods after they’re stopped is not possible. Kubernetes pods are ephemeral, you can read more about them here:

If you need to maintain some persistent data you can use persistent volumes.

meng-yu-github · August 17, 2021, 9:28am

@manics hello！Thank you for your explanation！

I will mount the data volume according to your opinion.

There was a problem mounting the data volume.

Problem description:
After the data volume is mounted successfully, there is a problem in the page display. Check the pod log and find that there are permission problems and the file cannot be found.
I mount /home/jovyan/work/(jovyan permission) in the pod to /opt/jupyterhub/(root permission) in the Linux virtual machine. I modified the /opt/jupyterhub/ permission of Linux virtual machine to jovyan permission, but it had no effect.

Pod log information：

[W 2021-08-17 08:55:28.011 SingleUserLabApp configurable:168] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2021-08-17 08:55:29.103 SingleUserLabApp extension:168] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2021-08-17 08:55:29.104 SingleUserLabApp extension:169] JupyterLab application directory is /opt/conda/share/jupyter/lab
[W 2021-08-17 08:55:29.105 SingleUserLabApp labapp:375] JupyterLab server extension not enabled, manually loading...
[I 2021-08-17 08:55:29.108 SingleUserLabApp extension:168] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2021-08-17 08:55:29.109 SingleUserLabApp extension:169] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2021-08-17 08:55:29.109 SingleUserLabApp singleuser:561] Starting jupyterhub-singleuser server version 1.0.0
[W 2021-08-17 08:55:29.115 SingleUserLabApp _version:56] jupyterhub version 1.4.2 != jupyterhub-singleuser version 1.0.0. This could cause failure to authenticate and result in redirect loops!
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] Serving notebooks from local directory: /home/jovyan/work
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] The Jupyter Notebook is running at:
[I 2021-08-17 08:55:29.115 SingleUserLabApp notebookapp:1774] http://(jupyter-limy or 127.0.0.1):8888/user/limy/?token=4c7e4c6f730b495a85de1b49628df3cc
[I 2021-08-17 08:55:29.116 SingleUserLabApp notebookapp:1775] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2021-08-17 08:55:29.119 SingleUserLabApp notebookapp:1796]

    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://(jupyter-limy or 127.0.0.1):8888/user/limy/?token=4c7e4c6f730b495a85de1b49628df3cc
[I 2021-08-17 08:55:29.120 SingleUserLabApp singleuser:542] Updating Hub with activity every 300 seconds
[I 2021-08-17 08:55:29.137 SingleUserLabApp log:174] 302 GET /user/limy/ -> /user/limy/lab? (@192.168.235.192) 0.71ms
[I 2021-08-17 08:55:29.204 SingleUserLabApp log:174] 302 GET /user/limy/ -> /user/limy/lab? (@191.168.2.218) 1.06ms
[I 2021-08-17 08:55:29.222 SingleUserLabApp log:174] 302 GET /user/limy/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-limy&redirect_uri=%2Fuser%2Flimy%2Foauth_callback&response_type=code&state=[secret] (@191.168.2.218) 2.65ms
[I 2021-08-17 08:55:29.315 SingleUserLabApp auth:981] Logged-in user {'kind': 'user', 'name': 'limy', 'admin': False, 'groups': [], 'server': '/user/limy/', 'pending': None, 'created': '2021-08-17T08:55:12.785259Z', 'last_activity': '2021-08-17T08:55:29.315874Z', 'servers': None}
[I 2021-08-17 08:55:29.317 SingleUserLabApp log:174] 302 GET /user/limy/oauth_callback?code=[secret]&state=[secret] -> /user/limy/lab? (@191.168.2.218) 47.65ms
[I 2021-08-17 08:55:29.340 SingleUserLabApp log:174] 200 GET /user/limy/lab? (limy@191.168.2.218) 7.80ms
[I 2021-08-17 08:55:31.107 SingleUserLabApp log:174] 200 GET /user/limy/api/kernelspecs?1629190539996 (limy@191.168.2.218) 16.59ms
[I 2021-08-17 08:55:31.110 SingleUserLabApp log:174] 200 GET /user/limy/api/terminals?1629190539999 (limy@191.168.2.218) 1.21ms
[I 2021-08-17 08:55:31.297 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/application-extension:sidebar?1629190540200 (limy@191.168.2.218) 3.70ms
[I 2021-08-17 08:55:31.323 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/shortcuts-extension:plugin?1629190540203 (limy@191.168.2.218) 23.99ms
[I 2021-08-17 08:55:31.328 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1629190540211 (limy@191.168.2.218) 4.05ms
[I 2021-08-17 08:55:31.330 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/apputils-extension:themes?1629190540215 (limy@191.168.2.218) 4.81ms
[I 2021-08-17 08:55:31.409 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190540313 (limy@191.168.2.218) 2.12ms
[I 2021-08-17 08:55:31.766 SingleUserLabApp log:174] 200 GET /user/limy/api/contents/?content=1&1629190540664 (limy@191.168.2.218) 4.37ms
[I 2021-08-17 08:55:31.788 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/extensionmanager-extension:plugin?1629190540689 (limy@191.168.2.218) 2.64ms
[I 2021-08-17 08:55:31.820 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/notebook-extension:tracker?1629190540723 (limy@191.168.2.218) 3.72ms
[I 2021-08-17 08:55:31.966 SingleUserLabApp log:174] 200 GET /user/limy/api/nbconvert?1629190540727 (limy@191.168.2.218) 144.55ms
[I 2021-08-17 08:55:31.970 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1629190540737 (limy@191.168.2.218) 2.86ms
[I 2021-08-17 08:55:32.110 SingleUserLabApp log:174] 200 GET /user/limy/api/nbconvert?1629190540875 (limy@191.168.2.218) 136.75ms
[I 2021-08-17 08:55:32.114 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/settings/@jupyterlab/codemirror-extension:commands?1629190540886 (limy@191.168.2.218) 3.53ms
[I 2021-08-17 08:55:32.115 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190540904 (limy@191.168.2.218) 3.24ms
[I 2021-08-17 08:55:32.338 SingleUserLabApp commands:1484] Node v11.11.0

[I 2021-08-17 08:55:32.342 SingleUserLabApp build_handler:43] Build is up to date
[I 2021-08-17 08:55:32.343 SingleUserLabApp log:174] 200 GET /user/limy/lab/api/build?1629190540656 (limy@191.168.2.218) 588.37ms
[W 2021-08-17 08:55:32.358 SingleUserLabApp web:1782] 404 GET /user/limy/lab/api/workspaces/user/limy/lab?1629190541260 (191.168.2.218): Workspace 'user/limy/lab' ('userlimylab-2cdd') not found
[W 2021-08-17 08:55:32.358 SingleUserLabApp handlers:620] Workspace 'user/limy/lab' ('userlimylab-2cdd') not found
[W 2021-08-17 08:55:32.359 SingleUserLabApp log:174] 404 GET /user/limy/lab/api/workspaces/user/limy/lab?1629190541260 (limy@191.168.2.218) 1.94ms
[I 2021-08-17 08:55:33.555 SingleUserLabApp log:174] 204 PUT /user/limy/lab/api/workspaces/user/limy/lab?1629190542459 (limy@191.168.2.218) 1.99ms
[I 2021-08-17 08:55:41.106 SingleUserLabApp log:174] 200 GET /user/limy/api/sessions?1629190550009 (limy@191.168.2.218) 1.66ms
[I 2021-08-17 08:55:41.134 SingleUserLabApp log:174] 200 GET /user/limy/api/terminals?1629190550012 (limy@191.168.2.218) 1.25ms
[I 2021-08-17 08:55:41.775 SingleUserLabApp log:174] 200 GET /user/limy/api/contents/?content=1&1629190550679 (limy@191.168.2.218) 2.70ms
......
# click create python3
[I 2021-08-17 08:58:03.530 SingleUserLabApp handlers:153] Creating new notebook in /
[I 2021-08-17 08:58:03.575 SingleUserLabApp sign:396] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[W 2021-08-17 08:58:03.577 SingleUserLabApp web:1782] 403 POST /user/limy/api/contents/?1629190692424 (191.168.2.218): Permission denied: Untitled.ipynb
[W 2021-08-17 08:58:03.578 SingleUserLabApp handlers:620] Permission denied: Untitled.ipynb
[W 2021-08-17 08:58:03.578 SingleUserLabApp log:174] 403 POST /user/limy/api/contents/?1629190692424 (limy@191.168.2.218) 49.51ms

Problem screenshot:

My deployment environment information：

jupyterhub environment information

jupyterhub                         1.4.2
jupyterhub-kubespawner             1.1.0
jupyterhub-dummyauthenticator      0.3.1

Jupyterhub brief configuration information

c.KubeSpawner.image = 'jupyterhub/singleuser:1.0.0'
c.KubeSpawner.service_account = 'default'
c.KubeSpawner.cmd = ["jupyter-labhub","--notebook-dir=/home/jovyan/work/"]
c.KubeSpawner.image_pull_policy = 'IfNotPresent'
c.KubeSpawner.namespace = 'jupyterhub-k8s'
c.KubeSpawner.volumes = [
  {
     'name': 'test-container-{username}--{servername}',
     'persistentVolumeClaim': {
         'claimName': 'test-pvc-jupyter'
     }
  }
]

c.KubeSpawner.volume_mounts = [
  {
      'name': 'test-container-{username}--{servername}',
      'mountPath': '/home/jovyan/work/',
      'subPath': 'sub-{username}--{servername}'
  }
]

c.KubeSpawner.storage_access_modes = ['ReadWriteMany']
c.KubeSpawner.storage_capacity = '5G'
c.KubeSpawner.storage_class = 'normal'
c.KubeSpawner.storage_extra_labels = {'pvc-label':'pvc-label-value'}
c.KubeSpawner.storage_pvc_ensure = True
c.KubeSpawner.delete_pvc = False
c.KubeSpawner.pvc_name_template = 'test-pvc-jupyter'

k8s PersistentVolume information (yaml)

apiVersion: v1
kind: PersistentVolume
metadata:  
  name: test-pv-jupyter-nfs  
  namespace: jupyterhub-k8s
  labels: 
    name: test-pv-jupyter-nfs
    storetype: nfs
spec:  
  storageClassName: normal
  accessModes:  
    - ReadWriteMany
  capacity: 
    storage: 5G
#  persistentVolumeReclaimPolicy: Retain 
  nfs:
    path: /opt/jupyterhub
    server: 191.168.6.1

Linux virtual machine /opt/jupyterhub/ directory information
```
drwxr-xr-x. 3 jovyan jovyan  22 Aug 17 16:55 jupyterhub
```

manics · August 17, 2021, 7:23pm

Can you show us the output of

kubectl get pv,pvc

in your JupyterHub namespace?

meng-yu-github · August 18, 2021, 5:48am

pv，pvc status

# kubectl get pvc,pv -n jupyterhub-k8s
NAME                                     STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/test-pvc-jupyter   Bound    test-pv-jupyter-nfs   5G         RWX            normal         19h

NAME                                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                             STORAGECLASS   REASON   AGE
persistentvolume/test-pv-jupyter-nfs   5G         RWX            Retain           Bound    jupyterhub-k8s/test-pvc-jupyter   normal                  19h

manics · August 18, 2021, 6:25pm

It sounds like the permissions on your volume are still incorrect.

Can you run ls -lR /opt/jupyterhub on the VM and and ls -lR /home/jovyan in the pod?

meng-yu-github · August 19, 2021, 1:42am

ls -lR /opt/jupyterhub

/opt/jupyterhub:
total 0
drwxr-xr-x. 2 root root 6 Aug 19 09:35 sub-limy

/opt/jupyterhub/sub-limy:
total 0

kubectl exec jupyter-limy -n jupyterhub-k8s -it – /bin/bash

(base) jovyan@jupyter-limy:~$ ls -lR /home/jovyan
/home/jovyan:
total 0
drwxr-xr-x. 2 root root 6 Aug 19 01:35 work

/home/jovyan/work:
total 0

manics · August 19, 2021, 6:13pm

As you can see your directories under /opt/jupyterhub are still owned by root. One way this can occur is if the parent directory was initially empty, so when the volume is mounted the directory defaults to being root owned.

If this is the case running chmod -R ... to fix the permissions on everything under /opt/jupyterhub should fix things for now. However if you later add more users you may need to reset the permissions on their folder, unless you create the folders in advance with the correct permissions.

meng-yu-github · August 20, 2021, 6:02am

Thank you for your explanation

I use NFS to map all user identities to those that do not exist in the container. The reason for this problem is that the file inside the container does not belong to the master file, and other users can access it.

/opt/jupyterhub *(rw,sync,all_squash,anonuid=1002,anongid=1002)

manics · August 20, 2021, 8:08pm

If you want you could try running your container as the UID of the logged in user by overriding pre_spawn_start
See for example

Note the example is a bit out of date, but the general idea still holds.

meng-yu-github · August 23, 2021, 1:46am

Ok. Thanks.

I’ll try it later

Topic		Replies	Views
Proxy loses track of singleuser servers after k8s restarts them JupyterHub	3	576	February 27, 2019
Singleuser image(pod) lifetime Zero to JupyterHub on Kubernetes	2	236	January 4, 2024
How to cleanup orphaned user pods after bug in z2jh 3.0 and KubeSpawner 6.0 Zero to JupyterHub on Kubernetes announcement , how-to	1	1160	November 2, 2023
Jupyterhub-singleuser and jupyter_server JupyterHub	13	5380	April 27, 2022
When the server machine restarts, the users' data is lost JupyterHub community , jupyterhub , how-to	1	520	September 2, 2022

The `c.JupyterHub.cleanup_servers = False` parameter is invalid and the corresponding pod is deleted on the jupyterhub+k8s+jupyterlab

Related topics