[zero-to-jupyterhub] Create isolated and editable environments for each user in advance

JupyterHub Zero to JupyterHub on Kubernetes
1

Bug description

I want users to be able to use and edit the env environment in their own directory directly after logging in, and this env is created in advance.
I then created and activated environments in the Dockerfile.
But the env path created is not in /home/jovyan/.conda/envs, but in /opt/conda/envs.
This way changes made by user A to the env environment will be reset the next time he logs in

How to reproduce

Here is my dockerfile

FROM ideonate/containds-allr-datascience:latest

USER root

RUN pip3 install ipykernel

COPY tf1.yml .

RUN conda env create -f tf1.yml
# Because of the mounting problem, if you specify the installation path, nothing will appear after login
# RUN conda env create --prefix=/home/jovyan/.conda/envs/tf1_gpu -f tf1.yml

RUN conda init bash

RUN rm tf1.yml 

USER $NB_UID
2

You can use a lifecycle hook to copy files in to a user’s persistent volume:
https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#about-user-storage-and-adding-files-to-it

3

Thank you very much for your reply!
I tried to modify the following configuration,but it doesn’t seem to work. :sweat_smile:
The error message is ‘Unable to create folder’ or ‘The /home/jovyan/.conda/env/ folder was not found’ or timeout error

singleuser:
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            mkdir /home/jovyan/.conda/envs;
            cp -r /opt/conda/envs /home/jovyan/.conda/envs
4

If the /home/jovyan/.conda directory doesn’t exist you might get that error, try mkdir -p instead. If you still have problems the permissions on the mounted volume may be incorrect, you’ll have to look at your dynamic volume provisioner to fix it.

5

Hi :blush:
First of all, thank you very much for your timely reply
I changed the configuration according to your tips and there still seems to be a little problem

image
image950×705 87.3 KB 

Exec lifecycle hook ([sh -c mkdir -p /home/jovyan/.conda/; cp -r /opt/conda/envs /home/jovyan/.conda/ ]) for Container "notebook" in Pod "jupyter-yangyuandong_jhub(3171ff82-152b-436d-ac55-b9242b1c0f10)" failed - error: command 'sh -c mkdir -p /home/jovyan/.conda/; cp -r /opt/conda/envs /home/jovyan/.conda/ ' exited with 137: , message: ""
6

Can you show us the pod logs for the failing singleuser server?

7

I am sorry to reply to you only now
In fact the pod error cannot see the log
Maybe this activity could have some information

image
image990×714 34.1 KB

8

You could try deploying the pod without the lifecycle hook, then running it manually and see if it works?

9

deploying the pod without the lifecycle hook, pod is works

10

Can you show us the full pod logs from kubernetes?

11

I tried to modify the function and here is the configuration file

  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            mkdir -p /home/jovyan/.conda/;

This configuration allowed the pod to run successfully
The following is the log of a successfully run pod

[W 2021-09-15 03:37:56.016 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2021-09-15 03:37:56.052 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-09-15 03:37:56.955 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-09-15 03:37:56.964 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-09-15 03:37:56.964 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-09-15 03:37:57.752 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[W 2021-09-15 03:37:57.761 SingleUserNotebookApp _version:70] jupyterhub version 1.3.0 != jupyterhub-singleuser version 1.4.1. This could cause failure to authenticate and result in redirect loops!
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /data/User/lumeixi
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] http://jupyter-lumeixi:8888/user/lumeixi/
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-09-15 03:37:57.795 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds

But once I add cp -r /opt/conda/envs /home/jovyan/.conda/, the pod doesn’t work properly

12

Do you have the pod logs after adding the failing line (not just the logs from JupyterHub)?

13

Sorry, I didn’t find the failed pod log in the node until now
The logs are as follows

[W 2021-09-16 06:19:43.493 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2021-09-16 06:19:43.530 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-09-16 06:19:44.495 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-09-16 06:19:44.503 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-09-16 06:19:44.503 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-09-16 06:19:45.299 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[W 2021-09-16 06:19:45.308 SingleUserNotebookApp _version:70] jupyterhub version 1.3.0 != jupyterhub-singleuser version 1.4.1. This could cause failure to authenticate and result in redirect loops!
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /data/User/lumeixi
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] http://jupyter-lumeixi:8888/user/lumeixi/
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-09-16 06:19:45.341 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[C 2021-09-16 06:24:41.137 SingleUserNotebookApp notebookapp:1978] received signal 15, stopping
[I 2021-09-16 06:24:41.204 SingleUserNotebookApp notebookapp:2145] Shutting down 0 kernels
[I 2021-09-16 06:24:41.205 SingleUserNotebookApp notebookapp:2160] Shutting down 0 terminals
14

I’m running out of ideas. I think your best option is to try different commands and see what does or doesn’t work. For instance, try creating a single file in your hook, try reading a source file, copying a single file, etc. until you figure out what’s failing.

15

Thanks again for your reply and help! :blush:
My guess is that the folder is too big and it takes some time to copy it, but the time set is not enough, so there is a timeout error
I will continue to try and modify the startpod time

1 Like