[zero-to-jupyterhub] Create isolated and editable environments for each user in advance

Bug description

I want users to be able to use and edit the env environment in their own directory directly after logging in, and this env is created in advance.
I then created and activated environments in the Dockerfile.
But the env path created is not in /home/jovyan/.conda/envs, but in /opt/conda/envs.
This way changes made by user A to the env environment will be reset the next time he logs in

How to reproduce

Here is my dockerfile

FROM ideonate/containds-allr-datascience:latest

USER root

RUN pip3 install ipykernel

COPY tf1.yml .

RUN conda env create -f tf1.yml
# Because of the mounting problem, if you specify the installation path, nothing will appear after login
# RUN conda env create --prefix=/home/jovyan/.conda/envs/tf1_gpu -f tf1.yml

RUN conda init bash

RUN rm tf1.yml 

USER $NB_UID

You can use a lifecycle hook to copy files in to a user’s persistent volume:
https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#about-user-storage-and-adding-files-to-it

Thank you very much for your reply!
I tried to modify the following configuration,but it doesn’t seem to work. :sweat_smile:
The error message is ‘Unable to create folder’ or ‘The /home/jovyan/.conda/env/ folder was not found’ or timeout error

singleuser:
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            mkdir /home/jovyan/.conda/envs;
            cp -r /opt/conda/envs /home/jovyan/.conda/envs

If the /home/jovyan/.conda directory doesn’t exist you might get that error, try mkdir -p instead. If you still have problems the permissions on the mounted volume may be incorrect, you’ll have to look at your dynamic volume provisioner to fix it.

Hi :blush:
First of all, thank you very much for your timely reply
I changed the configuration according to your tips and there still seems to be a little problem

Exec lifecycle hook ([sh -c mkdir -p /home/jovyan/.conda/; cp -r /opt/conda/envs /home/jovyan/.conda/ ]) for Container "notebook" in Pod "jupyter-yangyuandong_jhub(3171ff82-152b-436d-ac55-b9242b1c0f10)" failed - error: command 'sh -c mkdir -p /home/jovyan/.conda/; cp -r /opt/conda/envs /home/jovyan/.conda/ ' exited with 137: , message: ""

Can you show us the pod logs for the failing singleuser server?

I am sorry to reply to you only now
In fact the pod error cannot see the log
Maybe this activity could have some information

You could try deploying the pod without the lifecycle hook, then running it manually and see if it works?

deploying the pod without the lifecycle hook, pod is works

Can you show us the full pod logs from kubernetes?

I tried to modify the function and here is the configuration file

  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            mkdir -p /home/jovyan/.conda/;

This configuration allowed the pod to run successfully
The following is the log of a successfully run pod

[W 2021-09-15 03:37:56.016 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2021-09-15 03:37:56.052 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-09-15 03:37:56.955 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-15 03:37:56.956 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-09-15 03:37:56.964 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-09-15 03:37:56.964 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-09-15 03:37:57.752 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[W 2021-09-15 03:37:57.761 SingleUserNotebookApp _version:70] jupyterhub version 1.3.0 != jupyterhub-singleuser version 1.4.1. This could cause failure to authenticate and result in redirect loops!
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /data/User/lumeixi
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2302] http://jupyter-lumeixi:8888/user/lumeixi/
[I 2021-09-15 03:37:57.762 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-09-15 03:37:57.795 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds

But once I add cp -r /opt/conda/envs /home/jovyan/.conda/, the pod doesn’t work properly

Do you have the pod logs after adding the failing line (not just the logs from JupyterHub)?

Sorry, I didn’t find the failed pod log in the node until now
The logs are as follows

[W 2021-09-16 06:19:43.493 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2021-09-16 06:19:43.530 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-09-16 06:19:44.495 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-09-16 06:19:44.495 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-09-16 06:19:44.503 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-09-16 06:19:44.503 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-09-16 06:19:45.299 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[W 2021-09-16 06:19:45.308 SingleUserNotebookApp _version:70] jupyterhub version 1.3.0 != jupyterhub-singleuser version 1.4.1. This could cause failure to authenticate and result in redirect loops!
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /data/User/lumeixi
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2302] http://jupyter-lumeixi:8888/user/lumeixi/
[I 2021-09-16 06:19:45.308 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-09-16 06:19:45.341 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[C 2021-09-16 06:24:41.137 SingleUserNotebookApp notebookapp:1978] received signal 15, stopping
[I 2021-09-16 06:24:41.204 SingleUserNotebookApp notebookapp:2145] Shutting down 0 kernels
[I 2021-09-16 06:24:41.205 SingleUserNotebookApp notebookapp:2160] Shutting down 0 terminals

I’m running out of ideas. I think your best option is to try different commands and see what does or doesn’t work. For instance, try creating a single file in your hook, try reading a source file, copying a single file, etc. until you figure out what’s failing.

Thanks again for your reply and help! :blush:
My guess is that the folder is too big and it takes some time to copy it, but the time set is not enough, so there is a timeout error
I will continue to try and modify the startpod time

1 Like