How to make created custom (conda) environments persist across user sessions in Jupyter Hub on AWS EKS cluster

JupyterHub Zero to JupyterHub on Kubernetes
, ,
1

Hi All,

Greetings!!

Hope everyone is doing well.

Thanks in advance.

I have installed the Jupyter hub on the AWS EKS cluster.

Objective 1: We want to allow users to create conda environments using Jupyter lab UI which needs to be persisted across all user sessions meaning user 1 has created conda environment = xyz and the same environment can be accessed by user 2 when that user logs in to JH.

As mentioned in the article below, I have followed the below steps.

https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html

  1. Ensure the “nb_conda_kernels” package is installed in the root environment - have mentioned “nb_conda_kernels” in the docker file.

Here is my Dockerfile:
##########################################################################
[#] Get the jupyter hub image of minimal-notebook from Docker hub
FROM jupyter/minimal-notebook:latest

[#] Allow users to create their own conda environments for notebooks
USER root

RUN conda install -c conda-forge nb_conda_kernels
RUN conda install -c conda-forge ipykernel

COPY .condarc /tmp/work/

[#] Copy environment.yml file to /tmp/work/ directory
COPY custom_environment.yml /tmp/work/

[#] Switch back to jovyan to avoid accidental container runs as root
USER ${NB_UID}
##########################################################################

  1. Configure Anaconda to install user environments to a folder within “$HOME” = /home/jovyan/.
    Created a file called .condarc and copied it to the home folder for all users.

JupyterHub config.yaml - lifecycle hooks:

lifecycleHooks:
postStart:
exec:
command:
- “sh”
- “-c”
- >
cp /tmp/work/custom_environment.yml /home/jovyan/;
cp /tmp/work/.condarc /home/jovyan/;

  1. Did the helm upgrade, restarted all user’s servers. Login with user 1 and created the below environments which can be seen in home directory.

Commands used to create the environments in Jupyter lab terminal:

  1. conda create -n myenv --yes python=3.8 ipykernel
  2. conda activate myenv
  3. python -m ipykernel install --user --name myenv
  4. conda env list

Same commands used to create the conda environment = d2l

conda environments:

d2l /home/jovyan/my-conda-envs/d2l
myenv /home/jovyan/my-conda-envs/myenv
base * /opt/conda

  1. When I login in with user 2 then we can’t see “d2l” and “myenv” in the “/home/jovyan/my-conda-envs/” directory.
    But as per the documentation, created conda environments need to persist across user sessions but it’s not happening. Could you please check and help?

Note: I am publishing a docker image to docker hub and pulling the same while installing JH.

Objective 2: As an admin we will create conda environments in advance in docker image and replicate those conda virtual environments across all the users.

I have followed below steps to create conda environments in advance in docker image.

  1. Created the environments in docker image. Here is my docker file.
    ##########################################################################
    [#] Get the jupyter hub image of minimal-notebook from Docker hub
    FROM jupyter/minimal-notebook:latest

[#] Allow users to create their own conda environments for notebooks
USER root

RUN conda install -c conda-forge nb_conda_kernels
RUN conda install -c conda-forge ipykernel

COPY .condarc $HOME
COPY .condarc /tmp/work/

[#] Copy environment.yml file to /tmp/work/ directory
COPY custom_environment.yml /tmp/work/

[#] Create a custom environment 1 = d2l
RUN conda env create -f /tmp/work/custom_environment.yml

RUN python -m ipykernel install --name d2l

[#] Create a custom environment 2 = myenv with specific version of python
RUN conda create -n myenv --yes python=3.8 ipykernel

[#] Switch back to jovyan to avoid accidental container runs as root
USER ${NB_UID}
##########################################################################

  1. Build the docker image, run it local system to verify the created conda environments in docker image and can see those environments.

Docker image - conda environments
Docker image - conda environments736×305 11 KB

  1. JupyterHub config.yaml - lifecycle hooks:

lifecycleHooks:
postStart:
exec:
command:
- “sh”
- “-c”
- >
cp /tmp/work/.condarc /home/jovyan/;

  1. Did the helm upgrade, restarted all users servers. When users login with their credentials then they can’t see the conda environments which we created in docker image.

Note: I am publishing a docker image to docker hub and pulling the same while installing JH.

Could you please check and help on these issues?

Thanks & Regards,
Vinayak

2

could you provide .condarc file?

3

Thanks for your reply.

As requested, please find the lines inside the .condarc file.

condarc

4

Thanks for sharing file. Although, in Notebook, we’ve better way to run pre-start scripts before notebook actual starts. click here for more info
So, you can keep a script file inside a folder /usr/local/bin/start-notebook.d. The run-hooks function looks for .sh scripts to source and executable files to run within a passed directory. You can keep a script which will perform all above copying operations and can setup .condarc file on fly like this

CONDARC_FILE=~/.condarc
if [ -f "$CONDARC_FILE" ]; then
    echo "conda configuration exists."
else 
    echo "$CONDARC_FILE does not exist. writing one..."
    cat > "$CONDARC_FILE" <<EOF
envs_dirs:
  - /home/jovyan/.conda-envs/
EOF
fi
5

Thanks for your reply.

As suggested, I copied your .condarc file into “/usr/local/bin/start-notebook.d/” folder then built the docker image using below file contents.

After I login, I have created a conda environment which is located in “/opt/conda/envs/myenv” but as per the condarc file it supposed to create under “/home/jovyan/.conda-envs/” location.

Here is the Dockerfile:
##########################################################################
[#] Get the jupyter hub image of minimal-notebook from Docker hub
FROM jupyter/minimal-notebook:latest

[#] Allow users to create their own conda environments for notebooks
USER root

RUN conda install -c conda-forge nb_conda_kernels
RUN conda install -c conda-forge ipykernel

COPY .condarc /usr/local/bin/start-notebook.d/

[#] Copy environment.yml file to /tmp/work/ directory
COPY custom_environment.yml /tmp/work/

[#] Switch back to jovyan to avoid accidental container runs as root
USER ${NB_UID}
##########################################################################

Could you please break it down and explain?

Thank you

6

Sorry for late reply.

I checked your dockerfile.

you should be creating a script with above bash script which will be copied to /usr/local/bin/start-notebook.d/

so, create conda_setup.sh file with below code

#!/bin/bash
CONDARC_FILE=~/.condarc
if [ -f "$CONDARC_FILE" ]; then
    echo "conda configuration exists."
else 
    echo "$CONDARC_FILE does not exist. writing one..."
    cat > "$CONDARC_FILE" <<EOF
envs_dirs:
  - /home/jovyan/.conda-envs/
EOF
fi

and now your Dockerfile should look like this

[#] Get the jupyter hub image of minimal-notebook from Docker hub
FROM jupyter/minimal-notebook:latest

[#] Allow users to create their own conda environments for notebooks
USER root

RUN conda install -c conda-forge nb_conda_kernels
RUN conda install -c conda-forge ipykernel

COPY conda_setup.sh /usr/local/bin/start-notebook.d/

[#] Copy environment.yml file to /tmp/work/ directory
COPY custom_environment.yml /tmp/work/

[#] Switch back to jovyan to avoid accidental container runs as root
USER ${NB_UID}

Look if you can locate file .condarc at location /home/jovyan/.condarc with following content

envs_dirs:
  - /home/jovyan/.conda-envs/