I have installed jupyterhub by installing miniconda3 in /usr/local/miniconda3 and create a new environment:
conda create -n jupyterhub python=3.8 conda activate jupyterhub conda install jupyterhub jupyterlab
and it works fine.
I use the
LocalProcessSpawner and create new users if there is not a existing user.
Now I want that users can install python packages by their own in their notebooks, for example, if I want to install
xgboost, I can do like this in a notebook:
!pip install xgboost
Since the user is not a linux superuser, the /usr/local/miniconda dir is not writable (which is what I hope to achieve isolation), the package will be installed to
If I type
python in the terminal and
import xgboost, it works well. However, If I import this package in a notebook (note that the python interpreter in the kernel is exactly the same as the python interpreter in the terminal), it says “no module named xgboost”. I believe the difference is caused because in the notebook environment,
~/.local/lib/python3.8/site-packages is not added to
sys.path. this can by verified by
import sys; sys.path in both notebook and terminal.
in the notebook:
['/home/movie', '/usr/local/miniconda3/envs/deeplearning/lib/python37.zip', '/usr/local/miniconda3/envs/deeplearning/lib/python3.7', '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/lib-dynload', '', '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages', '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages/IPython/extensions', '/home/movie/.ipython']
I installed another kernel named “deeplearning” in another conda env currently I’m using this kernel in the notebook. However, when I type
!which python, it is still the python in the jupyterhub conda env:
!which python /usr/local/miniconda3/envs/jupyterhub/bin/python
['', '/usr/local/miniconda3/envs/jupyterhub/lib/python38.zip', '/usr/local/miniconda3/envs/jupyterhub/lib/python3.8', '/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/lib-dynload', '/home/movie/.local/lib/python3.8/site-packages', '/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/site-packages']
I’m wondering the reason that caused this problem and is there a workaround to let user can install packages in their own HOME dir and use them in the notebook in jupyterhub?