Hi guys,
I have installed jupyterhub by installing miniconda3 in /usr/local/miniconda3 and create a new environment:
conda create -n jupyterhub python=3.8
conda activate jupyterhub
conda install jupyterhub jupyterlab
and it works fine.
I use the LocalProcessSpawner
and create new users if there is not a existing user.
Now I want that users can install python packages by their own in their notebooks, for example, if I want to install xgboost
, I can do like this in a notebook:
!pip install xgboost
Since the user is not a linux superuser, the /usr/local/miniconda dir is not writable (which is what I hope to achieve isolation), the package will be installed to ~/.local/lib/python3.8/site-packages
.
If I type python
in the terminal and import xgboost
, it works well. However, If I import this package in a notebook (note that the python interpreter in the kernel is exactly the same as the python interpreter in the terminal), it says “no module named xgboost”. I believe the difference is caused because in the notebook environment, ~/.local/lib/python3.8/site-packages
is not added to sys.path
. this can by verified by import sys; sys.path
in both notebook and terminal.
in the notebook:
['/home/movie',
'/usr/local/miniconda3/envs/deeplearning/lib/python37.zip',
'/usr/local/miniconda3/envs/deeplearning/lib/python3.7',
'/usr/local/miniconda3/envs/deeplearning/lib/python3.7/lib-dynload',
'',
'/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages',
'/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages/IPython/extensions',
'/home/movie/.ipython']
I installed another kernel named “deeplearning” in another conda env currently I’m using this kernel in the notebook. However, when I type !which python
, it is still the python in the jupyterhub conda env:
!which python
/usr/local/miniconda3/envs/jupyterhub/bin/python
instead of
/usr/local/miniconda3/envs/deeplearning/bin/python
in terminal:
['',
'/usr/local/miniconda3/envs/jupyterhub/lib/python38.zip',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/lib-dynload',
'/home/movie/.local/lib/python3.8/site-packages',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/site-packages']
I’m wondering the reason that caused this problem and is there a workaround to let user can install packages in their own HOME dir and use them in the notebook in jupyterhub?
Thanks!