Can not read the packages in ~/.local/lib/python3.8/site-packages installed in the notebook

Hi guys,
I have installed jupyterhub by installing miniconda3 in /usr/local/miniconda3 and create a new environment:

conda create -n jupyterhub python=3.8
conda activate jupyterhub
conda install jupyterhub jupyterlab

and it works fine.
I use the LocalProcessSpawner and create new users if there is not a existing user.
Now I want that users can install python packages by their own in their notebooks, for example, if I want to install xgboost, I can do like this in a notebook:

!pip install xgboost

Since the user is not a linux superuser, the /usr/local/miniconda dir is not writable (which is what I hope to achieve isolation), the package will be installed to ~/.local/lib/python3.8/site-packages.
If I type python in the terminal and import xgboost, it works well. However, If I import this package in a notebook (note that the python interpreter in the kernel is exactly the same as the python interpreter in the terminal), it says “no module named xgboost”. I believe the difference is caused because in the notebook environment, ~/.local/lib/python3.8/site-packages is not added to sys.path. this can by verified by import sys; sys.path in both notebook and terminal.

in the notebook:

['/home/movie',
 '/usr/local/miniconda3/envs/deeplearning/lib/python37.zip',
 '/usr/local/miniconda3/envs/deeplearning/lib/python3.7',
 '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/lib-dynload',
 '',
 '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages',
 '/usr/local/miniconda3/envs/deeplearning/lib/python3.7/site-packages/IPython/extensions',
 '/home/movie/.ipython']

I installed another kernel named “deeplearning” in another conda env currently I’m using this kernel in the notebook. However, when I type !which python, it is still the python in the jupyterhub conda env:

!which python
/usr/local/miniconda3/envs/jupyterhub/bin/python

instead of

/usr/local/miniconda3/envs/deeplearning/bin/python

in terminal:

['',
'/usr/local/miniconda3/envs/jupyterhub/lib/python38.zip',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/lib-dynload',
'/home/movie/.local/lib/python3.8/site-packages',
'/usr/local/miniconda3/envs/jupyterhub/lib/python3.8/site-packages']

I’m wondering the reason that caused this problem and is there a workaround to let user can install packages in their own HOME dir and use them in the notebook in jupyterhub?

Thanks!

I think the real reason is that if I install a kernel using another conda environment, when I use this kernel, the python interpreter in the notebook is still the python in the conda env that installed jupyterhub (even though the packages in another conda env can be read). Can anyone tell me the difference between a kernel and a conda virtual environment?