I have multiple virtual environment and corresponding kernels. And when i try to call python from the notebook with !python xxxx.py or pip install xxx , i realized that the python or pip is the one that starts the Jupyter Server, not the kernel one.
So is there a way to use the kernel version of python or pip when using in notebook with !pip or !python ?
There is a pip magic, so you can use %pip install to always get the right version for the current interpreter. The other way, which works for any Python entrypoint, is to use variable expansion with {sys.executable}:
import sys
!{sys.executable} -m pip install ...
which will expand to !/path/to/kernel/bin/python -m ...
I would have steered you towards using %run xxxx.py magic in a cell instead of!python xxxx.py. I find that has better features in relating feedback/linking to the notebook output format. I usually describe it as more feature-rich than the !python. See details here.