Jupyter Kernel not running same Python version as the Virtual Environment

Notebook
1

I want to be able to run both Python 3.8 (currrent version) and Python 3.7 in my Jupyter Notebook. I understand creating different IPython kernels from virtual environments is the way.
So I downloaded Python 3.7 and locally installed it in my home directory. Used this python binary file to create a virtual environment by

> virtualenv -p ~/Python3.7/bin/python3 py37
> source py37/bin/activate

This works perfectly and gives ‘Python 3.7’ correctly on checking with python --version and sys.version.
Then for creating IPython kernel,

(py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
(py37) > jupyter notebook

This also runs without error and the kernel can be confirmed to be added in the Notebook. However it does not run Python 3.7 like the virtual environment, but Python 3.8 like the default kernel. (confirmed with sys.version)

I checked ~/.local/share/jupyter/kernels/py37/kernel.json and saw its contents as

{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3.7",
 "language": "python"

So naturally I tried editing the /usr/bin/python3 to point to my Python 3.7 binary file path that is ~/Python3.7/bin/python3, but then even the kernel doesn’t work properly in the notebook.

What can I possibly do?

NB: I use Arch Linux, so I installed jupyter, virtualenv, … through pacman not pip as its recommended in Arch.

2

Found it myself, the hard way. Let me share anyway, in case this helps anyone.

I guess, the problem was that, jupyter notebook installed through pacman searches for python binary files in the PATH variable and not in the path specified by the virtual environment. Since I installed Python 3.7 locally in my home directory, Jupyter can’t find it and it might have defaulted to the default python version.

So the possible solutions are:

  1. Install Jupyter Notebook through pip (instead of pacman) within the virtual environment set on Python 3.7 (This is not at all recommended for Arch Linux users, as installing packages through pip can probably cause issues in future)
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37     
 > make     
 > make install   

 > virtualenv -p ~/Python3.7/bin/python3 py37   
 > source py37/bin/activate 
  
 (py37) > pip install notebook   
 (py37) > python -m notebook
  1. Install Python 3.7 within default directory (instead of specifying somewhere else). Create a new IPython kernel using the suitable virtual environment and use jupyter-notebook installed through pacman. (Recommended for Arch Linux users)
    Note 1: > python points to the updated global Python 3.8 version and > python3 or > python3.7 points to newly installed Python 3.7
    Note 2: Once the required kernel is created, you might even be able to use that python version outside the virtual environment.
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure   
 > make     
 > sudo make install   

 > virtualenv -p $(which python3.7) py37   
 > source py37/bin/activate 
  
 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook
  1. Add the path of the directory where you have locally installed the new Python version to the $PATH variable, create an IPython kernel and run Jupyter Notebook within suitable virtual environment. (Haven’t yet tried this one personally. Just felt that this should work. So no guarantee. Also I don’t think this is a good solution)
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37 
 > make     
 > make install   
 > export PATH="$HOME/Python37/bin:$PATH"          

 > virtualenv -p  py37   
 > source py37/bin/activate 
  
 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook
1 Like
3

This is probably a rare edge case, and I know it doesn’t apply to the original post here, but there aren’t many threads coming up on the subject.

For me, it wasn’t working with my virtualenvwrapper environment. When I switched to a normal virtualenv, it worked.

Maybe this will help some people. I know it won’t help the OP, since he’s already using virtualenv.

1 Like
5

I’ve experienced the same problem for a while now. My system is a Mac OSX 13.6.3, but this has happened through many os versions, same machine intel core i5, zsh in term2, zsh version 5.9 (x86_64-apple-darwin22.0). I created a venv using venv, activated it, pip installed jp nb (obv within the env), when I launched jp-nb, the kernel points to the global python on my system and not the venv version, therefore no libs from the virtualenv site packages which were pip installed accessable. Only way to get it to work is to append to sys.path the venv site packages manually in jp-nb. Forcing the kernel via ipython kernel install was one of the options that worked, but I felt it was a work around. There is a way modify the kernel spec of jp-nb, but I didnt have time to try it. Check this article good article. Installing Python Packages from a Jupyter Notebook | Pythonic Perambulations

So I had thought the python -m pip install module in an activate virtualenv was working but in fact it isn’t. I do not think this post should be closed.

1 Like
6

That resource is woefully outdated @sndselecta .
The magic install commands were added in 2019 to make it much easier to install packages from within the running notebook.
See here for more about the modern magic install commands.
You’ll see one of those listed resources, this one in particular, is by the author of Pythonic Permabulations suggesting the way added in 2019 is the improvement since that 2017 post you referenced.