Jupyter Kernel not running same Python version as the Virtual Environment

I want to be able to run both Python 3.8 (currrent version) and Python 3.7 in my Jupyter Notebook. I understand creating different IPython kernels from virtual environments is the way.
So I downloaded Python 3.7 and locally installed it in my home directory. Used this python binary file to create a virtual environment by

> virtualenv -p ~/Python3.7/bin/python3 py37
> source py37/bin/activate

This works perfectly and gives ‘Python 3.7’ correctly on checking with python --version and sys.version.
Then for creating IPython kernel,

(py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
(py37) > jupyter notebook

This also runs without error and the kernel can be confirmed to be added in the Notebook. However it does not run Python 3.7 like the virtual environment, but Python 3.8 like the default kernel. (confirmed with sys.version)

I checked ~/.local/share/jupyter/kernels/py37/kernel.json and saw its contents as

{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3.7",
 "language": "python"

So naturally I tried editing the /usr/bin/python3 to point to my Python 3.7 binary file path that is ~/Python3.7/bin/python3, but then even the kernel doesn’t work properly in the notebook.

What can I possibly do?

NB: I use Arch Linux, so I installed jupyter, virtualenv, … through pacman not pip as its recommended in Arch.

Found it myself, the hard way. Let me share anyway, in case this helps anyone.

I guess, the problem was that, jupyter notebook installed through pacman searches for python binary files in the PATH variable and not in the path specified by the virtual environment. Since I installed Python 3.7 locally in my home directory, Jupyter can’t find it and it might have defaulted to the default python version.

So the possible solutions are:

  1. Install Jupyter Notebook through pip (instead of pacman) within the virtual environment set on Python 3.7 (This is not at all recommended for Arch Linux users, as installing packages through pip can probably cause issues in future)
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37     
 > make     
 > make install   

 > virtualenv -p ~/Python3.7/bin/python3 py37   
 > source py37/bin/activate 
  
 (py37) > pip install notebook   
 (py37) > python -m notebook
  1. Install Python 3.7 within default directory (instead of specifying somewhere else). Create a new IPython kernel using the suitable virtual environment and use jupyter-notebook installed through pacman. (Recommended for Arch Linux users)
    Note 1: > python points to the updated global Python 3.8 version and > python3 or > python3.7 points to newly installed Python 3.7
    Note 2: Once the required kernel is created, you might even be able to use that python version outside the virtual environment.
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure   
 > make     
 > sudo make install   

 > virtualenv -p $(which python3.7) py37   
 > source py37/bin/activate 
  
 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook
  1. Add the path of the directory where you have locally installed the new Python version to the $PATH variable, create an IPython kernel and run Jupyter Notebook within suitable virtual environment. (Haven’t yet tried this one personally. Just felt that this should work. So no guarantee. Also I don’t think this is a good solution)
 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37 
 > make     
 > make install   
 > export PATH="$HOME/Python37/bin:$PATH"          

 > virtualenv -p  py37   
 > source py37/bin/activate 
  
 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook
1 Like