Unable to reliably use single JypyterLab server with multiple conda environments

JupyterLab
3

Creating regular kernels for conda envs

I seemed to be able to get this working on my linux machine without nb_conda_kernels. Here are the steps I took:

To create an environment with a kernel follow these steps (replace conda create with conda install in the first command to create a kernel for an existing environment):

conda create -n kernelenv ipykernel
conda activate kernelenv
python -m ipykernel install --user --name kernelenv --display-name 'Python (kernelenv)'

The last command above should output something like:

Installed kernelspec kernelenv in /home/username/.local/share/jupyter/kernels/kernelenv

An easy way to reuse the same commands for multiple environments would be to put the env name in a variable so you only have to change it in one place:

KERNELENV=some-env-name
conda create -n "$KERNELENV" ipykernel
conda activate "$KERNELENV"
python -m ipykernel install --user --name "$KERNELENV" --display-name "Python ($KERNELENV)"

To create a dedicated jupyterlab conda env:

conda create -n jupyterenv -c conda-forge jupyterlab

Then to run jupyter lab:

conda activate jupyterenv
jupyter lab

You should now have a kernel available named Python (kernelenv) (or whatever you passed to the --display-name option in the kernel env’s python -m ipykernel command.

I referenced the IPython documentation’s section Kernels for different environments, which has some more info that might be helpful.

Kernel interrupt works fine for me with this method and I verified that I can import a package when using the created kernel only if it’s installed in the kernel env.

Custom kernel startup commands

If you find that the method above doesn’t completely work or you or if you need to run anything else before starting the kernel, here is how you can customize your kernel further to run any command (including conda activate env-name) before the kernel starts:

Create a kernel for a conda env as above. Then edit the kernel.json file in the created kernel directory (i.e. /home/username/.local/share/jupyter/kernels/kernelenv/kernel.json). It should have something like this:

{
 "argv": [
  "/home/arch-jon/.conda/envs/kernelenv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (kernelenv)",
 "language": "python"
}

You can change the argv list to contain any command, but the easiest thing to do is probably to create a script which you’ll pass the env name and connection file:

{
 "argv": [
  "launch-custom-kernel",
  "env-name",
  "{connection_file}"
 ],
 "display_name": "Python (env-name)",
 "language": "python"
}

Then, simply create a script called launch-custom-kernel with:

#!/bin/bash
if [ $# -lt 2 ]; then
        echo "Must pass virtualenv_name and connection_file" 1>&2
        exit 1
fi
# Run any setup commands here. For example:
#   conda activate "$1"

# Efficiently check if ipykernel is installed and install it if not. Replace `pip` with `conda` if using `conda`
python -c 'import pkgutil, sys; exit(0 if pkgutil.find_loader(sys.argv[1]) is not None else 1)' ipykernel || pip install ipykernel
python -m ipykernel_launcher -f "$2"

Make sure this script is executable. Also either make sure it’s on your PATH or simply use the full path to launch-custom-kernel in the kernel.json file. For example if you put launch-custom-kernel in your ~/bin directory and it’s not on your PATH, use:

{
 "argv": [
  "/home/username/bin/launch-custom-kernel",
...

Another example of when such a custom script is useful is if you use virtualenvwrapper. I personally use the following launch-virtualenvwrapper-kernel script, which allows me to easier configure any setup scripts or environment variables for each virtualenv I use:

#!/bin/bash

# Allows a jupyterlab installation that exists outside of a virtualenv
# to have a kernel that automatically activates a virtualenvwrapper
# virtualenv and runs inside it.
# This could be done by creating a file ~/.local/share/jupyter/kernels/kernel-name/kernel.json with:
# {
#  "argv": [
#   "launch-virtualenvwrapper-kernel",
#   "virtualenv-name",
#   "{connection_file}"
#  ],
#  "display_name": "kernel-display-name",
#  "language": "python"
# }

if [ $# -lt 2 ]; then
        echo "Must pass virtualenv_name and connection_file" 1>&2
        exit 1
fi
if ! command -v workon; then
        source "$(command -v virtualenvwrapper.sh)"
fi
workon "$1"
python -c 'import pkgutil, sys; exit(0 if pkgutil.find_loader(sys.argv[1]) is not None else 1)' ipykernel || pip install ipykernel
python -m ipykernel_launcher -f "$2"
2 Likes