Unable to reliably use single JypyterLab server with multiple conda environments

JupyterLab
1

The workflow that I try to setup is the following:

  1. have a dedicated conda environment with jupyterlab installed in it (no other jupyterlab installations in any other conda envs);
  2. for each new project create a separate conda environment with all the needed packages and register the ipykernel (which is also installed in the project’s env) for it.

There are two possibilities to do the latter.

One way is to just manually register ipykernel via:

conda install -n conda_env ipykernel
conda activate conda_env
ipython kernel install --user --name=my-conda-env-kernel

When I start jupyterlab from its dedicated environment, the project’s kernel gets visible in Jupyter Notebook’s kernel selection box. However, in this case, it doesn’t actually activate the project’s environment. There are some hacky workarounds with modifying system path manually (e.g., see jupyter notebook issue #4569), but it doesn’t really solve the main problem in general plus is really inconvenient.

An alternative way is to use nb_conda_kernels extension for registering kernels. It properly activates the project’s environment when its kernel is selected from the dropdown menu, but has a substantial flaw: when the picked kernel is being run, it cannot be interrupted. I have created a cross-issue in both jupyterlab and nb-conda-kernels repositories that describes steps to reproduce the problem:

  1. unable to interrupt cell execution with a kernel from an external environment · Issue #180 · Anaconda-Platform/nb_conda_kernels · GitHub
  2. unable to interrupt cell execution if jupyter uses kernel from a different environment · Issue #8388 · jupyterlab/jupyterlab · GitHub

Unfortunately, it remains unclear for me at which side the source of the problem sits. There is no feedback in these issues and it seems that no progress is made to provide a fix. I hoped that a new version of JupyertLab would resolve the issue, but it didn’t.

Maybe someone has already found the solution for making the described workflow work? Or maybe I’m just doing something that jupyterlab isn’t designed for? I know that possible workaround is just to have jupyterlab installed in each project’s env, but it leads to an overhead with keeping track of jupyterlab version in each env and updating it, etc. I’d prefer to maintain just a single jupyterlab installation and work with all my other conda environments from it.

I’d appreciate any help/feedback. Thanks!

1 Like
2

I was desperate in attempts to find out what is the source of the problem and created a new issue in the jupyter notebook repo: kernel interrupt command is ignored · Issue #5985 · jupyter/notebook (github.com)

The good news, now it’s clear that the problem is at the nb_conda_kernels side. The bad news, the related issue is open for a long time and it’s unclear whether it will get a fix.

I thought that the usage scenario that I described is a common and intuitive way of using jupyterlab for multiple different projects that have dedicated conda environments. I believe that inability to interrupt running kernels is a major issue, as the level of exploration that jupyter notebooks offer sort of invites to test ideas that may lead to long-running or even infinite execution. Having to kill the kernel instead of simply interrupting it breaks the workflow and significantly reduces productivity. I’m still in a search of a viable workaround…

3

Creating regular kernels for conda envs

I seemed to be able to get this working on my linux machine without nb_conda_kernels. Here are the steps I took:

To create an environment with a kernel follow these steps (replace conda create with conda install in the first command to create a kernel for an existing environment):

conda create -n kernelenv ipykernel
conda activate kernelenv
python -m ipykernel install --user --name kernelenv --display-name 'Python (kernelenv)'

The last command above should output something like:

Installed kernelspec kernelenv in /home/username/.local/share/jupyter/kernels/kernelenv

An easy way to reuse the same commands for multiple environments would be to put the env name in a variable so you only have to change it in one place:

KERNELENV=some-env-name
conda create -n "$KERNELENV" ipykernel
conda activate "$KERNELENV"
python -m ipykernel install --user --name "$KERNELENV" --display-name "Python ($KERNELENV)"

To create a dedicated jupyterlab conda env:

conda create -n jupyterenv -c conda-forge jupyterlab

Then to run jupyter lab:

conda activate jupyterenv
jupyter lab

You should now have a kernel available named Python (kernelenv) (or whatever you passed to the --display-name option in the kernel env’s python -m ipykernel command.

I referenced the IPython documentation’s section Kernels for different environments, which has some more info that might be helpful.

Kernel interrupt works fine for me with this method and I verified that I can import a package when using the created kernel only if it’s installed in the kernel env.

Custom kernel startup commands

If you find that the method above doesn’t completely work or you or if you need to run anything else before starting the kernel, here is how you can customize your kernel further to run any command (including conda activate env-name) before the kernel starts:

Create a kernel for a conda env as above. Then edit the kernel.json file in the created kernel directory (i.e. /home/username/.local/share/jupyter/kernels/kernelenv/kernel.json). It should have something like this:

{
 "argv": [
  "/home/arch-jon/.conda/envs/kernelenv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (kernelenv)",
 "language": "python"
}

You can change the argv list to contain any command, but the easiest thing to do is probably to create a script which you’ll pass the env name and connection file:

{
 "argv": [
  "launch-custom-kernel",
  "env-name",
  "{connection_file}"
 ],
 "display_name": "Python (env-name)",
 "language": "python"
}

Then, simply create a script called launch-custom-kernel with:

#!/bin/bash
if [ $# -lt 2 ]; then
        echo "Must pass virtualenv_name and connection_file" 1>&2
        exit 1
fi
# Run any setup commands here. For example:
#   conda activate "$1"

# Efficiently check if ipykernel is installed and install it if not. Replace `pip` with `conda` if using `conda`
python -c 'import pkgutil, sys; exit(0 if pkgutil.find_loader(sys.argv[1]) is not None else 1)' ipykernel || pip install ipykernel
python -m ipykernel_launcher -f "$2"

Make sure this script is executable. Also either make sure it’s on your PATH or simply use the full path to launch-custom-kernel in the kernel.json file. For example if you put launch-custom-kernel in your ~/bin directory and it’s not on your PATH, use:

{
 "argv": [
  "/home/username/bin/launch-custom-kernel",
...

Another example of when such a custom script is useful is if you use virtualenvwrapper. I personally use the following launch-virtualenvwrapper-kernel script, which allows me to easier configure any setup scripts or environment variables for each virtualenv I use:

#!/bin/bash

# Allows a jupyterlab installation that exists outside of a virtualenv
# to have a kernel that automatically activates a virtualenvwrapper
# virtualenv and runs inside it.
# This could be done by creating a file ~/.local/share/jupyter/kernels/kernel-name/kernel.json with:
# {
#  "argv": [
#   "launch-virtualenvwrapper-kernel",
#   "virtualenv-name",
#   "{connection_file}"
#  ],
#  "display_name": "kernel-display-name",
#  "language": "python"
# }

if [ $# -lt 2 ]; then
        echo "Must pass virtualenv_name and connection_file" 1>&2
        exit 1
fi
if ! command -v workon; then
        source "$(command -v virtualenvwrapper.sh)"
fi
workon "$1"
python -c 'import pkgutil, sys; exit(0 if pkgutil.find_loader(sys.argv[1]) is not None else 1)' ipykernel || pip install ipykernel
python -m ipykernel_launcher -f "$2"
2 Likes
4

hi @jonburdo, thanks for the detailed examples! Unfortunately, it doesn’t solve the problem. I’m aware of the standard way of registering kernels. This is what I also describe in my initial post after the words:

You’re absolutely right, in this case it is possible to interrupt the kernel. But the problem with this approach is that it doesn’t actually use the correct environment. It leads to numerous problems. I mentioned one of them in my initial post as well, it was about the jupyter notebook issue #4569. Discourse didn’t allow me post the link to this issue as I already had two other links, but I can do it here: Fails to load dll when in notebook, but not in ipython · Issue #4569 · jupyter/notebook · GitHub

Regarding the custom kernel startup commands, it seems that it is what nb_conda_kernels does. It properly activates the correct environment, but it turns out that it leads to problems with process inheritance, which makes interrupting the kernel impossible. See the related link in my second post in this thread. Based on that, I have also updated the issue description in the nb_conda_kernels repo that I mentioned in my initial post.

5

Thanks for the link and for clarifying that. Figured sharing what I’d gotten to work was at least worth a shot. Looking at the code for nb_conda_kernels, the amount of custom logic in there and also the fact that it’s in written in python both make me a little uneasy, and I’m not surprised there are issues with process inheritence. I think figuring out what command you’d run a shell and sticking those in a bash (or sh, zsh, etc) script will be your best bet. I also wonder if there’s something specific to Windows going on. These things often seem to work a bit differently between Windows and unix-like systems. It’d be interesting to see if it works any better for you on Windows subsystem for linux - might not be worth the hassle though.

In case it helps, I was able to get a kernel working with a bash script for conda containing just these two lines

conda activate "$1"
python -m ipykernel_launcher -f "$2"

preceded by the result of conda init. In my case, copying and pasting what that command appended to my ~/.bashrc with those lines resulted in:

#!/bin/bash
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/opt/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/opt/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/opt/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate "$1"
python -m ipykernel_launcher -f "$2"

Then I called this script from kernel.json with the env-name and {connection_file} as I mentioned in my last answer.

I also got it working by just putting everything in the kernel.json file. I first hardcoded env-nameand {connection_file} in place of "$1" and "$2". Then to get the script in json format, I just pasted the script into a multiline string in a python shell, storing it in a variable, a and ran:

import json
print(json.dumps(a))

and finally passed the result to bash in kernel.json:

{
 "argv": [
  "bash",
  "-c",
  "# >>> conda initialize >>>\n# !! Contents within this block are managed by 'conda init' !!\n__conda_setup=\"$('/opt/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)\"\nif [ $? -eq 0 ]; then\n    eval \"$__conda_setup\"\nelse\n    if [ -f \"/opt/miniconda3/etc/profile.d/conda.sh\" ]; then\n        . \"/opt/miniconda3/etc/profile.d/conda.sh\"\n    else\n        export PATH=\"/opt/miniconda3/bin:$PATH\"\n    fi\nfi\nunset __conda_setup\n# <<< conda initialize <<<\nconda activate kernelenv\npython -m ipykernel_launcher -f {connection_file}"
 ],
 "display_name": "Python (kernelenv)",
 "language": "python"
}

If that doesn’t work, I hope you find something that does. Kernel interrupt is kind of a nice function to have!