Python kernel that picks up environment from directory

How to best configure different kernels with different python virtualenvs (or similar) in different projects and directories?

Instead of installing a separate kernel per virtualenv, I want to install just one versatile “meta-kernel” or “kernel trampoline” that launches the relevant python kernel for the directory that a notebook belongs to. It is conceptually simpler, and it also should work a lot better when working in a team and sharing code.

I’m working on a prototype for that, that will use Rye to manage python versions, dependencies and virtualenv per jupyter(lab) project.

I was inspired by poetry-kernel which is based on the same concept - it’s the same idea, except they use Poetry as the manager. Do you all know other similar projects like this? Any more established ones?

My (extremely short) code is here: GitHub - bluss/ryeish-kernel: Python Jupyter kernel using Rye for reproducible notebooks and it’s a usable proof of concept. (It’s using Rye which itself is up and coming and not really ready.) There’s a directory called example there in the source code, that maybe can help explain how you would use it.

5 Likes

Does anybody know about other project that already do something like this? Like, activate a local venv per path and use kernel & dependencies from that path? Or similar. I did expect it to already exist. Maybe in a version that’s quite general and can maybe autodetect some common ways these environments are named.

That’s cool! Not directly what you are asking for but JupyterLab Desktop remembers environment by writing it’s name into a file in .jupyter folder locally. This is however used on per server not per kernel basis.

I wonder if an env manager-agnostic solution would have its place as a (semi-?)official tool in Jupyter ecosystem. By that I mean something that would recognise .python-version from pyenv, support conda/mamba/etc as well as poetry, Rye and whatever comes next.

1 Like

Hi, I am interested in trying out rye, and would need a tool similar to what poetry-kernel does for poetry. So I found ryeish-kernel! Seems like a nice tool but it wasn’t clear to me from the readme how to use it? Could yo add some (brief) instructions to it? Thanks a lot!

1 Like

I’ve started working on a kernel that delegates to any existing .venv in the notebook directory: GitHub - goerz/python-localvenv-kernel: A Jupyter kernel that delegates to ipykernel in the .venv environment of a project folder

The virtual environment in .venv can be generated with pip, poetry, or any other tool. As such, this project can replace poetry-kernel.

The documentation is still work in progress, but feel free to give it a try and report back any issues.

2 Likes

Nice goerz, that’s along exactly the same lines and a nice generalization, like you say, it will work with most any tool, the main idea is to follow the current notebook’s local venv and use it.

1 Like

My project has been updated and generalized - now supports Rye, Poetry, Hatch, Pdm and custom configurations. Name updated from ryeish-kernel to pyproject-local-kernel and published on PyPI.

I would be delighted if this happened - and if people close to the jupyter project had some interest and wanted to guide this project closer to the jupyter community norms.
And for that matter, shared maintenance is always welcome.

The new version has a more clear readme, and there’s an examples directory.

I have one disclaimer - I haven’t tested the package on Windows, so there might be small problems there, and bug reports would be welcome.

I have continued on my project since it’s slightly different, and I’ve been using it all along, it says “PoC” but it has been a working PoC. I prefer pyproject-local-kernel’s fallback behaviour - it starts a fallback kernel when there is a problem. It allows me to use jupyter’s interface to setup a new pyproject for a new group of notebooks from scratch.

2 Likes

Adding to the list of similar projects, I created GitHub - renan-r-santos/pixi-kernel: Python Jupyter kernel using Pixi for reproducible notebooks.
By the way, take a look at Pixi. It is a really cool tool for managing and locking Conda and PyPI packages living in the same virtual environment.

2 Likes