Kernel Packaging and Distribution

General
1

I’ve recently been struggling trying to help someone run some notebooks against a Python kernel with a range of Python package dependencies.

They’re on a Windows box — I’m not — and have had a range of issues:

  • repo2docker didn’t work ( (return os.geteuid() AttributeError: module 'os' has no attribute 'geteuid'?);
  • Docker didn’t work using a container I prebuilt (how do you reliably mount the current directory against a Docker volume?!);
  • installing packages didn’t work / couldn’t be found (I have no idea what environmental catastrophe was going on!).

There was a preference for not working on a public machine (so no Binderhub and uploading notebooks there) and we don’t have an internal JupyterHub or BinderHub server.

Which got me thinking. Is there a way of simply packaging a Python kernel with custom packages preinstalled so that it can be shared?

For example, installing the kernel might do something like:

  • make sure env support is available;
  • create a new py env;
  • install the required packages;
  • register that env as a new, custom-named Jupyter kernel in an existing Jupyter supporting environment.

Or perhaps rather than something like having a binder folder in a directory and running repo2docker, having a kernel directory containing a requirements.txt and running jupyter addkernel --name myPykernel . ?

1 Like
2

“Packaging” is a pretty broad problem! While we’d love to solve this for the general case, Jupyter serves many languages and platforms, and each of them (and their combinations) have their own idiosyncrasies.

For the julia-python-r-and-friends-on-windows case, I personally (and Jupyter, for that matter) can’t really recommend anything more heartily than conda (with conda-forge), and an environment.yml.

# environment.yml
name: my-project-env
channels:
  - conda-forge
  - defaults
dependencies:
  - python =3.7  # pip cannot do this for you
  - ipykernel
  - some-other-package
  - pip:
    - some-pip-package-not-in-conda

and run from (just about any) terminal…

# install-or-update-kernel.bat
conda env update --prefix ./a-new-env
jupyter kernelspec install --sys-prefix --name a-new-kernel ./a-new-env/share/jupyter/kernels/python3

… or you can check these into version control, put them on a shared drive, whatever. Just don’t actually check in your environment!

If you need to go really offline, you can put miniconda, and the downloaded results of conda list --explicit on a thumb drive, and reconstitute your environment without any internet at all.

For more advanced use cases, you can try anaconda-project which goes one further and lets you lock/unlock cross-platform environments.

3

Yeah, I get the problem re: generalising.

I guess in the py case I could script something like https://janakiev.com/blog/jupyter-virtual-envs/ (which was my immediate concern) but for other kernels they’d need an equivalent environment mechanism.

4

Actually, https://containds.com/ (I’m guessing a fork of Kitematic?) looks like it could be handy…

Simple reusable Jupyter environments

Use Jupyter through Docker without learning computer science

A simple user interface allowing you to easily run Jupyter Lab or Notebooks through virtual environments on your Windows or Mac computer

I should have thought of using Kitematic…

5

Do you think Kitematic or containds could help solve the problems associated with docker on windows?

6

@betatim If Kitematic / containds work properly on WIndows, then yes?! Certainly, the directory mapping from host into a container is handled via file selection dialogues, so hopefully any case issues in the path names are robustly handled by the app (which they donlt appear to be on the docker cli?)

7

Sounds like they do some kind of magic on top of docker :-/ Need to find out what that is :slight_smile: