Apologies for the link formatting, I am only allowed to put 2 links in my post.
I am interested in a setup where I use Jupyter kernels on a remote/headless system (or in a Docker container, or both)). I would like to interact with these kernels using a local-based client such as Jupyter Lab, Jupyter Console, or Jupyter-compatible code editor like VS Code or Dataspell.
I understand that Jupyterlab can be hosted on the same remote system, but the other clients cannot be hosted. I would prefer to use my own Jupyterlab instance anyway, with all of the plugins and settings that I like to use.
Ideally, I would also be able to disconnect and reconnect at will, so that my connection to the kernel can be restored if I need to turn off my PC, or if my network connection drops.
What options currently exist to address this problem? What options are in active development that are expected to become available in the near future?
The easiest solution to this problem would be the ability to start a kernel using static, user-provided ports, instead of randomly selecting ports from a range. That way I can control exactly what ports I need to make available through SSH tunnels, and I can control the name of the connection file so that it can be fetched automatically. Is this currently possible with Jupyter?
I am aware of the “provisioner” system (jupyter-client.readthedocs /en/stable/provisioning.html) that is in development, but it is still in the proof-of-concept development stage. Can I solve this problem with the provisioner system?
Even if I have to do some hands-on
yak-shaving hacking, I am willing to put in the effort if it will benefit the community to have a simple provisioner implementation. I am sure I’m not the only one with this use case, so hopefully any work I put into this area would result in benefit for many people in the community.
I am also aware of Jupyter Kernel Gateway (jupyter-kernel-gateway.readthedocs). But it looks like a standard Jupyter protocol client is not able to interact with a Gateway-provided kernel, because the protocol is not the standard Jupyter protocol. Jupyter Notebook specifically has special support for Gateway connections (
jupyter notebook --gateway-url=...), but that isn’t the client I want to use.
As a last resort, I can write a script to start a kernel over SSH, parse the name of the connection file from the logs (ugh!), and then copy the connection file to the local system. But even then, Jupyter Lab does not yet support connecting to an existing kernel. This seems to be what the 5-years-unmaintained
pypi:project/remote_ikernel) tool attempted to do, and perhaps the even-older
github:korniichuk/rk) tool. I remember trying to work with a setup like this around 2018, and the experience was generally so poor that I gave up and used Jupyter Notebook on the remote system.