JupyterHub from jupyter/*-notebook spawns JupyterLab on wrong port 8888

JupyterHub
,
1

I have recently put to use a JupyterLab instance running in a docker container with fair success. The docker image is a mixture of the jupyter/*-notebook with a great selection of packages.

After that I considered giving individual users a distinct session to avoid collisions, by running jupyterhub. I tried a simple setup with the jupyterhub/jupyterhub image, configured a couple of settings for SSL, and seemed working nice with a notebook.

The latest step I wanted to do was to launch jupyterhub from my jupyter/*-notebook image, since it also bundles JupyterHub. It all seems fine, but when it asks the spawner jupyter-labhub or jupyterhub-singleuser (tried both), it requests a server to spawn at 127.0.0.1 and random port. However, the server is instantiated at 0.0.0.0 port 8888, sort of a default. Then, the spawner never reaches the http://127.0.0.1:xxxxx/user/<user>/lab.

Digging a bit into the spawning process, I found that jupyterhub/singleuser/mixins.py is called on: initialize() and start(). By putting a couple of print()s, I saw that the call to initialize has correctly set the port at which the spawner is expecting the server. But when the call to start() arrives immediately after, the port is set to 8888. This comes as a surprise to me, since traitlets/config/application.py launch_instance() calls those two functions consecutively in the same app object.

Some configuration:

c.Spawner.cmd = ['jupyter-labhub']
c.JupyterHub.ssl_cert = '/etc/jupyterhub/uuuu.com.cer'
c.JupyterHub.ssl_key = '/etc/jupyterhub/uuuu.com.key'
c.JupyterHub.bind_url = 'https://jupyterhub.uuuu.com:443'
c.Authenticator.admin_users = {'uuuu'}

Some log:

 +-+_+-+ Spawner: start(), ip=127.0.0.1, port=36063
[I 2022-03-04 14:18:42.977 JupyterHub spawner:1527] Spawning jupyter-labhub
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
Patching auth into notebook.base.handlers.IPythonHandler(notebook.base.handlers.AuthenticatedHandler) -> IPythonHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, notebook.base.handlers.AuthenticatedHandler)
 ++++ Starting application with:

None
 )) Application == app type=<class 'jupyterlab.labhubapp.SingleUserLabApp'>
 )) Application == cls type=<class 'traitlets.traitlets.MetaHasTraits'>
+++ singleUserNotebookApp init, ip=127.0.0.1, port=36063
[I 2022-03-04 14:18:43.772 SingleUserLabApp mixins:615] Starting jupyterhub single-user server version 2.1.1
[I 2022-03-04 14:18:43.773 SingleUserLabApp mixins:629] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.2.9
[I 2022-03-04 14:18:43.773 SingleUserLabApp mixins:629] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.13.5
[W 2022-03-04 14:18:43.773 SingleUserLabApp mixins:638]  + Initialize: ip=127.0.0.1, port=36063
 == initialize(): type of self: <class 'jupyterlab.labhubapp.SingleUserLabApp'>
 == initialize(): type of super: <class 'super'>
[W 2022-03-04 14:18:43.787 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[W 2022-03-04 14:18:43.802 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2022-03-04 14:18:43.802 SingleUserLabApp manager:345] jupyterlab | extension was successfully linked.
[W 2022-03-04 14:18:43.805 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-04 14:18:43.805 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-04 14:18:43.805 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-04 14:18:43.811 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2022-03-04 14:18:43.881 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/uuuu (uuuu@172.25.0.1) 1005.19ms
[I 2022-03-04 14:18:43.963 JupyterHub pages:405] uuuu is pending spawn
[I 2022-03-04 14:18:43.972 SingleUserLabApp manager:345] nbclassic | extension was successfully linked.
[I 2022-03-04 14:18:43.985 JupyterHub log:189] 200 GET /hub/spawn-pending/uuuu (uuuu@172.25.0.1) 24.60ms
[I 2022-03-04 14:18:43.994 SingleUserLabApp manager:367] nbclassic | extension was successfully loaded.
[I 2022-03-04 14:18:43.995 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2022-03-04 14:18:43.995 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2022-03-04 14:18:43.999 SingleUserLabApp manager:367] jupyterlab | extension was successfully loaded.
+++ singleUserNotebookApp init next
[I 2022-03-04 14:18:43.999 SingleUserLabApp mixins:646] Starting jupyterhub-singleuser server version 2.1.1
[W 2022-03-04 14:18:43.999 SingleUserLabApp mixins:647]  + Start: ip=0.0.0.0, port=8888
 == start(): type of self: <class 'jupyterlab.labhubapp.SingleUserLabApp'>
 == start(): type of super: <class 'super'>
[I 2022-03-04 14:18:44.001 JupyterHub log:189] 200 GET /hub/api (@127.0.0.1) 0.53ms
[I 2022-03-04 14:18:44.002 SingleUserLabApp serverapp:2607] Serving notebooks from local directory: /home/uuuu
[I 2022-03-04 14:18:44.002 SingleUserLabApp serverapp:2607] Jupyter Server 1.13.5 is running at:
[I 2022-03-04 14:18:44.002 SingleUserLabApp serverapp:2607] http://jupyterhub:8888/user/uuuu/lab
[I 2022-03-04 14:18:44.002 SingleUserLabApp serverapp:2607]  or http://127.0.0.1:8888/user/uuuu/lab
[I 2022-03-04 14:18:44.002 SingleUserLabApp serverapp:2608] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2022-03-04 14:18:44.005 SingleUserLabApp mixins:597] Updating Hub with activity every 300 seconds
[I 2022-03-04 14:18:44.033 JupyterHub log:189] 200 POST /hub/api/users/uuuu/activity (uuuu@127.0.0.1) 27.72ms
[W 2022-03-04 14:18:52.881 JupyterHub base:1076] User uuuu is slow to become responsive (timeout=10)
[W 2022-03-04 14:19:12.722 JupyterHub user:812] uuuu's server never showed up at http://127.0.0.1:36063/user/uuuu/ after 30 seconds. Giving up.

Did I skip something I don’t understand from my configuration?

2

Hi! Please can you show us your full configuration with secrets redacted, and tell us exactly how you launched all your docker image(s) including version information? Ideally please provide enough information for someone to reproduce your setup.

Thanks!

3

All the configuration that I added is the file /etc/jupyterhub/jupyterhub_config.py, which contents are in the original post.

The run command was:

jupyterhub -f /etc/jupyterhub/jupyterhub_config.py

root@jupyterhub:/srv/jupyterhub# jupyterhub --version
2.1.1

The docker run command:

docker run --rm -t -v /path/to/etc/jupyterhub:/etc/jupyterhub -v /path/to/srv/jupyterhub:/srv/jupyterhub --hostname jupyterhub --workdir /srv/jupyterhub --user 0 jupyter-notebook jupyterhub -f /etc/jupyterhub/jupyterhub_config.py

where jupyter-notebook is based on jupyter/datascience-notebook from jupyter/docker-stacks.

4

A simpler way to reproduce, without the need for SSL keys or custom docker images, is:

mkdir -p /tmp/${USER}
chmod a+w /tmp/${USER}
docker run --rm -t -p 8000:8000 -v /tmp/${USER}:/home/${USER} -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -u 0 jupyter/datascience-notebook jupyterhub

Then login at http://localhost:8000 with your system credentials for the current user.
The error is the same as explained on the post.

5

JupyterHub is designed to support multiple users, and spawns a singleuser server for each user. Since you’re running JupyterHub inside Docker with the default LocalProcessSpawner each server will get a different random port.

If you want to spawn separate containers for each user then have a look at https://jupyterhub-dockerspawner.readthedocs.io/

6

Indeed, @manics , that is what I would expect, a different server from a different port. However, the server is spawned at port 8888. That is why I titled this thread “… on wrong port 8888”.

The question is, why is the user uuuu server spawning on port 8888, and not in random (e.g. 42513)?

7

The jupyter/* containers aren’t designed to run JupyterHub.

It looks like the configuration files in /etc/jupyter/jupyter_*.py force the port to 8888, so the port requested by JupyterHub is ignored. Deleting those files seems to work, but there may be other unforeseen problems related to the use of this image.

8

Thanks @manics, this really solved the configuration the way I was using it.

So I learned that the explicit configurations in the /etc/jupyter/jupyter_* files are applied after (overwrite) the arguments passed by the default LocalProcessSpawner. Then, replacing the default configuration files for configurations that don’t set explicitly c.ServerApp.ip and c.ServerApp.port solved my deployment.

I will let you know if I notice side effects when using jupyter/*-based docker images as a JupyterHub server.