Multiple jupyterhub servers per user ("named servers") does not work

Hello all,

we have a jupyterhub server running on an ubuntu headless server 18.0.4. LTS. We are using the docker-spawner of the type systemuser-spawner, in order to map in the user’s home dirs into the spawned container. Everything works fine so far.

Now we want to enable “named servers” in jupyterhub, so that every user can have more than one jupyter notebook spawned at the same time.
For this I added the line

c.JupyterHub.allow_named_servers = True

to the jupyterhub_config.py. After restaring the jupyterhub server this should be working, but isn’t-
I can see all the new mechanics of the activated feature “named servers”, and I can also add new servers to the list. Then I can spawn the first container, this works. As soon as I spawn the second container within another created server, it results in an error:

Error: 409 Client Error: Conflict (“Conflict. The container name “/jupyter-rschauf” is already in use by container “ab4c3a0d8f00”. You have to remove (or rename) that container to be able to reuse that name.”)

When I check in the shell of the server to see the info about the first container spawned, with docker ps after starting the first container I get:

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                       NAMES
ab4c3a0d8f00        hhn/0-gpu-tfnb:latest   "tini -g -- start-no…"   14 seconds ago      Up 10 seconds       127.0.0.1:32769->8888/tcp   jupyter-rschauf

After this error, also the first container is gone again.

You can see the name “jupyter-rschauf” which has been assigned to the container. The error message above shows, that when spawning the second container, it again tries to assign the same name, which already has been assignet to the first container. This is done by the singleuser-spawner.

Does this mean, multiple servers per user does not work with singleuser-spawner?
Or do I miss some configuration yet?

thanks and regards,
rschauf

Do you know if the docker spawner takes into account the name of the “named server” when naming the container it creates? From what you reported it looks like the answer is no. Not familiar with docker spawner but maybe it has an option to enabled “named server” support or it needs to be updated.

Naively I would have expected the containers to be named jupyter-rschau-<namedservernamehere> to make them unique.

Hi @betatim,

you are absolutely right, I have the same assumption - like you, that the systemuser-spawner we use in our jupyterhub server does not take the names of the new created servers into account. The correct way would be, that it would have to append the new servers name to the container name. Therefore the attempt of creation of the second container fails, as it tries to set it up with the same name of the server. But this is only an assumption, as I am - like you - no expert.

So if I for instance set up a 2nd server with the name “rs-2”, then the name of the container created for it should be:

jupyter-rschauf-rs-2

or something similar at least…
By the way, here is the version of my dockerspawner installation on the server:

pip show dockerspawner
Name: dockerspawner
Version: 0.11.1
Summary: Dockerspawner: A custom spawner for Jupyterhub.
Home-page: https://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /usr/local/lib/python3.6/dist-packages
Requires: docker, escapism, jupyterhub
Required-by:

And docker is the version:

docker --version
Docker version 19.03.2, build 6a30dfc

I think dockerspawner is the most recent version…
Maybe some other module, needed for running the jupyterhub is out of date?
Any ideas?

Thanks and regards,
rschauf

I think if you have the latest versions of the spawners you want to use and it doesn’t work it means no one has worked on adding the functionality. This means the way forward is to make or facilitate a PR that adds this functionality.

Hi @betatim,
hi all,

a few minutes ago I found the solution to this problem: I was missing a string template, that advises the systemuser-spawner how to assemble the container names:

c.SystemUserSpawner.name_template = '{prefix}-{username}-{servername}'

I added this to the jupyterhub_config.py, restarted the server and was able to spawn more than one container at the same time:

I created two more servers with my user, et voilà:

and when I do a dpocker ps, I can see all three recently created containers with different names, as expected:

linux:~$ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                       NAMES
792bce9a8ca9        hhn/1-gpu-tfnb:latest       "tini -g -- start-no…"   2 minutes ago       Up 2 minutes        127.0.0.1:32772->8888/tcp   jupyter-rschauf-rs-1
6127a4ab6ad1        hhn/0-gpu-tfnb:latest       "tini -g -- start-no…"   5 minutes ago       Up 5 minutes        127.0.0.1:32771->8888/tcp   jupyter-rschauf-rs-0
5bd4dee7fe80        jupyterhub/singleuser:1.0   "tini -g -- start-no…"   6 minutes ago       Up 6 minutes        127.0.0.1:32770->8888/tcp   jupyter-rschauf-

I found this solution in this post

@betatim: thanks a lot for all your hints anyway,
hope this helps someone else yet,
best regards,
rschauf

2 Likes

Hi @betatim,

btw, I run into the problem, as on the docu page of jupyterhub, sub-chapter for named servers, there is no hint, that you have to supply a meaningful name_template for the servers at all.

Here is the docu page of named servers.

Could someone who has acces please add a hint for the name_template which has to be used with all spawner types on this documentation page for jupyterhib/named servers? Maybe also, how such a string can look like in other cases?

In my case it was the dockerspawner type SingleUserSpawner, and it has to be configured the following way:

c.JupyterHub.allow_named_servers=True
c.SystemUserSpawner.name_template = '{prefix}-{username}-{servername}'

thanks and regards,
rschauf

I think the fastest way to get the docs updated is if you can make a PR that edits https://github.com/jupyterhub/jupyterhub/blob/89b0c421d52d823088b6eb42a232197919dbe00e/docs/source/reference/config-user-env.md in the way you just described.

Hi @betatim ,

yes, of course, you are right, … in the meantime to realize your suggestion I have sent already a pull request with my user rschauf to the project. Now the responsible must check if it suits their quality standards!
Thanks for this hint,
and again, thanks for your tips (they actually helped me to find the solution),

best regards,
rschauf

You are welcome!

Eine Hand waescht die andere :wink: