Jupyterhub with Docker Spawner

JupyterHub
1

I have a public facing Linux server with JupyterHub running on it’s own, not as a container. I’m trying to use the docker spawner to serve up Data Science notebook from quay.io (As you will see I have several docker images, none of which work but right now I’m only concerned about the data science image, I need to update the other ones for Jupyterhub5).

There are so many possible options I can’t decipher which ones need to be set and with which values. My server has 2 IPs, one for docker0 (a 172. IP) and it’s regular IP. When I try to spawn the notebook I get( I added \ before http so discourse doesn’t linkify it by default)

Server at \http://127.0.0.1:32780 didn’t respond in 30 seconds

I assume that one of the settings is trying to use localhost instead of some other address but I don’t know which one and I don’t know whether to use the 172 IP or the server’s IP. Here is what I have in my config (What I’m posting is my settings only, anything not posted is using the default value) Thank you for any help

c.JupyterHub.hub_connect_ip = socket.gethostname()
c.JupyterHub.hub_ip = ‘’
c.JupyterHub.spawner_class = ‘dockerspawner.DockerSpawner’
c.JupyterHub.ssl_cert = ‘/etc/ssl/certs/mycert.crt’
c.JupyterHub.ssl_key = ‘/etc/ssl/private/mykey.key’
c.Authenticator.allow_all = True
c.Spawner.ip = ‘0.0.0.0’
notebook_dir = os.environ.get(‘DOCKER_NOTEBOOK_DIR’) or ‘/home/jovyan/work’
c.Spawner.notebook_dir = notebook_dir
c.Spawner.port = 8888
def create_dir_hook(spawner):
“”" Create notebooks directory in users home directory if it doesn’t exist “”"
username = spawner.user.name # get the username
home_path = os.path.join(‘/home/’, username)
volume_path = os.path.join(home_path, ‘notebooks’)
if not os.path.exists(volume_path):
os.mkdir(volume_path)
shutil.chown(volume_path, user=username, group=‘users’)

c.Spawner.pre_spawn_hook = create_dir_hook
c.DockerSpawner.allowed_images = {
“Python3”: “jupyter/turtle”,
“C++”: “jupyter/cling”,
“Java”: “jupyter/java”,
“Datascience (Python/Julia/R)”: “\quay.io/jupyter/datascience-notebook”,
“Geoscience”: “jupyter/geo”
}
c.DockerSpawner.volumes = { ‘/home/{username}/notebooks’: notebook_dir }

This is what syslog spits out when I try to spawn
Jan 7 14:47:37 jupyter jupyterhub[193568]: [I 2025-01-07 14:47:37.200 JupyterHub pages:397] aweaver1 is pending spawn
Jan 7 14:47:37 jupyter jupyterhub[193568]: [I 2025-01-07 14:47:37.201 JupyterHub log:192] 200 GET /hub/spawn-pending/aweaver1?_xsrf=[secret] (aweaver1@::ffff:10.0.134.36) 10.49ms
Jan 7 14:47:37 jupyter ntpd[930]: Listen normally on 55 docker0 172.17.0.1:123
Jan 7 14:47:37 jupyter ntpd[930]: Listen normally on 56 docker0 [fe80::42:92ff:fec3:5586%3]:123
Jan 7 14:47:37 jupyter ntpd[930]: bind(27) AF_INET6 fe80::3407:9dff:fe71:f415%31#123 flags 0x11 failed: Cannot assign requested address
Jan 7 14:47:37 jupyter ntpd[930]: unable to create socket on veth005ad69 (57) for fe80::3407:9dff:fe71:f415%31#123
Jan 7 14:47:37 jupyter ntpd[930]: failed to init interface for address fe80::3407:9dff:fe71:f415%31
Jan 7 14:47:37 jupyter ntpd[930]: new interface(s) found: waking up resolver
Jan 7 14:47:37 jupyter avahi-daemon[830]: Joining mDNS multicast group on interface veth005ad69.IPv6 with address fe80::3407:9dff:fe71:f415.
Jan 7 14:47:37 jupyter avahi-daemon[830]: New relevant interface veth005ad69.IPv6 for mDNS.
Jan 7 14:47:37 jupyter avahi-daemon[830]: Registering new address record for fe80::3407:9dff:fe71:f415 on veth005ad69.*.
Jan 7 14:47:39 jupyter ntpd[930]: Listen normally on 58 veth005ad69 [fe80::3407:9dff:fe71:f415%31]:123
Jan 7 14:47:39 jupyter ntpd[930]: new interface(s) found: waking up resolver
Jan 7 14:47:46 jupyter jupyterhub[193568]: [W 2025-01-07 14:47:46.149 JupyterHub base:1258] User aweaver1 is slow to become responsive (timeout=10)
Jan 7 14:47:51 jupyter kernel: [95665.396055] audit: type=1400 audit(1736279271.589:3023): apparmor=“ALLOWED” operation=“open” profile=“/usr/sbin/sssd” name=“/sys/devices/virtual/net/veth005ad69/type” pid=22736 comm=“sssd” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Jan 7 14:48:08 jupyter jupyterhub[193568]: [W 2025-01-07 14:48:08.157 JupyterHub user:1055] aweaver1’s server never showed up at \http://127.0.0.1:32781/user/aweaver1/ after 30 seconds. Giving up.

2

Remove these two config params. Its Hub’s responsibility to find a free port and bind the single user server to that port. I think part of your problem might be hard coding the port value.

If it still does not work, enable debug logging on JupyterHub and share more logs. Also try to get the logs of docker containers where the single user servers have been launched.

3

In addition to the above suggestions can you also replace

with an IP in case the result of socket.gethostname() is not resolvable from inside a container.

1 Like
4

I removed
c.Spawner.ip = ‘0.0.0.0’
c.Spawner.port = 8888

which did not resolve my problem. I then set the hub_connect_ip as suggest by manic which did resolve the issue. Importantly I set it to the IP of the docker0 network and that resolved my issue. To be clear I also left Spawner.ip and Spawner.port as the default values and not the values I manually set above

1 Like