JupyterHub with DockerSpawner: Failed to connect to Hub API

JupyterHub
1

Good afternoon!

I’ve been trying to setup a VM with a public-facing JupyterHub that provides users with Notebooks within Docker containers.

I’ve followed https://jupyterhub.readthedocs.io/en/stable/installation-guide.html which has worked absolutely fine. It resulted in a working core with GitHub OAuth and SSL encryption through nginx. But upon trying to incorporate DockerSpawner all attempts to create a server now yield 500 errors as below:

Failed to connect to Hub API at ‘http:// 172.17.0.1:8081/jupyter/hub/api’. Is the Hub accessible at this URL (from host: b1845483a200)?

I have been trying to debug this for at least a week now, and have read every post/resource mentioning anything similar that I can find with no luck. My relevant files/logs are below (hopefully I didn’t go overboard in this regard), please let me know if you have any suggestions or require further details. Thank you!

(note the random url spaces are due to the link limit)

jupyterhub_config.py

c.JupyterHub.bind_url = ‘https:// 127.0.0.1:8000/jupyter’
from dockerspawner import DockerSpawner
c.JupyterHub.spawner_class = DockerSpawner
import netifaces
docker0 = netifaces.ifaddresses( ‘docker0’ )
docker0_ipv4 = docker0[netifaces.AF_INET][0]
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.DockerSpawner.hub_ip_connect = docker0_ipv4[ ‘addr’ ]
c.Spawner.debug = True
c.LocalProcessSpawner.debug = True
c.JupyterHub.log_level = ‘DEBUG’

docker logs (repetition trimmed)

Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --debug --hub-api-url=http://172.17.0.1:8081/jupyter/hub/api
…
[I 2021-03-17 14:40:40.881 SingleUserNotebookApp mixins:557] Starting jupyterhub-singleuser server version 1.3.0
[E 2021-03-17 14:40:40.884 SingleUserNotebookApp mixins:430] Failed to connect to my Hub at http:// 172.17.0.1:8081/jupyter/hub/api (attempt 1/5). Is it running?
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py”, line 428, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
…
[E 2021-03-17 14:40:42.107 SingleUserNotebookApp auth:333] Error connecting to http:// 172.17.0.1:8081/jupyter/hub/api: HTTPConnectionPool(host=‘172.17.0.1’, port=8081): Max retries exceeded with url: /jupyter/hub/api/oauth2/token (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f1fd92394c0>: Failed to establish a new connection: [Errno 111] Connection refused’))

JupyterHub Logs (tried to trim it down)

[I 2021-03-17 16:47:57.682 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-03-17 16:47:57.682 JupyterHub app:2379] Using Authenticator: oauthenticator.github.LocalGitHubOAuthenticator-0.13.0
[I 2021-03-17 16:47:57.683 JupyterHub app:2379] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
[I 2021-03-17 16:47:57.683 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
…
[I 2021-03-17 16:47:57.755 JupyterHub proxy:666] Starting proxy @ http:// 127.0.0.1:8000/jupyter
[D 2021-03-17 16:47:57.755 JupyterHub proxy:667] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘127.0.0.1’, ‘–port’, ‘8000’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http:// SERVER.helsinki. fi:8081/jupyter/hub/error’]
…
16:47:58.376 [ConfigProxy] info: Proxying http:// 127.0.0.1:8000 to (no default)
16:47:58.379 [ConfigProxy] info: Proxy API at http:// 127.0.0.1:8001/api/routes
[D 2021-03-17 16:47:58.737 JupyterHub proxy:702] Proxy started and appears to be up
[D 2021-03-17 16:47:58.745 JupyterHub proxy:795] Proxy: Fetching GET http:// 127.0.0.1:8001/api/routes
[I 2021-03-17 16:47:58.760 JupyterHub app:2664] Hub API listening on http:// 0.0.0.0:8081/jupyter/hub/
[I 2021-03-17 16:47:58.761 JupyterHub app:2666] Private Hub API connect url http:// SERVER.helsinki. fi:8081/jupyter/hub/
[D 2021-03-17 16:47:58.761 JupyterHub proxy:314] Fetching routes to check
[D 2021-03-17 16:47:58.761 JupyterHub proxy:795] Proxy: Fetching GET http:// 127.0.0.1:8001/api/routes
16:47:58.763 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-17 16:47:58.765 JupyterHub proxy:319] Checking routes
[I 2021-03-17 16:47:58.766 JupyterHub proxy:399] Adding default route for Hub: /jupyter/ => http:// SERVER.helsinki. fi:8081
[D 2021-03-17 16:47:58.766 JupyterHub proxy:795] Proxy: Fetching POST http:// 127.0.0.1:8001/api/routes/jupyter
16:47:58.768 [ConfigProxy] info: 200 GET /api/routes
16:47:58.770 [ConfigProxy] info: Adding route /jupyter → http:// SERVER.helsinki. fi:8081
16:47:58.771 [ConfigProxy] info: Route added /jupyter → http:// SERVER.helsinki. fi:8081
[I 2021-03-17 16:47:58.772 JupyterHub app:2739] JupyterHub is now running at http:// 127.0.0.1:8000/jupyter
[D 2021-03-17 16:47:58.773 JupyterHub app:2342] It took 1.100 seconds for the Hub to start
16:47:58.774 [ConfigProxy] info: 201 POST /api/routes/jupyter
…
[D 2021-03-17 16:48:08.816 JupyterHub user:288] Creating <class ‘dockerspawner.dockerspawner.DockerSpawner’> for djprice:
…
[D 2021-03-17 16:48:08.916 JupyterHub dockerspawner:873] Starting host with config: {‘binds’: {}, ‘links’: {}, ‘port_bindings’: {8888: (‘127.0.0.1’,)}, ‘network_mode’: ‘bridge’}
[I 2021-03-17 16:48:08.979 JupyterHub dockerspawner:985] Created container jupyter-djprice (id: 2b021c7) from image jupyterhub/singleuser:1.3
…
[I 2021-03-17 16:48:09.938 JupyterHub pages:402] djprice is pending spawn
[I 2021-03-17 16:48:09.994 JupyterHub log:181] 200 GET /jupyter/hub/spawn-pending/djprice (djprice@SERVER-IP) 63.00ms
[D 2021-03-17 16:48:12.732 JupyterHub utils:224] Server at http:// 127.0.0.1:49158/jupyter/user/djprice/ responded with 302
[D 2021-03-17 16:48:12.732 JupyterHub _version:76] jupyterhub and jupyterhub-singleuser both on version 1.3.0
[I 2021-03-17 16:48:12.732 JupyterHub base:904] User djprice took 3.878 seconds to start
[I 2021-03-17 16:48:12.733 JupyterHub proxy:257] Adding user djprice to proxy /jupyter/user/djprice/ => http:// 127.0.0.1:49158
[D 2021-03-17 16:48:12.733 JupyterHub proxy:795] Proxy: Fetching POST http:// 127.0.0.1:8001/api/routes/jupyter/user/djprice
16:48:12.736 [ConfigProxy] info: Adding route /jupyter/user/djprice → http:// 127.0.0.1:49158
16:48:12.737 [ConfigProxy] info: Route added /jupyter/user/djprice → http:// 127.0.0.1:49158
16:48:12.737 [ConfigProxy] info: 201 POST /api/routes/jupyter/user/djprice
[I 2021-03-17 16:48:12.739 JupyterHub users:664] Server djprice is ready
[I 2021-03-17 16:48:12.741 JupyterHub log:181] 200 GET /jupyter/hub/api/users/djprice/server/progress (djprice@SERVER-IP) 2593.07ms
[I 2021-03-17 16:48:12.812 JupyterHub log:181] 302 GET /jupyter/hub/spawn-pending/djprice → /jupyter/user/djprice/ (djprice@SERVER-IP) 4.34ms
[D 2021-03-17 16:48:12.929 JupyterHub provider:409] Validating client id jupyterhub-user-djprice
[D 2021-03-17 16:48:12.930 JupyterHub provider:484] validate_redirect_uri: client_id=jupyterhub-user-djprice, redirect_uri=/jupyter/user/djprice/oauth_callback
[D 2021-03-17 16:48:12.932 JupyterHub auth:244] Skipping oauth confirmation for <User(djprice 1/1 running)> accessing Server at /jupyter/user/djprice/
[D 2021-03-17 16:48:12.933 JupyterHub provider:409] Validating client id jupyterhub-user-djprice
[D 2021-03-17 16:48:12.934 JupyterHub provider:484] validate_redirect_uri: client_id=jupyterhub-user-djprice, redirect_uri=/jupyter/user/djprice/oauth_callback
[D 2021-03-17 16:48:12.935 JupyterHub provider:231] Saving authorization code jupyterhub-user-djprice, OxC…, (), {}
[I 2021-03-17 16:48:12.954 JupyterHub log:181] 302 GET /jupyter/hub/api/oauth2/authorize?client_id=jupyterhub-user-djprice&redirect_uri=%2Fjupyter%2Fuser%2Fdjprice%2Foauth_callback&response_type=code&state=[secret] → /jupyter/user/djprice/oauth_callback?code=[secret]&state=[secret] (djprice@SERVER-IP) 26.87ms
[D 2021-03-17 16:48:39.447 JupyterHub dockerspawner:777] Getting container ‘jupyter-djprice’
[D 2021-03-17 16:48:39.451 JupyterHub dockerspawner:761] Container 2b021c7 status: {‘Dead’: False,
‘Error’: ‘’,
‘ExitCode’: 0,
‘FinishedAt’: ‘0001-01-01T00:00:00Z’,
‘OOMKilled’: False,
‘Paused’: False,
‘Pid’: 8919,
‘Restarting’: False,
‘Running’: True,
‘StartedAt’: ‘2021-03-17T14:48:09.429332729Z’,
‘Status’: ‘running’}
[W 2021-03-17 16:49:09.454 JupyterHub base:1067] User djprice server stopped, with exit code: ExitCode=1, Error=‘’, FinishedAt=2021-03-17T14:48:57.811885955Z
[I 2021-03-17 16:49:09.454 JupyterHub proxy:281] Removing user djprice from proxy (/jupyter/user/djprice/)
[D 2021-03-17 16:49:09.454 JupyterHub proxy:795] Proxy: Fetching DELETE http:// 127.0.0.1:8001/api/routes/jupyter/user/djprice
16:49:09.457 [ConfigProxy] info: Removing route /jupyter/user/djprice
16:49:09.457 [ConfigProxy] info: 204 DELETE /api/routes/jupyter/user/djprice
[D 2021-03-17 16:49:09.458 JupyterHub user:790] Stopping djprice

Docker Inspect

“Ports”: { “8888/tcp”: [ {
“HostIp”: “127.0.0.1”,
“HostPort”: “49159” } ] },
“Gateway”: “172.17.0.1”,
“IPAddress”: “172.17.0.3”,

The docker container spawned by the Hub dies too fast for testing, but I did manually run the jupyterhub/singleuser image and try to connect to the Hub from inside in a variety of ways with no luck beyond this one example:

$ sudo docker exec thirsty_kowalevski wget -O- https:// 172.17.0.1/jupyter/hub/api --no-check-certificate
–2021-03-12 14:51:14-- https:// 172.17.0.1/jupyter/hub/api
Connecting to 172.17.0.1:443… connected.
WARNING: cannot verify 172.17.0.1’s certificate, issued by ‘CN=GEANT OV RSA CA 4,O=GEANT Vereniging,C=NL’:
Unable to locally verify the issuer’s authority.
WARNING: certificate common name ‘SERVER.helsinki. fi’ doesn’t match requested host name ‘172.17.0.1’.
HTTP request sent, awaiting response… 200 OK
Length: 20 [application/json]
Saving to: ‘STDOUT’
0K 100% 3.89M=0s
2021-03-12 14:51:14 (3.89 MB/s) - written to stdout [20/20]

But connecting from the server to the container works using the forwarded port, albeit as an isolated test not connected to the Hub. It feels like I’m missing something regarding the communication, maybe with configurable-http-proxy and docker, but can’t quite track it down. Thanks again for any input.

2

I think what’s going on is that by binding to the loopback address you’re causing the spawned container to try to ask to itself for configuration and control plane information.

Since this is three days old, I sort of assume you’ve figured out the networking end of things.

3

Thank you for your response! Unfortunately the issue is still ongoing.

Do you know how I would determine what IP to place there instead?

The alternative I tried threw a new error, as did attempting 10.0.1.4:

00:15:48.890 [ConfigProxy] error: Uncaught Exception: listen EADDRNOTAVAIL: address not available xx.yy.zz.250:8000
00:15:48.890 [ConfigProxy] error: Error: listen EADDRNOTAVAIL: address not available xx.yy.zz.250:8000
at Server.setupListenHandle [as _listen2] (net.js:1263:19)
at listenInCluster (net.js:1328:12)
at doListen (net.js:1461:7)
at process._tickCallback (internal/process/next_tick.js:63:19)
at Function.Module.runMain (internal/modules/cjs/loader.js:834:11)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)

From ‘ip addr’ I thought that I could just change the 4th digit and use that but I’m not really sure what to set.

2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet xx.yy.zz.165/25 brd xx.yy.zz.255 scope global ens192
valid_lft forever preferred_lft forever

4

Connecting from containers to the host can be tricky. Binding on docker0 has usually worked for me, but for some reason the docker network configuration may not allow connections in that direction.

Honestly, one option is to just try all the ipv4 networks you have and see if one works!

for iface in netifaces.interfaces():
    ipv4 = netifaces.ifaddresses(iface).get(netifaces.AF_INET)
    if ipv4:
        print(iface, ipv4)

Another option is to put the hub itself in a container, as in this example. Networking between containers is often easier to get a handle on than connections from the containers to the host. Then you only have the one inbound connection from nginx to the hub container. (you can also go further and put the proxy in its own container, since it often has a logically different lifecycle for restarts than the hub).

5

Thank you for the advice!

It seems to have yielded the same outcome, but I’d just like to check I implemented it correctly before I go down the JupyterHub in a container route because I briefly tried it (GitHub - jupyterhub/jupyterhub-deploy-docker: Reference deployment of JupyterHub with docker) after my initial post but also ended up with networking issues there.

c.JupyterHub.bind_url = ‘http://172.17.0.1:8000/jupyter’
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.hub_ip_connect = docker0_ipv4[ ‘addr’ ]

Where 172.17.0.1 = docker0 gateway IP.

I thought nginx doesn’t affect this because it’s internal communication but I also have this set to host the actual website, otherwise it just listens on 443 and redirects 80 to https.

JupyterHub Server

location /jupyter/ {
proxy_pass http://172.17.0.1:8000;

6

I’d just like to note that I got GitHub - jupyterhub/jupyterhub-deploy-docker: Reference deployment of JupyterHub with docker to run successfully, so there is no longer a pressing need for a solution to this problem. But I’d like to leave this open just in case someone comes along and knows the answer out of curiosity and because it seems easier to manage the Hub without using a container - but maybe I’ll feel differently about that in a week or two.