I am trying to create a Containerized Multi-User JupyterHub on ubuntu VMs with 1 master and 2 nodes setup in a docker-swarm environment.
The jupyterhub-singleuser notebook spawns perfectly on master (works).
But, it errors out when spawning on nodes.
jupyterhub_config.py:
c.JupyterHub.spawner_class = ‘dockerspawner.SwarmSpawner’
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.DockerSpawner.hub_ip_connect = ‘X.X.X.X’ (internal IP of master host)
##also tried c.SwarmSpawner.hub_ip_connect = ‘X.X.X.X’ (internal IP of master host) #same issue
Docker logs of container spawned on the node (terminates after sometime) :
Entered start.sh with args: jupyterhub-singleuser --ip=0.0.0.0 --hub-api-url=http://X.X.X.X:####/hub/api
Ensuring /home/sse is owned by 1000:100
Running as sse: jupyterhub-singleuser --ip=0.0.0.0 --hub-api-url=http://X.X.X.X:####/hub/api
[I 2022-09-28 14:32:29.393 SingleUserLabApp mixins:592] Updating Hub with activity every 300 seconds
[D 2022-09-28 14:32:29.393 SingleUserLabApp mixins:554] Notifying Hub of activity 2022-09-28T14:32:29.343918Z
[D 2022-09-28 14:32:41.902 SingleUserLabApp auth:672] No user identified
[I 2022-09-28 14:32:41.904 SingleUserLabApp log:186] 200 GET /user/username/api (@127.0.0.1) 3.75ms
[E 2022-09-28 14:32:49.401 SingleUserLabApp mixins:574] Error notifying Hub of activity
Traceback (most recent call last):
File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 572, in notify
await client.fetch(req)
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
[D 2022-09-28 14:32:49.734 SingleUserLabApp mixins:554] Notifying Hub of activity 2022-09-28T14:32:29.343918Z
[D 2022-09-28 14:32:57.064 SingleUserLabApp auth:672] No user identified
[I 2022-09-28 14:32:57.065 SingleUserLabApp log:186] 200 GET /user/username/api (@127.0.0.1) 2.27ms
[E 2022-09-28 14:33:09.749 SingleUserLabApp mixins:574] Error notifying Hub of activity
Traceback (most recent call last):
File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 572, in notify
await client.fetch(req)
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
[D 2022-09-28 14:33:11.183 SingleUserLabApp mixins:554] Notifying Hub of activity 2022-09-28T14:32:29.343918Z
[D 2022-09-28 14:33:12.223 SingleUserLabApp auth:672] No user identified
[I 2022-09-28 14:33:12.224 SingleUserLabApp log:186] 200 GET /user/username/api (@127.0.0.1) 2.09ms
[D 2022-09-28 14:33:27.383 SingleUserLabApp auth:672] No user identified
[I 2022-09-28 14:33:27.384 SingleUserLabApp log:186] 200 GET /user/username/api (@127.0.0.1) 2.07ms
[E 2022-09-28 14:33:31.188 SingleUserLabApp mixins:574] Error notifying Hub of activity
Traceback (most recent call last):
File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 572, in notify
await client.fetch(req)
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting