Proxy does not accept SwarmSpawner

JupyterHub
,
1

Hi Everyone,

We are trying to implement SwarmSpawner on an existing Jupyerhub enviroment which currently are using DockerSpawner. The service is created, replicas is 1/1 and the container is spawned on the worker node. However, the proxy is not showing anything happening which might be why the single-user server is not starting in the web-browser. Here’s the proxy output when using DockerSpawner that works:

15:07:23.824 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/spawn-pending/lagerber to http://127.0.0.1:9040
15:07:23.832 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev
15:07:23.836 [ConfigProxy] debug: PROXY WEB /jupyter-dev/user/lagerber/ to http://172.18.0.4:80
15:07:23.838 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev/user/lagerber
15:07:23.840 [ConfigProxy] debug: PROXY WEB /jupyter-dev/user/lagerber/lab to http://172.18.0.4:80
15:07:23.844 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev/user/lagerber
15:07:23.847 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api/oauth2/authorize to http://127.0.0.1:9040
15:07:23.867 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev
15:07:23.871 [ConfigProxy] debug: PROXY WEB /jupyter-dev/user/lagerber/oauth_callback to http://172.18.0.4:80
15:07:23.883 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api/oauth2/token to http://127.0.0.1:9040
15:07:23.919 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api/user to http://127.0.0.1:9040
15:07:23.928 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev/user/lagerber
15:07:23.934 [ConfigProxy] debug: PROXY WEB /jupyter-dev/user/lagerber/lab to http://172.18.0.4:80

Meanwhile when starting the SwarmSpawner, just stays listening:

5:11:00.788 [ConfigProxy] debug: Not recording activity for status 302 on /jupyter-dev
15:11:00.793 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/spawn-pending/lagerber to http://127.0.0.1:9040
15:11:00.834 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/static/favicon.ico to http://127.0.0.1:9040
15:11:00.844 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api/users/lagerber/server/progress to http://127.0.0.1:9040
15:11:02.983 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api to http://127.0.0.1:9040
15:11:03.090 [ConfigProxy] debug: PROXY WEB /jupyter-dev/hub/api/users/lagerber/activity to http://127.0.0.1:9040

Jupyterhub+Proxy is running on the same server, is there any additions in jupyterhub_config.py or spawner options that needs to be added? The servers are on the same network so there isn’t any firewall blockage between them

2

I have set the same image but with two virtual machines as a test since it’s much less components involved, the result is the same but now there is an error message that indicates of DNS issues:
Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)

the result of service logs

jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.557 ServerApp] notebook_shim | extension was successfully linked.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.587 ServerApp] notebook_shim | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.588 ServerApp] jupyter_lsp | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.589 ServerApp] jupyter_server_terminals | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.600 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.600 ServerApp] jupyterhub | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.605 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/dist-packages/jupyterlab
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.605 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.606 LabApp] Extension Manager is 'pypi'.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.609 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.682 ServerApp] jupyterlab | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.682 ServerApp] [Jupytext Server Extension] NotebookApp.contents_manager_class is (a subclass of) jupytext.TextFileContentsManager already - OK
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.682 ServerApp] jupyterlab_jupytext | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp] notebook | extension was successfully loaded.
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp] Serving notebooks from local directory: /home/student/notebooks
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp] Jupyter Server 2.15.0 is running at:
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp] http://hostname:8888/user/test/lab?token=...
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp]     http://127.0.0.1:80/user/test/lab?token=...
jupyter-test.1.86wxzt3i74ej@cn1    | [I 2025-05-22 14:15:06.685 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

jupyterhub_config.py

import dockerspawner, os

c = get_config()  # noqa

c.JupyterHub.authenticator_class = 'dummy'
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'

c.JupyterHub.hub_ip = '192.168.10.10'
c.Jupyterhub.hub_port = '8000'
c.SwarmSpawner.hub_connect_url = 'http://192.168.10.10:8000'

c.ConfigurableHTTPProxy.should_start = True
c.ConfigurableHTTPProxy.api_url = 'http://192.168.10.10:8001'

c.SwarmSpawner.network_name = 'jupyterhub_swarm'
#c.SwarmSpawner.extra_host_config = {'network_mode': 'host'}
c.SwarmSpawner.ip = '0.0.0.0'
c.SwarmSpawner.port = 80
c.Spawner.start_timeout=400
c.Spawner.http_timeout=400
c.Spawner.cmd = ["jupyterhub-singleuser"]
c.Spawner.image = 'euclid_nvidiatorch:latest'
#c.Spawner.default_url = '/lab'

import logging
c.SwarmSpawner.debug = True
3

How are you running JupyterHub? Can you show us the full command you’re using? Can you also turn on debug logging and show us the full hub logs?

4

Sure, I will use the test enviroment since the issue and behaviour is the same but much more minimal and easy to troubleshoot. In my test enviroment, I only did part 1 and installed jupyterhub according to these instructions: Install JupyterHub and JupyterLab from the ground up — JupyterHub 1.2.0 documentation

For starting juptyerhub, I run the command sudo /opt/jupyterhub/bin/jupyterhub --debug -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

Output of the jupyerhub in debug mode:

sudo /opt/jupyterhub/bin/jupyterhub --debug -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py
[D 2025-05-23 07:18:25.983 JupyterHub application:908] Looking for /opt/jupyterhub/etc/jupyterhub/jupyterhub_config in /home/niceguy
[D 2025-05-23 07:18:25.983 JupyterHub application:929] Loaded config file: /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py
[I 2025-05-23 07:18:25.990 JupyterHub app:3354] Running JupyterHub version 5.3.0
[I 2025-05-23 07:18:25.990 JupyterHub app:3384] Using Authenticator: jupyterhub.auth.DummyAuthenticator-5.3.0
[I 2025-05-23 07:18:25.990 JupyterHub app:3384] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-14.0.0
[I 2025-05-23 07:18:25.990 JupyterHub app:3384] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.3.0
/opt/jupyterhub/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
  validate_schema(_schema)
[D 2025-05-23 07:18:25.993 JupyterHub app:3314] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2025-05-23 07:18:25.993 JupyterHub app:1835] Loading cookie_secret from /home/niceguy/jupyterhub_cookie_secret
[D 2025-05-23 07:18:25.994 JupyterHub app:2006] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2025-05-23 07:18:26.014 JupyterHub orm:1509] database schema version found: 4621fec11365
[I 2025-05-23 07:18:26.047 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2025-05-23 07:18:26.047 JupyterHub app:2346] Loading roles into database
[W 2025-05-23 07:18:26.051 JupyterHub auth:1522] Using testing authenticator DummyAuthenticator! This is not meant for production!
[D 2025-05-23 07:18:26.058 JupyterHub app:2693] Purging expired APITokens
[D 2025-05-23 07:18:26.059 JupyterHub app:2693] Purging expired OAuthCodes
[D 2025-05-23 07:18:26.060 JupyterHub app:2693] Purging expired Shares
[D 2025-05-23 07:18:26.060 JupyterHub app:2693] Purging expired ShareCodes
[D 2025-05-23 07:18:26.061 JupyterHub app:2467] Loading role assignments from config
[D 2025-05-23 07:18:26.070 JupyterHub app:2978] Initializing spawners
[D 2025-05-23 07:18:26.075 JupyterHub app:3128] Loaded users:
    
[I 2025-05-23 07:18:26.075 JupyterHub app:3424] Initialized 0 spawners in 0.005 seconds
[I 2025-05-23 07:18:26.078 JupyterHub metrics:425] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-05-23 07:18:26.079 JupyterHub metrics:425] Found 2 active users in the last ActiveUserPeriods.seven_days
[I 2025-05-23 07:18:26.079 JupyterHub metrics:425] Found 2 active users in the last ActiveUserPeriods.thirty_days
[W 2025-05-23 07:18:26.079 JupyterHub proxy:748] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2025-05-23 07:18:26.079 JupyterHub proxy:752] Starting proxy @ http://:8000
[D 2025-05-23 07:18:26.079 JupyterHub proxy:753] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '192.168.10.10', '--api-port', '8001', '--error-target', 'http://192.168.10.10:8081/hub/error', '--log-level', 'debug']
[D 2025-05-23 07:18:26.082 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
[D 2025-05-23 07:18:26.083 JupyterHub utils:277] Waiting 10s for server at master:8000
[D 2025-05-23 07:18:26.083 JupyterHub utils:121] Server at master:8000 not ready: [Errno 111] Connection refused
[D 2025-05-23 07:18:26.083 JupyterHub utils:277] Waiting 10s for server at 192.168.10.10:8001
[D 2025-05-23 07:18:26.083 JupyterHub utils:121] Server at 192.168.10.10:8001 not ready: [Errno 111] Connection refused
[D 2025-05-23 07:18:26.240 JupyterHub utils:121] Server at 192.168.10.10:8001 not ready: [Errno 111] Connection refused
[D 2025-05-23 07:18:26.274 JupyterHub utils:121] Server at master:8000 not ready: [Errno 111] Connection refused
[D 2025-05-23 07:18:26.343 JupyterHub utils:121] Server at master:8000 not ready: [Errno 111] Connection refused
2025-05-23T07:18:26.439Z [ConfigProxy] info: Proxying http://*:8000 to (no default)
2025-05-23T07:18:26.440Z [ConfigProxy] info: Proxy API at http://192.168.10.10:8001/api/routes
[D 2025-05-23 07:18:26.620 JupyterHub utils:285] Server at 192.168.10.10:8001 responded in 0.54s
[D 2025-05-23 07:18:27.040 JupyterHub utils:285] Server at master:8000 responded in 0.96s
[D 2025-05-23 07:18:27.040 JupyterHub proxy:832] Proxy started and appears to be up
[D 2025-05-23 07:18:27.055 JupyterHub proxy:925] Proxy: Fetching GET http://192.168.10.10:8001/api/routes
2025-05-23T07:18:27.070Z [ConfigProxy] info: 200 GET /api/routes 
[I 2025-05-23 07:18:27.071 JupyterHub app:3747] Hub API listening on http://192.168.10.10:8081/hub/
[D 2025-05-23 07:18:27.071 JupyterHub proxy:389] Fetching routes to check
[D 2025-05-23 07:18:27.071 JupyterHub proxy:925] Proxy: Fetching GET http://192.168.10.10:8001/api/routes
2025-05-23T07:18:27.073Z [ConfigProxy] info: 200 GET /api/routes 
[D 2025-05-23 07:18:27.073 JupyterHub proxy:392] Checking routes
[I 2025-05-23 07:18:27.073 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.10.10:8081
[D 2025-05-23 07:18:27.074 JupyterHub proxy:925] Proxy: Fetching POST http://192.168.10.10:8001/api/routes/
2025-05-23T07:18:27.075Z [ConfigProxy] info: Adding route / -> http://192.168.10.10:8081
2025-05-23T07:18:27.075Z [ConfigProxy] info: Route added / -> http://192.168.10.10:8081
2025-05-23T07:18:27.075Z [ConfigProxy] info: 201 POST /api/routes/ 
[I 2025-05-23 07:18:27.076 JupyterHub app:3778] JupyterHub is now running at http://:8000
[D 2025-05-23 07:18:27.076 JupyterHub app:3347] It took 1.096 seconds for the Hub to start
2025-05-23T07:19:36.658Z [ConfigProxy] debug: PROXY WEB / to http://192.168.10.10:8081
[I 2025-05-23 07:19:36.668 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.0.2.2) 0.43ms
2025-05-23T07:19:36.670Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-05-23T07:19:36.672Z [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.10.10:8081
[D 2025-05-23 07:19:36.703 JupyterHub base:411] Refreshing auth for test
[D 2025-05-23 07:19:36.703 JupyterHub user:496] Creating <class 'dockerspawner.swarmspawner.SwarmSpawner'> for test:
[I 2025-05-23 07:19:36.704 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (test@::ffff:10.0.2.2) 30.79ms
2025-05-23T07:19:36.706Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-05-23T07:19:36.711Z [ConfigProxy] debug: PROXY WEB /hub/spawn to http://192.168.10.10:8081
[D 2025-05-23 07:19:36.716 JupyterHub scopes:1013] Checking access to /hub/spawn via scope servers!server=test/
[D 2025-05-23 07:19:36.717 JupyterHub pages:216] Triggering spawn with default options for test
[D 2025-05-23 07:19:36.717 JupyterHub base:1097] Initiating spawn for test
[D 2025-05-23 07:19:36.717 JupyterHub base:1101] 0/100 concurrent spawns
[D 2025-05-23 07:19:36.717 JupyterHub base:1106] 0 active servers
[I 2025-05-23 07:19:36.737 JupyterHub provider:661] Creating oauth client jupyterhub-user-test
[D 2025-05-23 07:19:36.755 JupyterHub user:913] Calling Spawner.start for test
[D 2025-05-23 07:19:36.756 JupyterHub dockerspawner:1254] Skipping pull of euclid_nvidiatorch:latest
[D 2025-05-23 07:19:36.756 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[I 2025-05-23 07:19:36.770 JupyterHub dockerspawner:1040] Service 'jupyter-test' is gone
[I 2025-05-23 07:19:36.780 JupyterHub dockerspawner:1318] Created service jupyter-test (id: kvr99k7) from image euclid_nvidiatorch:latest
[I 2025-05-23 07:19:36.780 JupyterHub dockerspawner:1342] Starting service jupyter-test (id: kvr99k7)
[D 2025-05-23 07:19:36.780 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:36.780 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:36.786 JupyterHub swarmspawner:281] Service kvr99k7 state: pending
[I 2025-05-23 07:19:37.718 JupyterHub log:192] 302 GET /hub/spawn -> /hub/spawn-pending/test (test@::ffff:10.0.2.2) 1005.41ms
2025-05-23T07:19:37.719Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-05-23T07:19:37.722Z [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/test to http://192.168.10.10:8081
[D 2025-05-23 07:19:37.727 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/test via scope servers!server=test/
[I 2025-05-23 07:19:37.727 JupyterHub pages:400] test is pending spawn
[I 2025-05-23 07:19:37.754 JupyterHub log:192] 200 GET /hub/spawn-pending/test (test@::ffff:10.0.2.2) 30.79ms
2025-05-23T07:19:37.782Z [ConfigProxy] debug: PROXY WEB /hub/logo to http://192.168.10.10:8081
[D 2025-05-23 07:19:37.789 JupyterHub log:192] 200 GET /hub/logo (@::ffff:10.0.2.2) 4.79ms
[D 2025-05-23 07:19:37.789 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:37.790 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:37.796 JupyterHub swarmspawner:281] Service kvr99k7 state: preparing
2025-05-23T07:19:37.919Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://192.168.10.10:8081
2025-05-23T07:19:37.932Z [ConfigProxy] debug: PROXY WEB /hub/api/users/test/server/progress to http://192.168.10.10:8081
[D 2025-05-23 07:19:37.936 JupyterHub log:192] 200 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@::ffff:10.0.2.2) 15.70ms
[D 2025-05-23 07:19:37.941 JupyterHub scopes:1013] Checking access to /hub/api/users/test/server/progress via scope read:servers!server=test/
2025-05-23T07:19:38.040Z [ConfigProxy] debug: PROXY WEB /hub/static/favicon.ico to http://192.168.10.10:8081
[D 2025-05-23 07:19:38.043 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:10.0.2.2) 0.82ms
[D 2025-05-23 07:19:39.298 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:39.298 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:39.304 JupyterHub swarmspawner:281] Service kvr99k7 state: running
[D 2025-05-23 07:19:39.312 JupyterHub spawner:1693] Polling subprocess every 30s
[D 2025-05-23 07:19:39.312 JupyterHub dockerspawner:977] Persisting state for test: service name=jupyter-test, id=kvr99k758j8urb9hh0t1ut6xy
[D 2025-05-23 07:19:39.317 JupyterHub utils:297] Waiting 400s for server at http://jupyter-test:80/user/test/api
[W 2025-05-23 07:19:39.318 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:39.364 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
2025-05-23T07:19:39.434Z [ConfigProxy] debug: PROXY WEB /hub/home to http://192.168.10.10:8081
[I 2025-05-23 07:19:39.445 JupyterHub log:192] 200 GET /hub/home (test@::ffff:10.0.2.2) 8.68ms
2025-05-23T07:19:39.484Z [ConfigProxy] debug: PROXY WEB /hub/static/js/home.js to http://192.168.10.10:8081
[D 2025-05-23 07:19:39.488 JupyterHub log:192] 200 GET /hub/static/js/home.js?v=20250523071826 (@::ffff:10.0.2.2) 1.65ms
2025-05-23T07:19:39.499Z [ConfigProxy] debug: PROXY WEB /hub/static/components/moment/moment.js to http://192.168.10.10:8081
2025-05-23T07:19:39.501Z [ConfigProxy] debug: PROXY WEB /hub/static/js/jhapi.js to http://192.168.10.10:8081
[D 2025-05-23 07:19:39.502 JupyterHub log:192] 200 GET /hub/static/components/moment/moment.js?v=20250523071826 (@::ffff:10.0.2.2) 1.92ms
[W 2025-05-23 07:19:39.503 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[D 2025-05-23 07:19:39.507 JupyterHub log:192] 200 GET /hub/static/js/jhapi.js?v=20250523071826 (@::ffff:10.0.2.2) 1.34ms
2025-05-23T07:19:39.510Z [ConfigProxy] debug: PROXY WEB /hub/static/js/utils.js to http://192.168.10.10:8081
[D 2025-05-23 07:19:39.513 JupyterHub log:192] 200 GET /hub/static/js/utils.js?v=20250523071826 (@::ffff:10.0.2.2) 1.35ms
[W 2025-05-23 07:19:40.083 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
2025-05-23T07:19:40.492Z [ConfigProxy] debug: PROXY WEB /hub/api/users/test/server to http://192.168.10.10:8081
[D 2025-05-23 07:19:40.497 JupyterHub scopes:1013] Checking access to /hub/api/users/test/server via scope delete:servers!server=test/
[W 2025-05-23 07:19:40.499 JupyterHub users:717] Interrupting spawner test, pending spawn
Exception in callback chain_future.<locals>.copy(<Task cancell...base.py:1117>>) at /opt/jupyterhub/lib/python3.12/site-packages/tornado/concurrent.py:164
handle: <Handle chain_future.<locals>.copy(<Task cancell...base.py:1117>>) at /opt/jupyterhub/lib/python3.12/site-packages/tornado/concurrent.py:164>
Traceback (most recent call last):
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/jupyterhub/lib/python3.12/site-packages/tornado/concurrent.py", line 170, in copy
    a_exc = a.exception()
            ^^^^^^^^^^^^^
  File "/opt/jupyterhub/lib/python3.12/site-packages/jupyterhub/handlers/base.py", line 1124, in finish_user_spawn
    await spawn_future
  File "/opt/jupyterhub/lib/python3.12/site-packages/jupyterhub/user.py", line 1030, in spawn
    await self._wait_up(spawner)
  File "/opt/jupyterhub/lib/python3.12/site-packages/jupyterhub/user.py", line 1044, in _wait_up
    resp = await server.wait_up(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/jupyterhub/lib/python3.12/site-packages/jupyterhub/utils.py", line 327, in wait_for_http_server
    re = await exponential_backoff(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/jupyterhub/lib/python3.12/site-packages/jupyterhub/utils.py", line 266, in exponential_backoff
    await asyncio.sleep(dt)
  File "/usr/lib/python3.12/asyncio/tasks.py", line 665, in sleep
    return await future
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError
[I 2025-05-23 07:19:40.504 JupyterHub users:899] Server test is ready
[W 2025-05-23 07:19:40.505 JupyterHub users:782] Stream closed while handling /hub/api/users/test/server/progress?_xsrf=MnwxOjB8MTA6MTc0NzkyODIzMnw1Ol94c3JmfDg4Oll6WXpZVEZoT1dSallqQTNORFJpT0dFell6bGhZbVptWXpReFl6ZzFOakU2TWpjeE1qY3labUZqTVdGaU5ERXhaamcxTmpobU5UTTVNREE1WTJFNU9XWT18NzYxODg3OThhN2JkNWFhNGMyZDU5MmMxYmQwNGM5ODdiN2IwNjE1NmEwMjYzYjIyZWQ0OGI4NGYyYTQ1NDZiOA
[I 2025-05-23 07:19:40.505 JupyterHub log:192] 200 GET /hub/api/users/test/server/progress?_xsrf=[secret] (test@::ffff:10.0.2.2) 2567.88ms
2025-05-23T07:19:40.600Z [ConfigProxy] debug: PROXY WEB /hub/api to http://192.168.10.10:8081
[I 2025-05-23 07:19:40.602 JupyterHub log:192] 200 GET /hub/api (@::ffff:192.168.10.11) 0.83ms
2025-05-23T07:19:40.717Z [ConfigProxy] debug: PROXY WEB /hub/api/users/test/activity to http://192.168.10.10:8081
[D 2025-05-23 07:19:40.722 JupyterHub base:366] Recording first activity for <APIToken('0b97...', user='test', client_id='jupyterhub')>
[D 2025-05-23 07:19:40.731 JupyterHub scopes:1013] Checking access to /hub/api/users/test/activity via scope users:activity!user=test
[D 2025-05-23 07:19:40.734 JupyterHub users:1006] Activity for user test: 2025-05-23T07:19:40.581144Z
[D 2025-05-23 07:19:40.734 JupyterHub users:1024] Activity on server test/: 2025-05-23T07:19:40.581144Z
[I 2025-05-23 07:19:40.739 JupyterHub log:192] 200 POST /hub/api/users/test/activity (test@::ffff:192.168.10.11) 19.97ms
[I 2025-05-23 07:19:41.502 JupyterHub proxy:356] Removing user test from proxy (/user/test/)
[D 2025-05-23 07:19:41.503 JupyterHub proxy:925] Proxy: Fetching DELETE http://192.168.10.10:8001/api/routes/user/test
2025-05-23T07:19:41.506Z [ConfigProxy] warn: 404 DELETE /api/routes/user/test 
[E 2025-05-23 07:19:41.506 JupyterHub proxy:949] api_request to proxy failed: HTTP 404: Not Found
[W 2025-05-23 07:19:41.506 JupyterHub proxy:976] Route /user/test/ already deleted
[D 2025-05-23 07:19:41.507 JupyterHub user:1097] Stopping test
[D 2025-05-23 07:19:41.507 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:41.507 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:41.513 JupyterHub swarmspawner:145] Service kvr99k7 status: {'ContainerStatus': {'ContainerID': '2cad24cee217e0433c2bb377cf3767a85e53e7a594487c1e5acfa95b3828860b',
                         'ExitCode': 0,
                         'PID': 1540},
     'Message': 'started',
     'PortStatus': {},
     'State': 'running',
     'Timestamp': '2025-05-23T07:19:38.886505592Z'}
[I 2025-05-23 07:19:41.513 JupyterHub dockerspawner:1431] Stopping service jupyter-test (id: kvr99k7)
[I 2025-05-23 07:19:41.513 JupyterHub swarmspawner:258] Removing service kvr99k758j8urb9hh0t1ut6xy
[D 2025-05-23 07:19:41.538 JupyterHub user:1119] Deleting oauth client jupyterhub-user-test
[D 2025-05-23 07:19:41.547 JupyterHub user:1122] Finished stopping test
[I 2025-05-23 07:19:41.560 JupyterHub base:1349] User test server took 0.058 seconds to stop
[I 2025-05-23 07:19:41.561 JupyterHub log:192] 204 DELETE /hub/api/users/test/server?_xsrf=[secret] (test@::ffff:10.0.2.2) 1067.60ms
2025-05-23T07:19:42.044Z [ConfigProxy] debug: PROXY WEB /hub/spawn/test to http://192.168.10.10:8081
[D 2025-05-23 07:19:42.052 JupyterHub scopes:1013] Checking access to /hub/spawn/test via scope servers!server=test/
[D 2025-05-23 07:19:42.052 JupyterHub user:496] Creating <class 'dockerspawner.swarmspawner.SwarmSpawner'> for test:
[D 2025-05-23 07:19:42.052 JupyterHub pages:216] Triggering spawn with default options for test
[D 2025-05-23 07:19:42.053 JupyterHub base:1097] Initiating spawn for test
[D 2025-05-23 07:19:42.053 JupyterHub base:1101] 0/100 concurrent spawns
[D 2025-05-23 07:19:42.053 JupyterHub base:1106] 0 active servers
[I 2025-05-23 07:19:42.080 JupyterHub provider:661] Creating oauth client jupyterhub-user-test
[D 2025-05-23 07:19:42.100 JupyterHub user:913] Calling Spawner.start for test
[D 2025-05-23 07:19:42.100 JupyterHub dockerspawner:1254] Skipping pull of euclid_nvidiatorch:latest
[D 2025-05-23 07:19:42.100 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[I 2025-05-23 07:19:42.104 JupyterHub dockerspawner:1040] Service 'jupyter-test' is gone
[I 2025-05-23 07:19:42.112 JupyterHub dockerspawner:1318] Created service jupyter-test (id: ubobgqa) from image euclid_nvidiatorch:latest
[I 2025-05-23 07:19:42.112 JupyterHub dockerspawner:1342] Starting service jupyter-test (id: ubobgqa)
[D 2025-05-23 07:19:42.112 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:42.112 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:42.116 JupyterHub swarmspawner:281] Service ubobgqa state: pending
[I 2025-05-23 07:19:43.054 JupyterHub log:192] 302 GET /hub/spawn/test -> /hub/spawn-pending/test (test@::ffff:10.0.2.2) 1008.04ms
2025-05-23T07:19:43.056Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-05-23T07:19:43.059Z [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/test to http://192.168.10.10:8081
[D 2025-05-23 07:19:43.065 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/test via scope servers!server=test/
[I 2025-05-23 07:19:43.066 JupyterHub pages:400] test is pending spawn
[D 2025-05-23 07:19:43.067 JupyterHub log:192] 304 GET /hub/spawn-pending/test (test@::ffff:10.0.2.2) 6.27ms
2025-05-23T07:19:43.068Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-05-23T07:19:43.095Z [ConfigProxy] debug: PROXY WEB /hub/static/favicon.ico to http://192.168.10.10:8081
[D 2025-05-23 07:19:43.098 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:10.0.2.2) 0.94ms
2025-05-23T07:19:43.102Z [ConfigProxy] debug: PROXY WEB /hub/api/users/test/server/progress to http://192.168.10.10:8081
[D 2025-05-23 07:19:43.108 JupyterHub scopes:1013] Checking access to /hub/api/users/test/server/progress via scope read:servers!server=test/
[D 2025-05-23 07:19:43.117 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:43.117 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:43.122 JupyterHub swarmspawner:281] Service ubobgqa state: preparing
[D 2025-05-23 07:19:44.624 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:44.624 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:44.630 JupyterHub swarmspawner:281] Service ubobgqa state: running
[D 2025-05-23 07:19:44.639 JupyterHub spawner:1693] Polling subprocess every 30s
[D 2025-05-23 07:19:44.639 JupyterHub dockerspawner:977] Persisting state for test: service name=jupyter-test, id=ubobgqa8ch2mefy44y6wvorfy
[D 2025-05-23 07:19:44.646 JupyterHub utils:297] Waiting 400s for server at http://jupyter-test:80/user/test/api
[W 2025-05-23 07:19:44.648 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:44.682 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
2025-05-23T07:19:44.775Z [ConfigProxy] debug: PROXY WEB /hub/api to http://192.168.10.10:8081
[I 2025-05-23 07:19:44.778 JupyterHub log:192] 200 GET /hub/api (@::ffff:192.168.10.11) 0.73ms
[W 2025-05-23 07:19:44.840 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
2025-05-23T07:19:44.900Z [ConfigProxy] debug: PROXY WEB /hub/api/users/test/activity to http://192.168.10.10:8081
[D 2025-05-23 07:19:44.903 JupyterHub base:366] Recording first activity for <APIToken('2042...', user='test', client_id='jupyterhub')>
[D 2025-05-23 07:19:44.913 JupyterHub scopes:1013] Checking access to /hub/api/users/test/activity via scope users:activity!user=test
[D 2025-05-23 07:19:44.915 JupyterHub users:1006] Activity for user test: 2025-05-23T07:19:44.762377Z
[D 2025-05-23 07:19:44.915 JupyterHub users:1024] Activity on server test/: 2025-05-23T07:19:44.762377Z
[I 2025-05-23 07:19:44.921 JupyterHub log:192] 200 POST /hub/api/users/test/activity (test@::ffff:192.168.10.11) 19.80ms
[W 2025-05-23 07:19:44.922 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:45.431 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:46.833 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:50.971 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[D 2025-05-23 07:19:52.054 JupyterHub swarmspawner:156] Getting task of service 'jupyter-test'
[D 2025-05-23 07:19:52.054 JupyterHub dockerspawner:1034] Getting service 'jupyter-test'
[D 2025-05-23 07:19:52.062 JupyterHub swarmspawner:145] Service ubobgqa status: {'ContainerStatus': {'ContainerID': '17e3634ec7df66deaed2be49c53da7bc5a0c256e817ca9125e671dde400a2487',
                         'ExitCode': 0,
                         'PID': 1671},
     'Message': 'started',
     'PortStatus': {},
     'State': 'running',
     'Timestamp': '2025-05-23T07:19:43.904558436Z'}
[W 2025-05-23 07:19:52.062 JupyterHub base:1274] User test is slow to become responsive (timeout=10)
[D 2025-05-23 07:19:52.063 JupyterHub base:1279] Expecting server for test at: http://jupyter-test:80/user/test/
[W 2025-05-23 07:19:52.554 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:57.242 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:19:57.968 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:20:01.518 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)
[W 2025-05-23 07:20:03.145 JupyterHub utils:322] Failed to connect to http://jupyter-test:80/user/test/api ([Errno -3] Temporary failure in name resolution)

Interestingly, the container can resolve both the single-user server address and the hub’s:

sudo docker exec b0dfe6ac5386 curl -vvv http://jupyter-test:80/user/test/ && curl -vvv http://192.168.10.10:8000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.0.1.102:80...
* Connected to jupyter-test (10.0.1.102) port 80 (#0)
> GET /user/test/ HTTP/1.1
> Host: jupyter-test
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: TornadoServer/6.4
< Content-Type: text/html; charset=UTF-8
< Date: Fri, 23 May 2025 11:30:47 GMT
< Location: /user/test/lab?
< Content-Length: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host jupyter-test left intact
*   Trying 192.168.10.10:8000...
* Connected to 192.168.10.10 (192.168.10.10) port 8000
> GET / HTTP/1.1
> Host: 192.168.10.10:8000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< server: TornadoServer/6.5
< content-type: text/html
< date: Fri, 23 May 2025 11:30:47 GMT
< x-jupyterhub-version: 5.3.0
< access-control-allow-headers: accept, content-type, authorization
< content-security-policy: frame-ancestors 'none'; report-uri /hub/security/csp-report
< location: /hub/
< content-length: 0
< connection: keep-alive
< 
* Connection #0 to host 192.168.10.10 left intact