Spawn failed: Server at http://127.0.0.1:42783/user/ccadmin/api didn't respond in 30 seconds

JupyterHub
, , ,
1

Hi Guys,
my first issue is “A Jupyterhub Server is running”, this is my post, due to manually start Jupyterlab by “sudo /usr/local/bin/jupyter lab” is working so I added it into c.Spawner.cmd to force start Jupyterlab,I have to increase timeout to 600, but still getting Spawn failed issue

here are my config and debug logs:
BaiduShurufa_2024-7-10_6-35-1

[ccadmin@cccdsw ~]$ sudo /usr/local/bin/jupyterhub -f jupyterhub_config.py
[W 2024-07-10 06:24:44.965 JupyterHub app:1598]
extra_log_file is DEPRECATED in jupyterhub-0.8.2.

extra_log_file only redirects logs of the Hub itself,
and will discard any other output, such as
that of subprocess spawners or the proxy.

It is STRONGLY recommended that you redirect process
output instead, e.g.

    jupyterhub &>> '/var/log/jupyterhub.log'

[I 2024-07-10 06:24:44.966 JupyterHub app:3286] Running JupyterHub version 5.0.0
[I 2024-07-10 06:24:44.966 JupyterHub app:3316] Using Authenticator: jupyterhub.auth.PAMAuthenticator-5.0.0
[I 2024-07-10 06:24:44.966 JupyterHub app:3316] Using Spawner: jupyterhub.spawner.SimpleLocalProcessSpawner-5.0.0
[I 2024-07-10 06:24:44.966 JupyterHub app:3316] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.0.0
[D 2024-07-10 06:24:44.969 JupyterHub app:3246] Could not load pycurl: No module named ‘pycurl’
pycurl is recommended if you have a large number of users.
[I 2024-07-10 06:24:44.969 JupyterHub app:1817] Loading cookie_secret from /home/ccadmin/jupyterhub_cookie_secret
[D 2024-07-10 06:24:44.969 JupyterHub app:1984] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2024-07-10 06:24:44.988 JupyterHub orm:1510] database schema version found: 4621fec11365
[I 2024-07-10 06:24:45.054 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2024-07-10 06:24:45.055 JupyterHub app:2291] Loading roles into database
[D 2024-07-10 06:24:45.076 JupyterHub app:2638] Purging expired APITokens
[D 2024-07-10 06:24:45.079 JupyterHub app:2638] Purging expired OAuthCodes
[D 2024-07-10 06:24:45.080 JupyterHub app:2638] Purging expired Shares
[D 2024-07-10 06:24:45.083 JupyterHub app:2638] Purging expired ShareCodes
[D 2024-07-10 06:24:45.085 JupyterHub app:2412] Loading role assignments from config
[D 2024-07-10 06:24:45.102 JupyterHub app:2923] Initializing spawners
[D 2024-07-10 06:24:45.111 JupyterHub app:3061] Loaded users:

[I 2024-07-10 06:24:45.111 JupyterHub app:3355] Initialized 0 spawners in 0.009 seconds
[I 2024-07-10 06:24:45.116 JupyterHub metrics:371] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-07-10 06:24:45.117 JupyterHub metrics:371] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2024-07-10 06:24:45.118 JupyterHub metrics:371] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I 2024-07-10 06:24:45.118 JupyterHub proxy:752] Starting proxy @ https://0.0.0.0:8000/
[D 2024-07-10 06:24:45.119 JupyterHub proxy:753] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘0.0.0.0’, ‘–port’, ‘8000’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http://127.0.0.1:8081/hub/error’, ‘–log-level’, ‘debug’, ‘–ssl-key’, ‘/etc/jupyterhub/key.pem’, ‘–ssl-cert’, ‘/etc/jupyterhub/cert.pem’]
[D 2024-07-10 06:24:45.124 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
[D 2024-07-10 06:24:45.125 JupyterHub utils:272] Waiting 10s for server at cccdsw.cai.chen.sg:8000
[D 2024-07-10 06:24:45.126 JupyterHub utils:119] Server at cccdsw.cai.chen.sg:8000 not ready: [Errno 111] Connection refused
[D 2024-07-10 06:24:45.126 JupyterHub utils:272] Waiting 10s for server at 127.0.0.1:8001
[D 2024-07-10 06:24:45.126 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-07-10 06:24:45.146 JupyterHub utils:119] Server at cccdsw.cai.chen.sg:8000 not ready: [Errno 111] Connection refused
[D 2024-07-10 06:24:45.207 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
06:24:45.246 [ConfigProxy] info: Proxying https://0.0.0.0:8000 to (no default)
06:24:45.248 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-07-10 06:24:45.284 JupyterHub utils:280] Server at 127.0.0.1:8001 responded in 0.16s
[D 2024-07-10 06:24:45.438 JupyterHub utils:280] Server at cccdsw.cai.chen.sg:8000 responded in 0.31s
[D 2024-07-10 06:24:45.439 JupyterHub proxy:832] Proxy started and appears to be up
[D 2024-07-10 06:24:45.441 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
06:24:45.447 [ConfigProxy] info: 200 GET /api/routes
[I 2024-07-10 06:24:45.447 JupyterHub app:3669] Hub API listening on http://127.0.0.1:8081/hub/
[D 2024-07-10 06:24:45.447 JupyterHub proxy:389] Fetching routes to check
[D 2024-07-10 06:24:45.447 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
06:24:45.449 [ConfigProxy] info: 200 GET /api/routes
[D 2024-07-10 06:24:45.449 JupyterHub proxy:392] Checking routes
[I 2024-07-10 06:24:45.449 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081
[D 2024-07-10 06:24:45.449 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
06:24:45.451 [ConfigProxy] info: Adding route / → http://127.0.0.1:8081
06:24:45.451 [ConfigProxy] info: Route added / → http://127.0.0.1:8081
06:24:45.451 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-07-10 06:24:45.452 JupyterHub app:3710] JupyterHub is now running at https://0.0.0.0:8000/
[D 2024-07-10 06:24:45.453 JupyterHub app:3279] It took 0.501 seconds for the Hub to start
06:24:47.326 [ConfigProxy] debug: PROXY WEB /hub/api/users/ccadmin/server/progress to http://127.0.0.1:8081
[D 2024-07-10 06:24:47.340 JupyterHub base:411] Refreshing auth for ccadmin
[D 2024-07-10 06:24:47.341 JupyterHub scopes:1010] Checking access to /hub/api/users/ccadmin/server/progress via scope read:servers!server=ccadmin/
[W 2024-07-10 06:24:47.346 JupyterHub log:192] 404 GET /hub/api/users/ccadmin/server/progress?_xsrf=[secret] (ccadmin@172.16.25.7) 14.92ms
06:24:47.348 [ConfigProxy] debug: Not recording activity for status 404 on /
06:24:57.094 [ConfigProxy] debug: PROXY WEB /hub/home to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.099 JupyterHub user:496] Creating <class ‘jupyterhub.spawner.SimpleLocalProcessSpawner’> for ccadmin:
[I 2024-07-10 06:24:57.132 JupyterHub log:192] 200 GET /hub/home (ccadmin@172.16.25.7) 37.06ms
06:24:57.173 [ConfigProxy] debug: PROXY WEB /hub/static/css/style.min.css to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.175 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=767a5bba6db14f852eca424d08f508ae6c199900b0c5ef7009723188e8bee2e877675e443eaaeab51bae1a12cedffddf99ccf4a4e5684ade405134325e7af7e2 (@172.16.25.7) 0.92ms
06:24:57.285 [ConfigProxy] debug: PROXY WEB /hub/static/components/requirejs/require.js to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.287 JupyterHub log:192] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@172.16.25.7) 0.65ms
06:24:57.321 [ConfigProxy] debug: PROXY WEB /hub/static/components/jquery/dist/jquery.min.js to http://127.0.0.1:8081
06:24:57.322 [ConfigProxy] debug: PROXY WEB /hub/static/components/bootstrap/dist/js/bootstrap.bundle.min.js to http://127.0.0.1:8081
06:24:57.322 [ConfigProxy] debug: PROXY WEB /hub/logo to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.324 JupyterHub log:192] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=de027062931edd07b01842eff24fc15fdbdcaa1af245dcd133155faba9e0c965f0a34dc6144ce3b149bc43b4597073c792cb6dabbfc6168c63095523923bcf77 (@172.16.25.7) 0.82ms
[D 2024-07-10 06:24:57.325 JupyterHub log:192] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.bundle.min.js?v=ecf8bfa2d7656db091f8b9d6f85ecfc057120c93ae5090773b1b441db838bd232fcef26375ee0fa35bf8051f4675cf5a5cd50d155518f922b9d70593f161741a (@172.16.25.7) 1.14ms
[D 2024-07-10 06:24:57.325 JupyterHub log:192] 200 GET /hub/logo (@172.16.25.7) 0.59ms
06:24:57.520 [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.522 JupyterHub log:192] 200 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@172.16.25.7) 0.57ms
06:24:57.522 [ConfigProxy] debug: PROXY WEB /hub/static/js/home.js to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.523 JupyterHub log:192] 200 GET /hub/static/js/home.js?v=20240710062445 (@172.16.25.7) 0.49ms
06:24:57.561 [ConfigProxy] debug: PROXY WEB /hub/static/components/moment/moment.js to http://127.0.0.1:8081
06:24:57.562 [ConfigProxy] debug: PROXY WEB /hub/static/js/jhapi.js to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.562 JupyterHub log:192] 200 GET /hub/static/components/moment/moment.js?v=20240710062445 (@172.16.25.7) 0.60ms
[D 2024-07-10 06:24:57.563 JupyterHub log:192] 200 GET /hub/static/js/jhapi.js?v=20240710062445 (@172.16.25.7) 0.50ms
06:24:57.615 [ConfigProxy] debug: PROXY WEB /hub/static/js/utils.js to http://127.0.0.1:8081
[D 2024-07-10 06:24:57.617 JupyterHub log:192] 200 GET /hub/static/js/utils.js?v=20240710062445 (@172.16.25.7) 0.76ms
06:24:58.376 [ConfigProxy] debug: PROXY WEB /hub/spawn/ccadmin to http://127.0.0.1:8081
[D 2024-07-10 06:24:58.382 JupyterHub scopes:1010] Checking access to /hub/spawn/ccadmin via scope servers!server=ccadmin/
[D 2024-07-10 06:24:58.382 JupyterHub pages:216] Triggering spawn with default options for ccadmin
[D 2024-07-10 06:24:58.382 JupyterHub base:1095] Initiating spawn for ccadmin
[D 2024-07-10 06:24:58.382 JupyterHub base:1099] 0/100 concurrent spawns
[D 2024-07-10 06:24:58.383 JupyterHub base:1104] 0 active servers
[I 2024-07-10 06:24:58.451 JupyterHub provider:661] Creating oauth client jupyterhub-user-ccadmin
[D 2024-07-10 06:24:58.534 JupyterHub user:912] Calling Spawner.start for ccadmin
[I 2024-07-10 06:24:58.536 JupyterHub spawner:1738] Spawning sudo /usr/local/bin/jupyter lab --ip=0.0.0.0 --port=8888
[D 2024-07-10 06:24:58.572 JupyterHub spawner:1432] Polling subprocess every 30s
[D 2024-07-10 06:24:58.598 JupyterHub utils:292] Waiting 30s for server at http://127.0.0.1:42783/user/ccadmin/api
[I 2024-07-10 06:24:59.384 JupyterHub log:192] 302 GET /hub/spawn/ccadmin → /hub/spawn-pending/ccadmin (ccadmin@172.16.25.7) 1006.39ms
06:24:59.384 [ConfigProxy] debug: Not recording activity for status 302 on /
06:24:59.403 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/ccadmin to http://127.0.0.1:8081
[D 2024-07-10 06:24:59.411 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/ccadmin via scope servers!server=ccadmin/
[I 2024-07-10 06:24:59.411 JupyterHub pages:397] ccadmin is pending spawn
[I 2024-07-10 06:24:59.416 JupyterHub log:192] 200 GET /hub/spawn-pending/ccadmin (ccadmin@172.16.25.7) 11.86ms
[I 2024-07-10 06:24:59.484 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-07-10 06:24:59.490 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-07-10 06:24:59.496 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-07-10 06:24:59.500 ServerApp] nbclassic | extension was successfully linked.
06:24:59.501 [ConfigProxy] debug: PROXY WEB /hub/api/users/ccadmin/server/progress to http://127.0.0.1:8081
[I 2024-07-10 06:24:59.504 ServerApp] notebook | extension was successfully linked.
[D 2024-07-10 06:24:59.507 JupyterHub scopes:1010] Checking access to /hub/api/users/ccadmin/server/progress via scope read:servers!server=ccadmin/
[I 2024-07-10 06:24:59.747 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-07-10 06:24:59.803 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-07-10 06:24:59.805 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-07-10 06:24:59.806 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-07-10 06:24:59.807 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/site-packages/jupyterlab
[I 2024-07-10 06:24:59.807 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-07-10 06:24:59.808 LabApp] Extension Manager is ‘pypi’.
[I 2024-07-10 06:24:59.819 ServerApp] jupyterlab | extension was successfully loaded.

| | | |_ __ | | | | ___
| || | ’ / / _ | / -)
_/| ./_,_,|____|
|_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2024-07-10 06:24:59.822 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-07-10 06:24:59.825 ServerApp] notebook | extension was successfully loaded.
[I 2024-07-10 06:24:59.825 ServerApp] Serving notebooks from local directory: /tmp/ccadmin
[I 2024-07-10 06:24:59.826 ServerApp] Jupyter Server 2.14.1 is running at:
[I 2024-07-10 06:24:59.826 ServerApp] http://cccdsw.cai.chen.sg:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9
[I 2024-07-10 06:24:59.826 ServerApp] http://127.0.0.1:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9
[I 2024-07-10 06:24:59.826 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-07-10 06:24:59.829 ServerApp]

To access the server, open this file in a browser:
    file:///root/.local/share/jupyter/runtime/jpserver-24610-open.html
Or copy and paste one of these URLs:
    http://cccdsw.cai.chen.sg:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9
    http://127.0.0.1:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9

[I 2024-07-10 06:25:00.017 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[W 2024-07-10 06:25:08.383 JupyterHub base:1258] User ccadmin is slow to become responsive (timeout=10)
[D 2024-07-10 06:25:08.383 JupyterHub base:1263] Expecting server for ccadmin at: http://127.0.0.1:42783/user/ccadmin/
[W 2024-07-10 06:25:26.965 JupyterHub user:1050] ccadmin’s server never showed up at http://127.0.0.1:42783/user/ccadmin/ after 30 seconds. Giving up.

Common causes of this timeout, and debugging tips:

1. The server didn't finish starting,
   or it crashed due to a configuration issue.
   Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
   This may be a configuration issue specific to your chosen Spawner.
   Check the single-user server logs and resource to make sure the URL
   is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
   To fix: increase `Spawner.http_timeout` configuration
   to a number of seconds that is enough for servers to become responsive.

[D 2024-07-10 06:25:26.965 JupyterHub user:1095] Stopping ccadmin
[D 2024-07-10 06:25:26.965 JupyterHub spawner:1826] Interrupting 24609
[I 2024-07-10 06:25:26.966 ServerApp] interrupted
[I 2024-07-10 06:25:26.967 ServerApp] Serving notebooks from local directory: /tmp/ccadmin
0 active kernels
Jupyter Server 2.14.1 is running at:
http://cccdsw.cai.chen.sg:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9
http://127.0.0.1:8888/lab?token=c63f96aa077e70647e40d67162c71878ebe60d1b279b1df9
Shut down this Jupyter server (y/[n])? [I 2024-07-10 06:25:31.972 ServerApp] No answer for 5s:
[I 2024-07-10 06:25:31.972 ServerApp] resuming operation…
[D 2024-07-10 06:25:37.174 JupyterHub spawner:1834] Terminating 24609
[C 2024-07-10 06:25:37.175 ServerApp] received signal 15, stopping
[I 2024-07-10 06:25:37.176 ServerApp] Shutting down 6 extensions
[D 2024-07-10 06:25:37.503 JupyterHub user:1117] Deleting oauth client jupyterhub-user-ccadmin
[D 2024-07-10 06:25:37.529 JupyterHub user:1120] Finished stopping ccadmin
[E 2024-07-10 06:25:37.576 JupyterHub gen:629] Exception in Future <Task finished name=‘Task-697’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py:1115> exception=TimeoutError(“Server at http://127.0.0.1:42783/user/ccadmin/api didn’t respond in 30 seconds”)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/tornado/gen.py”, line 624, in error_callback
future.result()
File “/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py”, line 1122, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1028, in spawn
await self._wait_up(spawner)
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1071, in _wait_up
raise e
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1042, in _wait_up
resp = await server.wait_up(
File “/usr/local/lib/python3.8/site-packages/jupyterhub/utils.py”, line 322, in wait_for_http_server
re = await exponential_backoff(
File “/usr/local/lib/python3.8/site-packages/jupyterhub/utils.py”, line 265, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:42783/user/ccadmin/api didn’t respond in 30 seconds

[I 2024-07-10 06:25:37.579 JupyterHub log:192] 200 GET /hub/api/users/ccadmin/server/progress?_xsrf=[secret] (ccadmin@172.16.25.7) 38076.82ms

2

You will have to launch jupyterhub-singleuser which is a wrapper around jupyterlanb/notebook that takes care of auth. Try using c.Spawner.cmd = ['/usr/local/bin/jupyterhub-singleuser'].

1 Like
3

Thank you mahendrapaipuri,
I have updated my config file, still not working, Jupyterjhub version is 5.0.0

BaiduShurufa_2024-7-10_17-55-13
BaiduShurufa_2024-7-10_17-55-13695×466 11.3 KB

17:52:11.089 [ConfigProxy] debug: PROXY WEB /hub/spawn/ccadmin to http://127.0.0.1:8081
[D 2024-07-10 17:52:11.095 JupyterHub scopes:1010] Checking access to /hub/spawn/ccadmin via scope servers!server=ccadmin/
[D 2024-07-10 17:52:11.095 JupyterHub pages:216] Triggering spawn with default options for ccadmin
[D 2024-07-10 17:52:11.095 JupyterHub base:1095] Initiating spawn for ccadmin
[D 2024-07-10 17:52:11.095 JupyterHub base:1099] 0/100 concurrent spawns
[D 2024-07-10 17:52:11.095 JupyterHub base:1104] 0 active servers
[I 2024-07-10 17:52:11.172 JupyterHub provider:661] Creating oauth client jupyterhub-user-ccadmin
[D 2024-07-10 17:52:11.264 JupyterHub user:912] Calling Spawner.start for ccadmin
[I 2024-07-10 17:52:11.266 JupyterHub spawner:1738] Spawning /usr/local/bin/jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --allow-root
[D 2024-07-10 17:52:11.310 JupyterHub spawner:1432] Polling subprocess every 30s
[D 2024-07-10 17:52:11.341 JupyterHub utils:292] Waiting 30s for server at http://127.0.0.1:37024/user/ccadmin/api
[I 2024-07-10 17:52:12.097 JupyterHub log:192] 302 GET /hub/spawn/ccadmin → /hub/spawn-pending/ccadmin (ccadmin@172.16.25.20) 1006.13ms
17:52:12.098 [ConfigProxy] debug: Not recording activity for status 302 on /
17:52:12.119 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/ccadmin to http://127.0.0.1:8081
[D 2024-07-10 17:52:12.127 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/ccadmin via scope servers!server=ccadmin/
[I 2024-07-10 17:52:12.128 JupyterHub pages:397] ccadmin is pending spawn
[I 2024-07-10 17:52:12.132 JupyterHub log:192] 200 GET /hub/spawn-pending/ccadmin (ccadmin@172.16.25.20) 11.36ms
17:52:12.257 [ConfigProxy] debug: PROXY WEB /hub/api/users/ccadmin/server/progress to http://127.0.0.1:8081
[D 2024-07-10 17:52:12.264 JupyterHub scopes:1010] Checking access to /hub/api/users/ccadmin/server/progress via scope read:servers!server=ccadmin/
[W 2024-07-10 17:52:12.574 ServerApp] A _jupyter_server_extension_points function was not found in nbclassic. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-07-10 17:52:12.578 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-07-10 17:52:12.583 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-07-10 17:52:12.583 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.0.0
[I 2024-07-10 17:52:12.584 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
[I 2024-07-10 17:52:12.588 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-07-10 17:52:12.589 LabApp] ‘extra_template_paths’ was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-07-10 17:52:12.593 ServerApp] jupyterlab | extension was successfully linked.
[W 2024-07-10 17:52:12.594 NotebookApp] ‘extra_template_paths’ was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-07-10 17:52:12.597 ServerApp] nbclassic | extension was successfully linked.
[W 2024-07-10 17:52:12.599 JupyterNotebookApp] ‘extra_template_paths’ was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-07-10 17:52:12.602 ServerApp] notebook | extension was successfully linked.
[I 2024-07-10 17:52:12.885 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-07-10 17:52:12.941 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-07-10 17:52:12.944 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-07-10 17:52:12.945 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-07-10 17:52:12.953 JupyterHub log:192] 200 GET /hub/api (@127.0.0.1) 0.80ms
[I 2024-07-10 17:52:12.955 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-07-10 17:52:12.956 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-07-10 17:52:12.963 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/site-packages/jupyterlab
[I 2024-07-10 17:52:12.963 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-07-10 17:52:12.965 LabApp] Extension Manager is ‘pypi’.
[I 2024-07-10 17:52:12.981 ServerApp] jupyterlab | extension was successfully loaded.

| | | |_ __ | | | | ___
| || | ’ / / _ | / -)
_/| ./_,_,|____|
|_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2024-07-10 17:52:12.985 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-07-10 17:52:12.989 ServerApp] notebook | extension was successfully loaded.
[I 2024-07-10 17:52:12.989 ServerApp] Serving notebooks from local directory: /tmp/ccadmin
[I 2024-07-10 17:52:12.989 ServerApp] Jupyter Server 2.14.1 is running at:
[I 2024-07-10 17:52:12.989 ServerApp] http://cccdsw.cai.chen.sg:8888/user/ccadmin/lab?token=…
[I 2024-07-10 17:52:12.989 ServerApp] http://127.0.0.1:8888/user/ccadmin/lab?token=…
[I 2024-07-10 17:52:12.989 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 2024-07-10 17:52:12.999 JupyterHub base:366] Recording first activity for <APIToken(‘8187…’, user=‘ccadmin’, client_id=‘jupyterhub’)>
[D 2024-07-10 17:52:13.031 JupyterHub scopes:1010] Checking access to /hub/api/users/ccadmin/activity via scope users:activity!user=ccadmin
[D 2024-07-10 17:52:13.035 JupyterHub users:959] Activity for user ccadmin: 2024-07-10T09:52:12.930785Z
[D 2024-07-10 17:52:13.035 JupyterHub users:977] Activity on server ccadmin/: 2024-07-10T09:52:12.930785Z
[I 2024-07-10 17:52:13.071 JupyterHub log:192] 200 POST /hub/api/users/ccadmin/activity (ccadmin@127.0.0.1) 75.67ms
[I 2024-07-10 17:52:13.146 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server
[W 2024-07-10 17:52:17.968 JupyterHub log:192] 403 GET /hub/api/user (@127.0.0.1) 2.80ms
[W 2024-07-10 17:52:17.968 ServerApp] No Hub user identified for request
[I 2024-07-10 17:52:17.969 ServerApp] Setting new xsrf cookie for b’:eXfDn8JlUg0AUwzFeUpugJlrr6AGLT9gReNHn7R6RTE=’ {‘path’: ‘/user/ccadmin/’, ‘max_age’: 3600}
[W 2024-07-10 17:52:17.984 ServerApp] 404 GET /api/kernels/f6e76c72-5aa5-4c4b-be7e-ca9119898f1a?1720605111321 (@172.16.25.18) 20.99ms
[W 2024-07-10 17:52:21.096 JupyterHub base:1258] User ccadmin is slow to become responsive (timeout=10)
[D 2024-07-10 17:52:21.097 JupyterHub base:1263] Expecting server for ccadmin at: http://127.0.0.1:37024/user/ccadmin/
[I 2024-07-10 17:52:33.971 ServerApp] Setting new xsrf cookie for b’:eXfDn8JlUg0AUwzFeUpugJlrr6AGLT9gReNHn7R6RTE=’ {‘path’: ‘/user/ccadmin/’, ‘max_age’: 3600}
[W 2024-07-10 17:52:33.972 ServerApp] 404 GET /api/kernels/f6e76c72-5aa5-4c4b-be7e-ca9119898f1a/channels?session_id=d092bede-238a-4bf2-965b-448b772be4f6 (@172.16.25.18) 1.54ms
[I 2024-07-10 17:52:33.987 ServerApp] Setting new xsrf cookie for b’:eXfDn8JlUg0AUwzFeUpugJlrr6AGLT9gReNHn7R6RTE=’ {‘path’: ‘/user/ccadmin/’, ‘max_age’: 3600}
[W 2024-07-10 17:52:33.987 ServerApp] 404 GET /api/kernels/f6e76c72-5aa5-4c4b-be7e-ca9119898f1a?1720605127367 (@172.16.25.18) 1.38ms
[W 2024-07-10 17:52:43.190 JupyterHub user:1050] ccadmin’s server never showed up at http://127.0.0.1:37024/user/ccadmin/ after 30 seconds. Giving up.

Common causes of this timeout, and debugging tips:

1. The server didn't finish starting,
   or it crashed due to a configuration issue.
   Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
   This may be a configuration issue specific to your chosen Spawner.
   Check the single-user server logs and resource to make sure the URL
   is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
   To fix: increase `Spawner.http_timeout` configuration
   to a number of seconds that is enough for servers to become responsive.

[D 2024-07-10 17:52:43.190 JupyterHub user:1095] Stopping ccadmin
[D 2024-07-10 17:52:43.190 JupyterHub spawner:1826] Interrupting 4742
[I 2024-07-10 17:52:43.190 ServerApp] interrupted
[C 2024-07-10 17:52:43.191 ServerApp] Shutting down…
[I 2024-07-10 17:52:43.191 ServerApp] Shutting down 7 extensions
[D 2024-07-10 17:52:43.426 JupyterHub user:1117] Deleting oauth client jupyterhub-user-ccadmin
[D 2024-07-10 17:52:43.458 JupyterHub user:1120] Finished stopping ccadmin
[E 2024-07-10 17:52:43.521 JupyterHub gen:629] Exception in Future <Task finished name=‘Task-811’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py:1115> exception=TimeoutError(“Server at http://127.0.0.1:37024/user/ccadmin/api didn’t respond in 30 seconds”)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/tornado/gen.py”, line 624, in error_callback
future.result()
File “/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py”, line 1122, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1028, in spawn
await self._wait_up(spawner)
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1071, in _wait_up
raise e
File “/usr/local/lib/python3.8/site-packages/jupyterhub/user.py”, line 1042, in _wait_up
resp = await server.wait_up(
File “/usr/local/lib/python3.8/site-packages/jupyterhub/utils.py”, line 322, in wait_for_http_server
re = await exponential_backoff(
File “/usr/local/lib/python3.8/site-packages/jupyterhub/utils.py”, line 265, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:37024/user/ccadmin/api didn’t respond in 30 seconds

[I 2024-07-10 17:52:43.523 JupyterHub log:192] 200 GET /hub/api/users/ccadmin/server/progress?_xsrf=[secret] (ccadmin@172.16.25.20) 31264.51ms

4

It looks like the port specified by the JupyterHub spawner is ignored because you’re forcibly overriding it in your c.Spawner.args '--ip=0.0.0.0', '--port=8888'. Try removing that.

2 Likes
5

Yes! Thank you very much!!!
May I know how did you find it from logs?

6

Your logs

and your configuration snippet above show that.

How did I know that’s the problem? It’s just how JupyterHub works. The port and IP are set by the spawner. Some spawners allow you to hard-code a port (e.g. container spawners may allow this since every singleuser server is in its own container and there’s no need to worry about port collisions). The LocalProcessSpawner doesn’t allow this since only one process on the host can listen on 8888, so the spawner chooses a random port for each singleuser server.

1 Like