Uncaught exception GET: Cannot send error response after headers written

Hello everyone,

We are trying to set up Jupyter Hub on a HPC cluster using the PBS batchspawner.
Everything seems to work fine until the job is submitted on the compute node and the single-user server tries to contact the Hub. Then we get the following error:

  [E 2025-02-24 16:28:41.668 ServerApp] Uncaught exception GET /user/testuser/oauth_callback?code=Q2H7F0NDwAaKljSB2p0gAJUB8mxler&state=Ux3C5imOSy2r3XTzILMzxA (10.51.1.234)
    HTTPServerRequest(protocol='https', host='192.168.97.161', method='GET', uri='/user/testuser/oauth_callback?code=Q2H7F0NDwAaKljSB2p0gAJUB8mxler&state=Ux3C5imOSy2r3XTzILMzxA', version='HTTP/1.1', remote_ip='10.51.1.234')
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/singleuser/extension.py", line 102, in get
        return await super().get()
[...]
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 363, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
[E 2025-02-24 16:28:42.051 ServerApp] Cannot send error response after headers written
[I 2025-02-24 16:28:42.051 ServerApp] 302 GET /user/testuser/oauth_callback?code=[secret]&state=[secret] -> /user/testuser/lab (@10.51.1.234) 588.60ms
[E 2025-02-24 16:28:42.053 ServerApp] Uncaught exception GET /user/testuser/lab (10.51.1.234)
    HTTPServerRequest(protocol='https', host='192.168.97.161', method='GET', uri='/user/testuser/lab', version='HTTP/1.1', remote_ip='10.51.1.234')
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
[...]

    jinja2.exceptions.TemplateNotFound: '500.html' not found in search path: '/data/software/mamba/24.11.3-0/envs/jupyterhub/share/jupyter/lab/static'

The cluster OS version is CentOS 7.9 and the Jupyter/Mamba stack is the following:

Mamba 24.11.3 
python 3.13
jupyter_core 5.7.2
jupyter_client 8.6.3
jupyter_events 0.12.0
jupyterhub 5.2.1
jupyterlab 4.3.5
jupyterlab_server 2.27.3
jupyter_server 2.15.0
tornado 6.4.2 
batchspawner 1.3.0
wrapspawner 1.0.1

This is the content of jupyter_config.py:

c = get_config()
c.JupyterHub.allow_named_servers = True
c.JupyterHub.cleanup_servers = False
c.JupyterHub.hub_ip = '192.168.1.253' 
c.JupyterHub.init_spawners_timeout = 120
c.JupyterHub.ip = '192.168.97.161'
c.JupyterHub.port = 443
c.JupyterHub.service_check_interval = 900
c.JupyterHub.ssl_cert = '/data/software/jupyterhub/5.2.1/etc/ssl/jupyter.pem'
c.JupyterHub.ssl_key = '/data/software/jupyterhub/5.2.1/etc/ssl/jupyter.key'
c.JupyterHub.tornado_settings = {'slow_spawn_timeout': 900}
c.Spawner.start_timeout = 900
c.LocalProcessSpawner.start_timeout = 900
c.Authenticator.admin_users = {'testuser', 'testuser2'}
c.Authenticator.allow_all = True
c.Authenticator.delete_invalid_users = True

import batchspawner

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 900

c.ProfilesSpawner.profiles = [
    ("OpenPBS, 1 core(s), 4GB", 'pbs1cores4gb', 'batchspawner.PBSSpawner', dict(req_nprocs='1', req_queue='ccmq', req_memory='4gb', req_runtime='48:00:00')),
]

Do you have an idea on what could possibly be the issue?

Thanks,

Could you restart the Hub and share the logs right from the beginning of the hub in debug mode? Seems like single user server is attempting to reach hub API server at 192.168.97.161? Could you verify if API server is actually running?

Hello,

Here’s the output with debug on (at least, I switched on all the debug settings I could find in the jupyterhub_config.py, if there’s a way to be even more verbose please let me know):

Hub:

[I 2025-03-03 14:39:21.690 JupyterHub app:3346] Running JupyterHub version 5.2.1
[I 2025-03-03 14:39:21.690 JupyterHub app:3376] Using Authenticator: jupyterhub.auth.PAMAuthenticator-5.2.1
[I 2025-03-03 14:39:21.690 JupyterHub app:3376] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2025-03-03 14:39:21.690 JupyterHub app:3376] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.1
/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
  validate_schema(_schema)
[I 2025-03-03 14:39:21.719 JupyterHub app:1876] Writing cookie_secret to /root/jh/jupyterhub_cookie_secret
[I 2025-03-03 14:39:21.803 alembic.runtime.migration migration:207] Context impl SQLiteImpl.
[I 2025-03-03 14:39:21.803 alembic.runtime.migration migration:210] Will assume non-transactional DDL.
[I 2025-03-03 14:39:21.870 alembic.runtime.migration migration:618] Running stamp_revision  -> 4621fec11365
[I 2025-03-03 14:39:22.761 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2025-03-03 14:39:22.826 JupyterHub roles:281] Adding role admin for User: testuser
[I 2025-03-03 14:39:22.887 JupyterHub roles:281] Adding role user for User: testuser
[I 2025-03-03 14:39:22.938 JupyterHub roles:281] Adding role admin for User: testuser2
[I 2025-03-03 14:39:22.996 JupyterHub roles:281] Adding role user for User: testuser2
[I 2025-03-03 14:39:23.108 JupyterHub app:3416] Initialized 0 spawners in 0.008 seconds
[I 2025-03-03 14:39:23.112 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-03-03 14:39:23.113 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-03-03 14:39:23.114 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-03-03 14:39:23.114 JupyterHub proxy:752] Starting proxy @ https://192.168.97.161:443/
14:39:24.200 [ConfigProxy] info: Proxying https://192.168.97.161:443 to (no default)
14:39:24.203 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[I 2025-03-03 14:39:26.261 JupyterHub app:3739] Hub API listening on http://192.168.1.253:8081/hub/
14:39:26.262 [ConfigProxy] info: 200 GET /api/routes 
14:39:26.264 [ConfigProxy] info: 200 GET /api/routes 
[I 2025-03-03 14:39:26.264 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.1.253:8081
14:39:26.268 [ConfigProxy] info: Adding route / -> http://192.168.1.253:8081
14:39:26.269 [ConfigProxy] info: Route added / -> http://192.168.1.253:8081
14:39:26.270 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2025-03-03 14:39:26.270 JupyterHub app:3770] JupyterHub is now running at https://192.168.97.161:443/
14:39:30.335 [ConfigProxy] debug: PROXY WEB / to http://192.168.1.253:8081
[I 2025-03-03 14:39:30.348 JupyterHub log:192] 302 GET / -> /hub/ (@10.51.1.234) 1.02ms
14:39:30.352 [ConfigProxy] debug: Not recording activity for status 302 on /
14:39:30.387 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.253:8081
[W 2025-03-03 14:39:30.390 JupyterHub base:482] Invalid or expired cookie token
[I 2025-03-03 14:39:30.391 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@10.51.1.234) 1.94ms
14:39:30.392 [ConfigProxy] debug: Not recording activity for status 302 on /
14:39:30.425 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.253:8081
[I 2025-03-03 14:39:30.428 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'3b936d731d7d44efa35a49d25f8dc8bf:omr6k98sqP1-qL8-er-KW9VzL-3EUkujkoVlx-Faocg=' {'path': '/hub/', 'max_age': 3600}
[I 2025-03-03 14:39:30.507 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@10.51.1.234) 79.67ms
14:39:34.591 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.253:8081
[I 2025-03-03 14:39:35.049 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'3b936d731d7d44efa35a49d25f8dc8bf:d81ee580132f4487bcb7f8535185c4ed' {'path': '/hub/'}
[I 2025-03-03 14:39:35.050 JupyterHub base:973] User logged in: testuser
[I 2025-03-03 14:39:35.050 JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (testuser@10.51.1.234) 455.26ms
14:39:35.052 [ConfigProxy] debug: Not recording activity for status 302 on /
14:39:35.179 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.253:8081
[W 2025-03-03 14:39:35.251 JupyterHub spawner:179] 
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.
    
    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.
    
[I 2025-03-03 14:39:35.254 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (testuser@10.51.1.234) 73.00ms
14:39:35.255 [ConfigProxy] debug: Not recording activity for status 302 on /
14:39:35.294 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://192.168.1.253:8081
[I 2025-03-03 14:39:35.309 JupyterHub log:192] 200 GET /hub/spawn (testuser@10.51.1.234) 13.12ms
14:39:38.684 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://192.168.1.253:8081
[I 2025-03-03 14:39:38.897 JupyterHub provider:661] Creating oauth client jupyterhub-user-testuser
[I 2025-03-03 14:39:39.079 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'epilogue': '', 'homedir': '/data/home/testuser', 'host': '', 'keepvars': 'JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,JUPYTERHUB_DEBUG', 'keepvars_extra': '', 'memory': '4gb', 'ngpus': '', 'nprocs': '1', 'options': '', 'partition': '', 'prologue': '', 'queue': 'ccmq', 'runtime': '48:00:00', 'username': 'testuser', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser', 'profile': 'pbs1cores4gb'}
[I 2025-03-03 14:39:39.079 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u testuser qsub
[I 2025-03-03 14:39:39.176 JupyterHub batchspawner:287] Job submitted. output: 4079703.hpcfe01
[W 2025-03-03 14:39:39.693 JupyterHub base:201] Rolling back dirty objects IdentitySet([<Server(:0)>])
14:39:39.695 [ConfigProxy] debug: Not recording activity for status 302 on /
[I 2025-03-03 14:39:39.703 JupyterHub log:192] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/testuser?_xsrf=[secret] (testuser@10.51.1.234) 1008.25ms
14:39:39.901 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/testuser to http://192.168.1.253:8081
[I 2025-03-03 14:39:39.913 JupyterHub pages:397] testuser is pending spawn
[I 2025-03-03 14:39:39.921 JupyterHub log:192] 200 GET /hub/spawn-pending/testuser?_xsrf=[secret] (testuser@10.51.1.234) 17.87ms
14:39:40.071 [ConfigProxy] debug: PROXY WEB /hub/api/users/testuser/server/progress to http://192.168.1.253:8081
[I 2025-03-03 14:40:28.580 JupyterHub log:192] 200 POST /hub/api/batchspawner (testuser@192.168.1.13) 65.54ms
[I 2025-03-03 14:40:28.889 JupyterHub batchspawner:455] Notebook server job 4079703.hpcfe01 started at cn13:47576
[I 2025-03-03 14:40:59.737 JupyterHub log:192] 200 GET /hub/api (@192.168.1.13) 1.07ms
[I 2025-03-03 14:41:01.366 JupyterHub log:192] 200 POST /hub/api/users/testuser/activity (testuser@192.168.1.13) 136.66ms
[I 2025-03-03 14:41:05.206 JupyterHub base:1124] User testuser took 86.514 seconds to start
[I 2025-03-03 14:41:05.207 JupyterHub proxy:331] Adding user testuser to proxy /user/testuser/ => http://cn13:47576
14:41:05.209 [ConfigProxy] info: Adding route /user/testuser -> http://cn13:47576
14:41:05.210 [ConfigProxy] info: Route added /user/testuser -> http://cn13:47576
14:41:05.211 [ConfigProxy] info: 201 POST /api/routes/user/testuser 
[I 2025-03-03 14:41:05.211 JupyterHub users:899] Server testuser is ready
[I 2025-03-03 14:41:05.211 JupyterHub log:192] 200 GET /hub/api/users/testuser/server/progress?_xsrf=[secret] (testuser@10.51.1.234) 85137.74ms
14:41:05.505 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/testuser to http://192.168.1.253:8081
[I 2025-03-03 14:41:05.513 JupyterHub log:192] 302 GET /hub/spawn-pending/testuser?_xsrf=[secret] -> /user/testuser/ (testuser@10.51.1.234) 6.12ms
14:41:05.513 [ConfigProxy] debug: Not recording activity for status 302 on /
14:41:05.547 [ConfigProxy] debug: PROXY WEB /user/testuser/ to http://cn13:47576
14:41:05.564 [ConfigProxy] debug: Not recording activity for status 302 on /user/testuser
14:41:05.594 [ConfigProxy] debug: PROXY WEB /user/testuser/lab to http://cn13:47576
14:41:05.599 [ConfigProxy] debug: Not recording activity for status 302 on /user/testuser
14:41:05.635 [ConfigProxy] debug: PROXY WEB /hub/api/oauth2/authorize to http://192.168.1.253:8081
[I 2025-03-03 14:41:05.747 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-testuser&redirect_uri=%2Fuser%2Ftestuser%2Foauth_callback&response_type=code&state=[secret] -> /user/testuser/oauth_callback?code=[secret]&state=[secret] (testuser@10.51.1.234) 110.75ms
14:41:05.748 [ConfigProxy] debug: Not recording activity for status 302 on /
14:41:05.790 [ConfigProxy] debug: PROXY WEB /user/testuser/oauth_callback to http://cn13:47576
[I 2025-03-03 14:41:06.038 JupyterHub log:192] 200 POST /hub/api/oauth2/token (testuser@192.168.1.13) 242.28ms
[I 2025-03-03 14:41:06.165 JupyterHub log:192] 200 GET /hub/api/user (testuser@192.168.1.13) 124.23ms
14:41:06.167 [ConfigProxy] debug: Not recording activity for status 302 on /user/testuser
14:41:06.317 [ConfigProxy] debug: PROXY WEB /user/testuser/lab to http://cn13:47576
14:41:06.874 [ConfigProxy] debug: Not recording activity for status 500 on /user/testuser
14:41:06.944 [ConfigProxy] debug: PROXY WEB /favicon.ico to http://192.168.1.253:8081
^C[C 2025-03-03 14:42:37.590 JupyterHub app:3861] Received signal SIGINT, initiating shutdown...
[I 2025-03-03 14:42:37.590 JupyterHub app:3477] Leaving single-user servers running
[I 2025-03-03 14:42:37.590 JupyterHub proxy:865] Cleaning up proxy[25646]...
[I 2025-03-03 14:42:37.591 JupyterHub app:3502] ...done

The single user server should actually contact the hub on the 192.168.1.253 IP since this is the cluster private network used for the login to compute node traffic. The other IP (192.168.97.161) is on the “user” network.

Thanks,

Single-user server:

[I 2025-03-03 14:40:36.319 ServerApp] Extension package jupyter_lsp took 0.4795s to import
[I 2025-03-03 14:40:44.028 ServerApp] Extension package jupyter_server_terminals took 7.7082s to import
[I 2025-03-03 14:40:44.506 ServerApp] Extension package jupyterlab took 0.4764s to import
[I 2025-03-03 14:40:51.552 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2025-03-03 14:40:51.562 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2025-03-03 14:40:51.563 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.2.1
[I 2025-03-03 14:40:51.563 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2025-03-03 14:40:51.569 ServerApp] jupyterhub | extension was successfully linked.
[W 2025-03-03 14:40:51.573 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-03-03 14:40:51.579 ServerApp] jupyterlab | extension was successfully linked.
[I 2025-03-03 14:40:59.273 ServerApp] notebook_shim | extension was successfully linked.
[I 2025-03-03 14:40:59.686 ServerApp] notebook_shim | extension was successfully loaded.
[I 2025-03-03 14:40:59.691 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2025-03-03 14:40:59.694 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2025-03-03 14:40:59.739 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2025-03-03 14:40:59.739 ServerApp] jupyterhub | extension was successfully loaded.
[I 2025-03-03 14:40:59.775 LabApp] JupyterLab extension loaded from /data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterlab
[I 2025-03-03 14:40:59.775 LabApp] JupyterLab application directory is /data/software/mamba/24.11.3-0/envs/jupyterhub/share/jupyter/lab
[I 2025-03-03 14:40:59.782 LabApp] Extension Manager is 'pypi'.
[I 2025-03-03 14:41:01.177 ServerApp] jupyterlab | extension was successfully loaded.
[I 2025-03-03 14:41:01.179 ServerApp] Serving notebooks from local directory: /data/home/testuser
[I 2025-03-03 14:41:01.179 ServerApp] Jupyter Server 2.15.0 is running at:
[I 2025-03-03 14:41:01.179 ServerApp] http://cn13.cluster.loc:47576/user/testuser/lab?token=...
[I 2025-03-03 14:41:01.179 ServerApp]     http://127.0.0.1:47576/user/testuser/lab?token=...
[I 2025-03-03 14:41:01.179 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2025-03-03 14:41:02.329 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2025-03-03 14:41:05.207 ServerApp] 200 GET /user/testuser/api (@192.168.1.253) 1.94ms
[I 2025-03-03 14:41:05.564 ServerApp] 302 GET /user/testuser/ -> /user/testuser/lab? (@10.51.1.234) 0.88ms
[I 2025-03-03 14:41:05.599 ServerApp] 302 GET /user/testuser/lab -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-testuser&redirect_uri=%2Fuser%2Ftestuser%2Foauth_callback&response_type=code&state=[secret] (@10.51.1.234) 3.01ms
[I 2025-03-03 14:41:06.166 ServerApp] Logged-in user testuser
[I 2025-03-03 14:41:06.167 ServerApp] Setting new xsrf cookie for b'3b936d731d7d44efa35a49d25f8dc8bf:cff5786fee5cc4146b6985275b68ab72e7e15f11d379886e8933d025d22ca679' {'path': '/user/testuser/'}
[E 2025-03-03 14:41:06.169 ServerApp] Uncaught exception GET /user/testuser/oauth_callback?code=TjrW8nHDemaMdNvLLcty4RMCU1FBU8&state=cdYsk3XGok16H3RIn7BuPw (10.51.1.234)
    HTTPServerRequest(protocol='https', host='192.168.97.161', method='GET', uri='/user/testuser/oauth_callback?code=TjrW8nHDemaMdNvLLcty4RMCU1FBU8&state=cdYsk3XGok16H3RIn7BuPw', version='HTTP/1.1', remote_ip='10.51.1.234')
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/singleuser/extension.py", line 102, in get
        return await super().get()
               ^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 1617, in get
        self.redirect(next_url or self.hub_auth.base_url)
        ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 895, in redirect
        self.finish()
        ~~~~~~~~~~~^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1240, in finish
        self._log()
        ~~~~~~~~~^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1823, in _log
        self.application.log_request(self)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 2339, in log_request
        self.settings["log_function"](handler)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/log.py", line 160, in log_request
        user = handler.current_user
               ^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1424, in current_user
        self._current_user = self.get_current_user()
                             ~~~~~~~~~~~~~~~~~~~~~^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 1498, in get_current_user
        self._hub_auth_user_cache = self.check_hub_user(user_model)
                                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 1425, in check_hub_user
        if self.allow_all:
           ^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 1354, in allow_all
        self.hub_scopes is None
        ^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 1346, in hub_scopes
        return self.hub_auth.access_scopes or None
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
                         ~~~~~~~~^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 510, in _default_scopes
        return set(json.loads(env_scopes))
                   ~~~~~~~~~~^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
               ~~~~~~~~~~~~~~~~~~~~~~~^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 345, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 363, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
[E 2025-03-03 14:41:06.528 ServerApp] Cannot send error response after headers written
[I 2025-03-03 14:41:06.528 ServerApp] 302 GET /user/testuser/oauth_callback?code=[secret]&state=[secret] -> /user/testuser/lab (@10.51.1.234) 735.01ms
[E 2025-03-03 14:41:06.530 ServerApp] Uncaught exception GET /user/testuser/lab (10.51.1.234)
    HTTPServerRequest(protocol='https', host='192.168.97.161', method='GET', uri='/user/testuser/lab', version='HTTP/1.1', remote_ip='10.51.1.234')
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
                 ^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 624, in prepare
        _user = await _user
                ^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/singleuser/extension.py", line 176, in get_user
        f"Checking user {user['name']} with scopes {user['scopes']} against {self.hub_auth.access_scopes}"
                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
                         ~~~~~~~~^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 510, in _default_scopes
        return set(json.loads(env_scopes))
                   ~~~~~~~~~~^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
               ~~~~~~~~~~~~~~~~~~~~~~~^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 345, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 363, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
[E 2025-03-03 14:41:06.723 ServerApp] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
                 ^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 624, in prepare
        _user = await _user
                ^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/singleuser/extension.py", line 176, in get_user
        f"Checking user {user['name']} with scopes {user['scopes']} against {self.hub_auth.access_scopes}"
                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
                         ~~~~~~~~^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/services/auth.py", line 510, in _default_scopes
        return set(json.loads(env_scopes))
                   ~~~~~~~~~~^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
               ~~~~~~~~~~~~~~~~~~~~~~~^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 345, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/json/decoder.py", line 363, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/extension/handler.py", line 29, in get_template
        template = cast(Template, self.settings[env].get_template(name))  # type:ignore[attr-defined]
                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/environment.py", line 1016, in get_template
        return self._load_template(name, globals)
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/environment.py", line 975, in _load_template
        template = self.loader.load(self, name, self.make_globals(globals))
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/loaders.py", line 126, in load
        source, filename, uptodate = self.get_source(environment, name)
                                     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/loaders.py", line 209, in get_source
        raise TemplateNotFound(
        ...<2 lines>...
        )
    jinja2.exceptions.TemplateNotFound: '500.html' not found in search path: '/data/software/mamba/24.11.3-0/envs/jupyterhub/share/jupyter/lab/static'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 740, in write_error
        html = self.render_template("%s.html" % status_code, **ns)
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/extension/handler.py", line 93, in render_template
        template = cast(Template, self.get_template(name))  # type:ignore[attr-defined]
                                  ~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/extension/handler.py", line 32, in get_template
        return cast(Template, super().get_template(name))  # type:ignore[misc]
                              ~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 662, in get_template
        return self.settings["jinja2_env"].get_template(name)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/environment.py", line 1016, in get_template
        return self._load_template(name, globals)
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/environment.py", line 975, in _load_template
        template = self.loader.load(self, name, self.make_globals(globals))
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/loaders.py", line 126, in load
        source, filename, uptodate = self.get_source(environment, name)
                                     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jinja2/loaders.py", line 209, in get_source
        raise TemplateNotFound(
        ...<2 lines>...
        )
    jinja2.exceptions.TemplateNotFound: '500.html' not found in search paths: '/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server', '/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/templates'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1298, in send_error
        self.write_error(status_code, **kwargs)
        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 742, in write_error
        html = self.render_template("error.html", **ns)
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/extension/handler.py", line 94, in render_template
        ns.update(self.template_namespace)  # type:ignore[attr-defined]
                  ^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 676, in template_namespace
        logged_in=self.logged_in,
                  ^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 199, in logged_in
        user = self.current_user
               ^^^^^^^^^^^^^^^^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/tornado/web.py", line 1424, in current_user
        self._current_user = self.get_current_user()
                             ~~~~~~~~~~~~~~~~~~~~~^^
      File "/data/software/mamba/24.11.3-0/envs/jupyterhub/lib/python3.13/site-packages/jupyter_server/base/handlers.py", line 178, in get_current_user
        raise RuntimeError(msg)
    RuntimeError: Calling `LabHandler.get_current_user()` directly is deprecated in jupyter-server 2.0. Use `self.current_user` instead (works in all versions).
[E 2025-03-03 14:41:06.874 ServerApp] {
      "X-Forwarded-Host": "192.168.97.161",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Forwarded-For": "10.51.1.234",
      "Priority": "u=4",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "jupyterhub-user-testuser=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
      "Connection": "keep-alive",
      "Referer": "https://192.168.97.161/hub/spawn",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0",
      "Host": "192.168.97.161"
    }
[E 2025-03-03 14:41:06.874 ServerApp] 500 GET /user/testuser/lab (@10.51.1.234) 344.79ms

Sorry, I had to split the response into two separate messages because I hit the maximum word limit.

Thanks for the logs. Seems like you hitting this same issue. There are few workarounds suggested in that thread that you can try.

Hello Mahendra,

That was it, thanks for your help!

1 Like