Thanks; I got the logs via kubectl logs hub-xxxxx
:
please note that the NginxProxyManager traffic is directed to the Kubernetes cluster via a VPN.
[I 2025-06-03 10:49:34.164 JupyterHub app:3416] Initialized 0 spawners in 0.005 seconds
[I 2025-06-03 10:49:34.168 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-06-03 10:49:34.169 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-06-03 10:49:34.170 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-06-03 10:49:34.170 JupyterHub app:3703] Not starting proxy
[D 2025-06-03 10:49:34.170 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:49:34.175 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-06-03 10:49:34.176 JupyterHub app:3739] Hub API listening on http://:8081/hub/
[I 2025-06-03 10:49:34.176 JupyterHub app:3741] Private Hub API connect url http://hub:8081/hub/
[I 2025-06-03 10:49:34.176 JupyterHub app:3615] Starting managed service jupyterhub-idle-culler
[I 2025-06-03 10:49:34.176 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2025-06-03 10:49:34.176 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2025-06-03 10:49:34.177 JupyterHub spawner:1475] Polling subprocess every 30s
[D 2025-06-03 10:49:34.177 JupyterHub proxy:389] Fetching routes to check
[D 2025-06-03 10:49:34.177 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:49:34.178 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-06-03 10:49:34.178 JupyterHub proxy:392] Checking routes
[I 2025-06-03 10:49:34.178 JupyterHub proxy:477] Adding route for Hub: / => http://hub:8081
[D 2025-06-03 10:49:34.178 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/
[I 2025-06-03 10:49:34.179 JupyterHub app:3772] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[D 2025-06-03 10:49:34.180 JupyterHub app:3339] It took 1.439 seconds for the Hub to start
[D 2025-06-03 10:49:34.316 JupyterHub base:366] Recording first activity for <APIToken('a69d...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2025-06-03 10:49:34.357 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 43.91ms
[D 2025-06-03 10:49:34.360 JupyterHub scopes:1010] Checking access to /hub/api/users via scope list:users
[I 2025-06-03 10:49:34.367 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 9.32ms
[D 2025-06-03 10:49:35.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:49:37.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[I 2025-06-03 10:49:39.032 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.176.13) 0.46ms
[I 2025-06-03 10:49:39.086 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.176.13) 0.47ms
[I 2025-06-03 10:49:39.139 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:to5KmHZDiaUmeo7-XIu8o9Lqq4D9ZDSHHrwHWMsDUn4=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:49:39.154 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.176.13) 15.50ms
[D 2025-06-03 10:49:39.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[D 2025-06-03 10:49:41.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.33ms
[D 2025-06-03 10:49:43.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:49:45.635 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:47.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:49:49.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:51.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.34ms
[D 2025-06-03 10:49:53.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:55.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:49:57.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:49:59.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:01.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.46ms
[D 2025-06-03 10:50:03.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:05.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.40ms
[D 2025-06-03 10:50:07.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[D 2025-06-03 10:50:09.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:11.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:13.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:15.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:17.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.34ms
[D 2025-06-03 10:50:19.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:21.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:50:23.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:25.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:27.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:29.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms
[D 2025-06-03 10:50:31.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.46ms
[I 2025-06-03 10:50:31.873 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.180.5) 0.39ms
[I 2025-06-03 10:50:31.898 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.180.5) 0.43ms
[I 2025-06-03 10:50:31.937 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:2F1efdEIIG_0gYLL7SnJ7V4BW6wraqk7a3cEjAUXx10=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:50:31.938 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.180.5) 1.24ms
[D 2025-06-03 10:50:32.000 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@145.5.180.5) 0.54ms
[I 2025-06-03 10:50:33.493 JupyterHub oauth2:126] OAuth redirect: https://jh-k8s.ecda.ai/hub/oauth_callback
[D 2025-06-03 10:50:33.493 JupyterHub base:681] Setting cookie oauthenticator-state: {'httponly': True, 'expires_days': 1}
[I 2025-06-03 10:50:33.494 JupyterHub log:192] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://keycloak.ecda.ai/realms/jupyter/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fjh-k8s.ecda.ai%2Fhub%2Foauth_callback&client_id=jupyterhub&code_challenge=[secret]&code_challenge_method=[secret]&state=[secret]&scope=openid+email+profile (@145.5.180.5) 0.88ms
[D 2025-06-03 10:50:33.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.33ms
[D 2025-06-03 10:50:34.181 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:50:34.184 JupyterHub proxy:392] Checking routes
[D 2025-06-03 10:50:35.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:37.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[I 2025-06-03 10:50:39.230 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.176.13) 0.41ms
[I 2025-06-03 10:50:39.285 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.176.13) 0.43ms
[I 2025-06-03 10:50:39.337 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:to5KmHZDiaUmeo7-XIu8o9Lqq4D9ZDSHHrwHWMsDUn4=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:50:39.338 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.176.13) 1.10ms
[D 2025-06-03 10:50:39.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:41.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.40ms
[E 2025-06-03 10:50:41.732 JupyterHub oauth2:857] Error fetching 401 POST https://keycloak.ecda.ai/realms/jupyter/protocol/openid-connect/token: {
"error": "unauthorized_client",
"error_description": "Invalid client or Invalid client credentials"
}
[E 2025-06-03 10:50:41.732 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwNTc2Mjk2MzEyNDg0NmYyYTM4OTI3OGRmM2U4YTQ1YiJ9&session_state=86e52d5a-c098-4fe7-8f9d-e517ccbeaf20&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=80527c3e-9eab-49d7-8150-7cd07511ae24.86e52d5a-c098-4fe7-8f9d-e517ccbeaf20.bc63d3a4-fd8e-4dd6-a16d-674f5c917efe (145.5.180.5)
HTTPServerRequest(protocol='http', host='jh-k8s.ecda.ai', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwNTc2Mjk2MzEyNDg0NmYyYTM4OTI3OGRmM2U4YTQ1YiJ9&session_state=86e52d5a-c098-4fe7-8f9d-e517ccbeaf20&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=80527c3e-9eab-49d7-8150-7cd07511ae24.86e52d5a-c098-4fe7-8f9d-e517ccbeaf20.bc63d3a4-fd8e-4dd6-a16d-674f5c917efe', version='HTTP/1.1', remote_ip='145.5.180.5')
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
result = await result
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 245, in get
user = await self.login_user()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jupyterhub/handlers/base.py", line 964, in login_user
authenticated = await self.authenticate(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jupyterhub/auth.py", line 688, in get_authenticated_user
authenticated = await maybe_future(self.authenticate(handler, data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1316, in authenticate
token_info = await self.get_token_info(handler, access_token_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1092, in get_token_info
token_info = await self.httpfetch(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 892, in httpfetch
return await self.fetch(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 858, in fetch
raise e
File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 837, in fetch
resp = await self.http_client.fetch(req, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.httpclient.HTTPClientError: HTTP 401: Unauthorized
[D 2025-06-03 10:50:41.734 JupyterHub base:1519] Using default error template for 500
[E 2025-06-03 10:50:41.744 JupyterHub log:184] {
"X-Forwarded-Host": "jh-k8s.ecda.ai",
"X-Forwarded-Port": "80",
"Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
"Priority": "u=0, i",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Sec-Ch-Ua-Platform": "\"macOS\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua": "\"Chromium\";v=\"136\", \"Brave\";v=\"136\", \"Not.A/Brand\";v=\"99\"",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-User": "?1",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "same-site",
"Accept-Language": "en-GB,en;q=0.5",
"Sec-Gpc": "1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
"Upgrade-Insecure-Requests": "1",
"Cache-Control": "max-age=0",
"X-Real-Ip": "145.5.180.5",
"X-Forwarded-For": "145.5.180.5,::ffff:10.244.6.244",
"X-Forwarded-Proto": "https,http",
"X-Forwarded-Scheme": "https",
"Host": "jh-k8s.ecda.ai",
"Connection": "keep-alive"
}
[E 2025-06-03 10:50:41.744 JupyterHub log:192] 500 GET /hub/oauth_callback?state=[secret]&session_state=[secret]&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=[secret] (@145.5.180.5) 81.13ms
[D 2025-06-03 10:50:43.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:45.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.37ms
[D 2025-06-03 10:50:47.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:49.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:51.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:53.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:55.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms
[D 2025-06-03 10:50:57.635 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms