OIDC Issue (Keycloak) - JH K8s (Helm chart) behind reverse-proxy

JupyterHub
1

When I configure JupyterHub Helm chart with the following configuration:
The JupyterHub is behind a reverse proxy (NginxProxyManager), and it is exposed via HTTP NodePort.
In the main page of JH I can click on Keycloak authentication, I enter username and password, and when returning to the JH (oauth_callback_url); the page shows 500: Internal Server Error. How can I fix it?

hub:
  revisionHistoryLimit:
  config:
    JupyterHub:
      admin_access: true
      authenticator_class: generic-oauth
    GenericOAuthenticator:
      client_id: jupyterhub
      client_secret: "XXXXXXXXXX"
      oauth_callback_url: "https://jh-url.com/hub/oauth_callback"
      authorize_url: "https://keycloak-url.com/realms/jupyter/protocol/openid-connect/auth"
      token_url: "https://keycloak-url.com/realms/jupyter/protocol/openid-connect/token"
      userdata_url: "https://keycloak-url.com/realms/jupyter/protocol/openid-connect/userinfo"
      username_claim: "email"  # Adjust based on your provider
      scope: ["openid", "email", "profile"]
      login_service: "Keycloak"
2

Please can you enable debug logging and show us your full hub logs?

3

Thanks; I got the logs via kubectl logs hub-xxxxx :
please note that the NginxProxyManager traffic is directed to the Kubernetes cluster via a VPN.

    
[I 2025-06-03 10:49:34.164 JupyterHub app:3416] Initialized 0 spawners in 0.005 seconds
[I 2025-06-03 10:49:34.168 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-06-03 10:49:34.169 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-06-03 10:49:34.170 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-06-03 10:49:34.170 JupyterHub app:3703] Not starting proxy
[D 2025-06-03 10:49:34.170 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:49:34.175 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-06-03 10:49:34.176 JupyterHub app:3739] Hub API listening on http://:8081/hub/
[I 2025-06-03 10:49:34.176 JupyterHub app:3741] Private Hub API connect url http://hub:8081/hub/
[I 2025-06-03 10:49:34.176 JupyterHub app:3615] Starting managed service jupyterhub-idle-culler
[I 2025-06-03 10:49:34.176 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2025-06-03 10:49:34.176 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2025-06-03 10:49:34.177 JupyterHub spawner:1475] Polling subprocess every 30s
[D 2025-06-03 10:49:34.177 JupyterHub proxy:389] Fetching routes to check
[D 2025-06-03 10:49:34.177 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:49:34.178 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-06-03 10:49:34.178 JupyterHub proxy:392] Checking routes
[I 2025-06-03 10:49:34.178 JupyterHub proxy:477] Adding route for Hub: / => http://hub:8081
[D 2025-06-03 10:49:34.178 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/
[I 2025-06-03 10:49:34.179 JupyterHub app:3772] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[D 2025-06-03 10:49:34.180 JupyterHub app:3339] It took 1.439 seconds for the Hub to start
[D 2025-06-03 10:49:34.316 JupyterHub base:366] Recording first activity for <APIToken('a69d...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2025-06-03 10:49:34.357 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 43.91ms
[D 2025-06-03 10:49:34.360 JupyterHub scopes:1010] Checking access to /hub/api/users via scope list:users
[I 2025-06-03 10:49:34.367 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 9.32ms
[D 2025-06-03 10:49:35.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:49:37.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[I 2025-06-03 10:49:39.032 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.176.13) 0.46ms
[I 2025-06-03 10:49:39.086 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.176.13) 0.47ms
[I 2025-06-03 10:49:39.139 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:to5KmHZDiaUmeo7-XIu8o9Lqq4D9ZDSHHrwHWMsDUn4=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:49:39.154 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.176.13) 15.50ms
[D 2025-06-03 10:49:39.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[D 2025-06-03 10:49:41.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.33ms
[D 2025-06-03 10:49:43.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:49:45.635 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:47.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:49:49.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:51.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.34ms
[D 2025-06-03 10:49:53.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:49:55.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:49:57.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:49:59.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:01.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.46ms
[D 2025-06-03 10:50:03.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:05.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.40ms
[D 2025-06-03 10:50:07.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.43ms
[D 2025-06-03 10:50:09.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:11.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:13.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:15.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:17.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.34ms
[D 2025-06-03 10:50:19.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:21.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.45ms
[D 2025-06-03 10:50:23.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:25.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:27.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:29.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms
[D 2025-06-03 10:50:31.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.46ms
[I 2025-06-03 10:50:31.873 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.180.5) 0.39ms
[I 2025-06-03 10:50:31.898 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.180.5) 0.43ms
[I 2025-06-03 10:50:31.937 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:2F1efdEIIG_0gYLL7SnJ7V4BW6wraqk7a3cEjAUXx10=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:50:31.938 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.180.5) 1.24ms
[D 2025-06-03 10:50:32.000 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@145.5.180.5) 0.54ms
[I 2025-06-03 10:50:33.493 JupyterHub oauth2:126] OAuth redirect: https://jh-k8s.ecda.ai/hub/oauth_callback
[D 2025-06-03 10:50:33.493 JupyterHub base:681] Setting cookie oauthenticator-state: {'httponly': True, 'expires_days': 1}
[I 2025-06-03 10:50:33.494 JupyterHub log:192] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://keycloak.ecda.ai/realms/jupyter/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fjh-k8s.ecda.ai%2Fhub%2Foauth_callback&client_id=jupyterhub&code_challenge=[secret]&code_challenge_method=[secret]&state=[secret]&scope=openid+email+profile (@145.5.180.5) 0.88ms
[D 2025-06-03 10:50:33.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.33ms
[D 2025-06-03 10:50:34.181 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-06-03 10:50:34.184 JupyterHub proxy:392] Checking routes
[D 2025-06-03 10:50:35.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:37.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[I 2025-06-03 10:50:39.230 JupyterHub log:192] 302 GET / -> /hub/ (@145.5.176.13) 0.41ms
[I 2025-06-03 10:50:39.285 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@145.5.176.13) 0.43ms
[I 2025-06-03 10:50:39.337 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:to5KmHZDiaUmeo7-XIu8o9Lqq4D9ZDSHHrwHWMsDUn4=' {'path': '/hub/', 'max_age': 3600}
[I 2025-06-03 10:50:39.338 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@145.5.176.13) 1.10ms
[D 2025-06-03 10:50:39.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:41.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.40ms
[E 2025-06-03 10:50:41.732 JupyterHub oauth2:857] Error fetching 401 POST https://keycloak.ecda.ai/realms/jupyter/protocol/openid-connect/token: {
     "error": "unauthorized_client",
     "error_description": "Invalid client or Invalid client credentials"
    }
[E 2025-06-03 10:50:41.732 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwNTc2Mjk2MzEyNDg0NmYyYTM4OTI3OGRmM2U4YTQ1YiJ9&session_state=86e52d5a-c098-4fe7-8f9d-e517ccbeaf20&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=80527c3e-9eab-49d7-8150-7cd07511ae24.86e52d5a-c098-4fe7-8f9d-e517ccbeaf20.bc63d3a4-fd8e-4dd6-a16d-674f5c917efe (145.5.180.5)
    HTTPServerRequest(protocol='http', host='jh-k8s.ecda.ai', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwNTc2Mjk2MzEyNDg0NmYyYTM4OTI3OGRmM2U4YTQ1YiJ9&session_state=86e52d5a-c098-4fe7-8f9d-e517ccbeaf20&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=80527c3e-9eab-49d7-8150-7cd07511ae24.86e52d5a-c098-4fe7-8f9d-e517ccbeaf20.bc63d3a4-fd8e-4dd6-a16d-674f5c917efe', version='HTTP/1.1', remote_ip='145.5.180.5')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 245, in get
        user = await self.login_user()
               ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyterhub/handlers/base.py", line 964, in login_user
        authenticated = await self.authenticate(data)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyterhub/auth.py", line 688, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1316, in authenticate
        token_info = await self.get_token_info(handler, access_token_params)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1092, in get_token_info
        token_info = await self.httpfetch(
                     ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 892, in httpfetch
        return await self.fetch(
               ^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 858, in fetch
        raise e
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 837, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    tornado.httpclient.HTTPClientError: HTTP 401: Unauthorized
    
[D 2025-06-03 10:50:41.734 JupyterHub base:1519] Using default error template for 500
[E 2025-06-03 10:50:41.744 JupyterHub log:184] {
      "X-Forwarded-Host": "jh-k8s.ecda.ai",
      "X-Forwarded-Port": "80",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Priority": "u=0, i",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Sec-Ch-Ua-Platform": "\"macOS\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Chromium\";v=\"136\", \"Brave\";v=\"136\", \"Not.A/Brand\";v=\"99\"",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-site",
      "Accept-Language": "en-GB,en;q=0.5",
      "Sec-Gpc": "1",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "X-Real-Ip": "145.5.180.5",
      "X-Forwarded-For": "145.5.180.5,::ffff:10.244.6.244",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Scheme": "https",
      "Host": "jh-k8s.ecda.ai",
      "Connection": "keep-alive"
    }
[E 2025-06-03 10:50:41.744 JupyterHub log:192] 500 GET /hub/oauth_callback?state=[secret]&session_state=[secret]&iss=https%3A%2F%2Fkeycloak.ecda.ai%2Frealms%2Fjupyter&code=[secret] (@145.5.180.5) 81.13ms
[D 2025-06-03 10:50:43.637 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:45.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.37ms
[D 2025-06-03 10:50:47.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.39ms
[D 2025-06-03 10:50:49.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.44ms
[D 2025-06-03 10:50:51.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.41ms
[D 2025-06-03 10:50:53.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.38ms
[D 2025-06-03 10:50:55.636 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms
[D 2025-06-03 10:50:57.635 JupyterHub log:192] 200 GET /hub/health (@10.244.3.162) 0.35ms
4

Looking forward for your input! Thanks.

5

This line clearly indicates that there is a problem with the client configuration. Double check with Keycloak if cliend_id and client_secret are correctly configured on JupyterHub.

1 Like
6

thanks that was correct!

There’s another issue that I’m facing; when authenticating via Keycloak; I get the following error:

500 : Internal Server Error and after refreshing the page: 400 : Bad Request OAuth state missing from cookies

What could be the issue?
Please note that the service is behind NginxProxyManager, and my third party SSO is connected to Keycloak.
Looking forward

7

Assuming you’ve fixed the client_id and client_secret, and restarted JupyterHub, can you share your full hub logs again?

1 Like