Connection died after OAuth redirect

JupyterHub Zero to JupyterHub on Kubernetes
1

Hey there,

I’m using the authenticator GenericOAuthenticator to authenticate users.
Now, I’m facing the strange issue that sometimes the redirect from the identity provider throws the following error, and the 500 error page is shown.

[E 2025-03-21 14:16:11.720 JupyterHub oauth2:857] Error fetching 599 POST https://<stripped>/protocol/openid-connect/token: HTTP 599: Connection died, tried 5 times before giving up
[E 2025-03-21 14:16:11.721 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?code=<stripped>&state=<stripped> (<stripped>)
    HTTPServerRequest(protocol='http', host='<stripped>', method='GET', uri='/hub/oauth_callback?code=<stripped>&state=<stripped>', version='HTTP/1.1', remote_ip='<stripped>')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 245, in get
        user = await self.login_user()
               ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyterhub/handlers/base.py", line 964, in login_user
        authenticated = await self.authenticate(data)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyterhub/auth.py", line 688, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1316, in authenticate
        token_info = await self.get_token_info(handler, access_token_params)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 1092, in get_token_info
        token_info = await self.httpfetch(
                     ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 892, in httpfetch
        return await self.fetch(
               ^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 858, in fetch
        raise e
      File "/usr/local/lib/python3.12/site-packages/oauthenticator/oauth2.py", line 837, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    tornado.curl_httpclient.CurlError: HTTP 599: Connection died, tried 5 times before giving up

[D 2025-03-21 14:16:11.721 JupyterHub base:1519] Using default error template for 500
[E 2025-03-21 14:16:11.722 JupyterHub log:184] {
      "X-Forwarded-For": "<stripped>",
      "Forwarded": "for=<stripped>;host=<stripped>;proto=https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "<stripped>",
      "Host": "<stripped>",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Referer": "https://idp.scc.kit.edu/",
      "Sec-Ch-Ua-Platform": "\"Windows\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Chromium\";v=\"134\", \"Not:A-Brand\";v=\"24\", \"Google Chrome\";v=\"134\"",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Connection": "keep-alive"
    }
[E 2025-03-21 14:16:11.722 JupyterHub log:192] 500 GET /hub/oauth_callback?code=[secret]&state=[secret] (<stripped>) 3.61ms

After reloading the page, the login page is shown. After clicking on the login button that redirected me to the identity provider before, I’m sent to the JupyterHub, I’m logged in, and everything works just fine.

Since the problem does not occur every time, it’s hard to say where the issue is.

Relevant Configuration

hub:
  config:
    GenericOAuthenticator:
      allow_all: true
      client_id: <stripped>
      # Client secret is set in jupyterhub_config.py
      oauth_callback_url: <stripped>/hub/oauth_callback
      authorize_url: <stripped>
      token_url: <stripped>
      userdata_url: <stripped>
      username_claim: preferred_username
      userdata_params:
        state: state
      enable_auth_state: true
      # Disable auth refresh due to the error
      # "No preferred_username found in {'error_description': 'Access denied by resource owner or authorization server',
      # 'error': 'access_denied'}. Maybe the hub needs to be configured to request more scopes?"
      auth_refresh_age: 0

Did someone already face the issue?

Best regads,
Paul

2

It sounds like they’re bath be a transient network problem between the hub and your identity provider, or perhaps your identity provider has temporarily gone down. Do you have any logging or monitoring to verify your provider is always accessible? If you have access to any kubernetes/network monitoring it’d be worth checking that too. Another option when this next happens is to exec into the hub pod and see if you can make a request using curl at the same time the existing request is failing.

1 Like
3

Thank you for providing some help, although this issue most likely depends on our environment!

Do you have any logging or monitoring to verify your provider is always accessible?

Yes, the IdP did not receive the request. Thus, it somehow gets lost between the pod, OpenShift application net, and the gateway.

Thanks to the heavy-configurable Jupyter eco-system, I can debug every outgoing request using the following configuration snippet.

import pycurl


def prepare(curl: pycurl.Curl):
    def debug(debug_type, debug_msg):
        print("debug(%d): %s" % (debug_type, debug_msg))

    curl.setopt(pycurl.VERBOSE, True)
    curl.setopt(pycurl.DEBUGFUNCTION, debug)


c.GenericOAuthenticator.http_request_kwargs = {"prepare_curl_callback": prepare}

Hopefully, it will provide further insights into this issue.