599 Empty Reply from Server - oauth

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,

I’m having an issue with Jupyterhub and Oauth using Jupyter 1.1.3 deployed on K8s with helm
The problem is I can log in with some users and some other users are having “500 Internal Server Error” and when refreshing “400 OAuth state missing from cookies”
The related log is below:

[I 2021-09-23 13:33:15.183 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@10.172.142.121) 4.77ms
[I 2021-09-23 13:33:17.979 JupyterHub oauth2:111] OAuth redirect: 'https://jhub-dev.domain.fr/hub/oauth_callback'
[I 2021-09-23 13:33:17.982 JupyterHub log:189] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://auth.domain.fr/auth/realms/master/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fjhub-dev.domain.fr%2Fhub%2Foauth_callback&client_id=jupyterhub&state=[secret]&scope=ldap-groups (@10.172.142.121) 3.07ms
[E 2021-09-23 13:33:18.161 JupyterHub oauth2:389] Error fetching user data 599 GET https://auth.domain.fr/auth/realms/master/protocol/openid-connect/userinfo: HTTP 599: Empty reply from server
[E 2021-09-23 13:33:18.162 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?code=e5a40fe1-02bb-4b81-94a4-b488ebd02bca.09a2f2a1-f45a-473b-93ed-581466b325d2.feae45a6-94c2-45f8-83c6-80dc56b6c4e3&state=eyJzdGF0ZV9pZCI6ICI1MTUzOWU1Y2U1MTA0OGZjYjMwN2E2NDYyYzViODZmYSIsICJuZXh0X3VybCI6ICIvaHViLyJ9 (10.172.142.121)
    HTTPServerRequest(protocol='https', host='jhub-dev.domain.fr', method='GET', uri='/hub/oauth_callback?code=e5a40fe1-02bb-4b81-94a4-b488ebd02bca.09a2f2a1-f45a-473b-93ed-581466b325d2.feae45a6-94c2-45f8-83c6-80dc56b6c4e3&state=eyJzdGF0ZV9pZCI6ICI1MTUzOWU1Y2U1MTA0OGZjYjMwN2E2NDYyYzViODZmYSIsICJuZXh0X3VybCI6ICIvaHViLyJ9', version='HTTP/1.1', remote_ip='10.172.142.121')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 231, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 171, in authenticate
        user_data_resp_json = await self._get_user_data(token_resp_json)
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 390, in fetch
        raise e
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 369, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
    tornado.curl_httpclient.CurlError: HTTP 599: Empty reply from server
    
[E 2021-09-23 13:33:18.187 JupyterHub log:181] {
      "Cookie": "oauthenticator-state=[secret]",
      "Accept-Language": "en",
      "Accept-Encoding": "gzip, deflate, br",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "X-Original-Forwarded-For": "10.200.207.155, 127.0.0.1",
      "X-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "jhub.kubernetes",
      "X-Forwarded-For": "10.172.142.121,::ffff:192.168.64.1",
      "X-Real-Ip": "10.172.142.121",
      "X-Request-Id": "4a63923bea89a856f64688eeb5cd705a",
      "Host": "jhub-dev.domain.fr",
      "Connection": "close"
    }

The error seems to be related to Error fetching user data 599 GET https://auth.domain.fr/auth/realms/master/protocol/openid-connect/userinfo: HTTP 599: Empty reply from server which is my keycloak server but as it does work for other services and only some users are impacted I narrow down the issue to jupyterhub or maybe the oauth library.

If anyone has informations on how to solve this I would be really gratefull.
Sincerely.

2

Could you

  • turn on debug logging in JupyterHub
  • see if there’s anything in the Keycloak logs?
3

I am also getting the same error @Yayg do you remember the cause of issue?