generic.oauthenticator: 500 Internal Server Error / 400 OAuth state missing from cookies

Context:

  • I’m creating a Dashboard service where multiple users can access multiple notebook servers via JupyterHub .
  • Users log in using a separate Authentication service, and once in Dashboard , JupyterHub gets rendered inside an <iframe> .

Setup:

  • I’m hosting JupyterHub in a Kubernetes cluster on GCP. These are the relevant portions of my config.yaml :
...

 # Changes the Jupyter's notebook headers to allow it to be displayed
 # inside of an iframe object.
 c.JupyterHub.tornado_settings = {
        "headers": {
          "Content-Security-Policy": "default-src *  data: blob: filesystem: about: ws: wss: 'unsafe-inline' 'unsafe-eval' 'unsafe-dynamic'; script-src * data: blob: 'unsafe-inline' 'unsafe-eval'; connect-src * data: blob: 'unsafe-inline'; img-src * data: blob: 'unsafe-inline'; frame-src * data: blob: ; style-src * data: blob: 'unsafe-inline'; font-src * data: blob: 'unsafe-inline';",
        }
      }

...

  extraEnv:
    #Endpoints
    OAUTH2_AUTHORIZE_URL: "http://35.232.66.179:8000/o/authorize"
    OAUTH2_TOKEN_URL: "http://35.232.66.179:8000/o/token"

    # Jupyterhub callback (default: https://<host>/hub/oauth_callback)
    OAUTH_CALLBACK_URL: "http://35.196.54.180/hub/oauth_callback"

    # User information is used to name server created by kubespawner.
    OAUTH2_USERDATA_URL: "http://35.232.66.179:8000/userdata"
    OAUTH2_USERNAME_KEY: "username"
    OAUTH2_USERDATA_METHOD: "GET"

    # General settings
    OAUTH2_BASIC_AUTH: "false"
    OAUTH2_TLS_VERIFY: "true"

...

auth:
  type: custom
  custom:
    className: oauthenticator.generic.GenericOAuthenticator
    config:
      login_service: "OpenID" 
      client_id: "<client_id>"
      client_secret: "<client_secret>"

Errors

However, when I try to sign in just via my JupyterHub endpoint I get: 500 Internal Server Error

logs:

IwN2YyNzVhNTk0MzM0ZTM5YmIyZTdmZDQzY2U2ZDI3YyIsICJuZXh0X3VybCI6IG51bGx9&session_state=641fb9aca6490d2c3332871d72ee0639b3ed46b8a9f02c363ee8f7bf46f1c989.74e37d534296
cad45421019d1d442153', version='HTTP/1.1', remote_ip='<>')

    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/oauthenticator/oauth2.py", line 213, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 699, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.6/dist-packages/oauthenticator/generic.py", line 136, in authenticate
        resp_json = json.loads(resp.body.decode('utf8', 'replace'))
      File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 5 column 1 (char 4)

I know it actually does redirect to the Authentication Server and the callback is correct. Even still when I attempt to access via Dashboard , within the <iframe> I get:

400 OAuth state missing from cookies

logs:

400 GET /hub/oauth_callback?code=d37de16a855a49dda94fb65c2c2c8ffa&state=eyJzdGF0ZV9pZCI6ICIwN2YyNzVhNTk0MzM0ZTM5Ym
IyZTdmZDQzY2U2ZDI3YyIsICJuZXh0X3VybCI6IG51bGx9&session_state=641fb9aca6490d2c3332871d72ee0639b3ed46b8a9f02c363ee8f7bf46f1c989.74e37d534296cad45421019d1d442153

OAuth state missing from cookies

Reference

I previously used the Github OAuthenticator class and registered an OAuth app within Github and got JupyterHub to work fine. However, Github does not allow their apps to be used within an <iframe> so I had to go the generic route.

Point is, the setup was very similar and at least had JupyterHub working outside of Dashboard , so I’m really stumped on what else I could be missing. Any help is appreciated.

Have you tried looking at your browser’s developer console for any warning messages during the authentication process?

Yeah the only warning messages I got were similar to what I described:

Within Dashboard checking out the network messages gave me “State missing from cookies”.

Attempting login through JupyterHub itself I get no indication just the “500 Internal Server” Error.

If you configure JupyterHub to use either the default or DummyAuthenticator does it work inside the iframe?

That was part of the issue we were trying to solve. Outside of Dashboard using the dummy authenticator works fine but when I attempted to sign in using the dummy authenticator via Dashboard, I noticed no session cookies were being created. It made it seem like the page was just “refreshing” but It was failing altogether.

OK, this means we can focus on JupyterHub for now. It’s possible there’s also a problem with the external authenticator but that can be dealt with seperately.

When you’re looking at your browser console do you have Persist Logs or the equivalent enabled? If you’re being redirected between pages your console and/or network requests may be cleared for each page, is it possible some errors or warnings were missed?

Thanks again for all the responses btw. So I’ve enabled Persist Logs within chrome and I notice a few errors I think I missed.

The one hat stands out the most to me is:
Uncaught TypeError: Cannot read property ‘className’ of null

For reference here is the relevant portion of my config.yaml file:

  extraEnv:
    OAUTH2_AUTHORIZE_URL: <url>/o/authorize
    OAUTH2_TOKEN_URL: <url>/o/token
    OAUTH_CALLBACK_URL: <jupyterhub_endpoint>/hub/oauth_callback

# Authentication configuration for JupyterHub specifically.
auth:
  type: custom
  custom:
    className: oauthenticator.generic.GenericOAuthenticator
    config:
      login_service: "Service"
      client_id: "<secret>"
      client_secret: "<somemoresecrecy>"
      token_url: <url>/o/token
      userdata_url: <url>/o/userinfo
      userdata_method: GET
      userdata_params: { "state": "state" }
      username_key: preferred_username

It looks like your Content Security Policy has some errors, or it uses some values that your browser doesn’t recognise. If that results in some scripts being blocked it could result in a wide variety of errors.

I see. I did have to setup some custom content security settings because of the iframe. I’ll show the other relevant portions to see if maybe I messed up something there.

      # Changes the Jupyter's notebook headers to allow it to be displayed
      # inside of an iframe object.
      c.JupyterHub.tornado_settings = {
        "headers": {
          "Content-Security-Policy": "default-src *  data: blob: filesystem: about: ws: wss: 'unsafe-inline' 'unsafe-eval' 'unsafe-dynamic'; script-src * data: blob: 'unsafe-inline' 'unsafe-eval'; connect-src * data: blob: 'unsafe-inline'; img-src * data: blob: 'unsafe-inline'; frame-src * data: blob: ; style-src * data: blob: 'unsafe-inline'; font-src * data: blob: 'unsafe-inline';",
        },
        "cookie_options": {
          "SameSite": "None", 
          "Secure": True},
      }

However, these settings were basically identical to the setup I had with github oauthenticator and that was at least working with JupyterHub by itself. I’ll try testing it without these settings.

Update: I commented out those settings entirely and updated the hub but it still shows the same errors.

This seems to be related to the Set-Cookie header not including a SameSite=none.
You’ll likely observe that this fails for chrome > 80. Looking further into this as i’m having similar issues…

I put together a hacky patch for this if you’re still interested. Its unlikely that this’d be merged into the build. It essentially involves some small changes to the base authenticator

diff --git a/jupyterhub/handlers/base.py b/jupyterhub/handlers/base.py
index 72e0096e..85900588 100644
--- a/jupyterhub/handlers/base.py
+++ b/jupyterhub/handlers/base.py
@@ -536,6 +536,7 @@ class BaseHandler(RequestHandler):
         if self.subdomain_host:
             kwargs['domain'] = self.domain
 
+        kwargs['SameSite'] = 'None'
         kwargs.update(self.settings.get('cookie_options', {}))
         kwargs.update(overrides)

i’ll hopefully put together something that enables this flag to be toggled sometime soon.

Hi @harrylepotter-win ,
I’m having a similar issue, although i dont have Jupterhub in an iframe, I have a identity provider who is authenticating and then redirecting to jupyterhub while both are on different servers. I get following:

Sep 01 08:36:21 xmaidesigner python3[10620]:     HTTPServerRequest(protocol='https', host='xmaidesigner.southeastasia.cloudapp.azure.com', method='GET', uri='/hub/oauth_callback?code=da75380b9e2925239e86999b7279bab3&state=eyJzdGF0ZV9pZCI6ICI0MmM2OWI0NTZiODA0NmE0YjAxYzFiMjBhYTEzZDRkZCIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=SC-Ia1IrsOUoFoyqO5I8jZMvvB05VX4JOIcWJrHXb5k.c0065a77ee7a7d6408fcb58b17877cf8', version='HTTP/1.1', remote_ip='115.70.35.46')
Sep 01 08:36:21 xmaidesigner python3[10620]:     Traceback (most recent call last):
Sep 01 08:36:21 xmaidesigner python3[10620]:       File "/opt/tljh/hub/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
Sep 01 08:36:21 xmaidesigner python3[10620]:         result = await result
Sep 01 08:36:21 xmaidesigner python3[10620]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 207, in get
Sep 01 08:36:21 xmaidesigner python3[10620]:         user = await self.login_user()
Sep 01 08:36:21 xmaidesigner python3[10620]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 754, in login_user
Sep 01 08:36:21 xmaidesigner python3[10620]:         authenticated = await self.authenticate(data)
Sep 01 08:36:21 xmaidesigner python3[10620]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
Sep 01 08:36:21 xmaidesigner python3[10620]:         authenticated = await maybe_future(self.authenticate(handler, data))
Sep 01 08:36:21 xmaidesigner python3[10620]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/generic.py", line 168, in authenticate
Sep 01 08:36:21 xmaidesigner python3[10620]:         resp = await http_client.fetch(req)
Sep 01 08:36:21 xmaidesigner python3[10620]:     tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
Sep 01 08:36:21 xmaidesigner python3[10620]:
Sep 01 08:36:21 xmaidesigner python3[10620]: [E 2021-09-01 08:36:21.915 JupyterHub log:181] {
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Host": "xmaidesigner.southeastasia.cloudapp.azure.com",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Accept-Encoding": "gzip, deflate, br",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Accept-Language": "en-US,en;q=0.9",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Cache-Control": "max-age=0",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Cookie": "oauthenticator-state=[secret]",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Referer": "https://onxmprodev.azurewebsites.net/",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Ch-Ua": "\"Chromium\";v=\"92\", \" Not A;Brand\";v=\"99\", \"Google Chrome\";v=\"92\"",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Ch-Ua-Mobile": "?0",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Fetch-Dest": "document",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Fetch-Mode": "navigate",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Fetch-Site": "cross-site",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Sec-Fetch-User": "?1",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "Upgrade-Insecure-Requests": "1",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Forwarded-For": "115.70.35.46",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Forwarded-Host": "xmaidesigner.southeastasia.cloudapp.azure.com",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Forwarded-Port": "443",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Forwarded-Proto": "https",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Forwarded-Server": "xmaidesigner",
Sep 01 08:36:21 xmaidesigner python3[10620]:       "X-Real-Ip": "115.70.35.46"
Sep 01 08:36:21 xmaidesigner python3[10620]:     }

You think your solution above will solve this too?

@malachisti that one looks like a 403 coming from your Authentication provider. First thing I’d do is check out if there’s a setting for allowed origins (had a similar problem on my side using keycloak as the auth provider)

Then it’s probably worth checking that all of your Jhub oauth2 Auth realm settings are lining up properly.