400 OAuth state mismatch or OAuth state missing from cookies on login

Running JupyterHub on Kubernetes through GKE.

JupyterHub: 1.4.1
Helm chart: 1.0.1

When I login, I get a 400: Bad Request error which either states:
OAuth state missing from cookies
or
OAuth state mismatch

If I click the JupyterHub logo in the top left, I am taken to JupyterHub and able to open a new notebook. And after I have logged in once, I am able to logout and log back in without getting the error.

I originally ran JupyterHub with 0.11 of the helm chart, but updated it to 1.0.1 which did not fix the issue.

I have tried using both the Auth0OAuthenticator and the GenericOAuthenticator configured both as follows:

hub:
  config:
    Auth0OAuthenticator:
      client_id: client-id
      client_secret: client-secret
      oauth_callback_url: https://my-jupyter-url/hub/oauth_callback
      scope:
        - openid
        - email
      auth0_subdomain: my-auth0-url
    Authenticator:
      admin_users:
        - devops@example.com
      auto_login: true
    JupyterHub:
      authenticator_class: auth0
hub:
  config:
    GenericOAuthenticator:
      client_id: client_id
      client_secret: secret
      oauth_callback_url: https://my-url/hub/oauth_callback
      authorize_url: https://my-auth0-domain/authorize
      token_url: https://my-auth0-domain/oauth/token
      userdata_url: https://my-auth0-domain.com/userinfo
      scope:
        - openid
        - name
        - profile
        - email
      username_key: name
    JupyterHub:
      authenticator_class: generic-oauth

So far my only insight is this response header:
set-cookie: oauthenticator-state=""; expires=Wed, 01 Jul 2020 15:49:49 GMT; Path=/

Which is sent in response to this url: Request URL: https://my-jupyter-url/hub/oauth_callback?code=BxqUQTValNRYI1_Q&state=1lyTjA1qbZPnToo3MrNDi_~a--oADBIZ

Something doesn’t seem right here, as the oauth state is empty, and the expiration date is from 1 year ago.

Does anyone have any insight as to what would be causing me to get this 400 error every time a user logs in for the first time?

Please let me know if you need further information from me!

Hi! Could you enable debug logging and show us the hub logs corresponding to the login attempt?

Thanks for the reply! Here are the relevant Hub logs:

[D 2021-07-02 15:33:03.626 JupyterHub reflector:353] pods watcher timeout
[D 2021-07-02 15:33:03.626 JupyterHub reflector:278] Connecting pods watcher
[D 2021-07-02 15:33:03.795 JupyterHub reflector:353] events watcher timeout
[D 2021-07-02 15:33:03.795 JupyterHub reflector:278] Connecting events watcher
[W 2021-07-02 15:33:04.316 JupyterHub oauth2:152] OAuth state mismatch: eyJzdGF0ZV9pZCI6ICJhZWM5ZTkxNGNmYjQ0ZDA1ODM4MTFlN2U4NzJiODk4MyIsICJuZXh0X3VybCI6ICIvaHViLyJ9 != opEHU2Lmqk199h0VzJeO5Npvg6-Q7a8R <--- First Login Attempt
[W 2021-07-02 15:33:04.316 JupyterHub web:1787] 400 GET /hub/oauth_callback?code=80DiewdLbaxq3p12&state=opEHU2Lmqk199h0VzJeO5Npvg6-Q7a8R (::ffff:35.191.3.192): OAuth state mismatch
[D 2021-07-02 15:33:04.316 JupyterHub base:1285] No template for 400
[W 2021-07-02 15:33:04.317 JupyterHub log:189] 400 GET /hub/oauth_callback?code=[secret]&state=[secret] (@::ffff:35.191.3.192) 2.01ms
[D 2021-07-02 15:33:04.388 JupyterHub log:189] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=f3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@::ffff:35.191.3.192) 1.84ms
[D 2021-07-02 15:33:04.395 JupyterHub log:189] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@::ffff:35.191.3.213) 0.53ms
[D 2021-07-02 15:33:04.395 JupyterHub log:189] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@::ffff:35.191.3.219) 1.73ms
[D 2021-07-02 15:33:04.396 JupyterHub log:189] 200 GET /hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@::ffff:35.191.3.210) 9.58ms
[D 2021-07-02 15:33:04.457 JupyterHub log:189] 200 GET /hub/logo (@::ffff:35.191.3.198) 0.89ms
[D 2021-07-02 15:33:04.522 JupyterHub log:189] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:35.191.3.231) 0.78ms
[D 2021-07-02 15:33:05.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.65ms
[D 2021-07-02 15:33:07.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.72ms
[D 2021-07-02 15:33:09.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.68ms
[D 2021-07-02 15:33:11.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.69ms
[D 2021-07-02 15:33:12.696 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.70ms
[D 2021-07-02 15:33:13.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.67ms
[D 2021-07-02 15:33:13.635 JupyterHub reflector:353] pods watcher timeout
[D 2021-07-02 15:33:13.635 JupyterHub reflector:278] Connecting pods watcher
[D 2021-07-02 15:33:13.806 JupyterHub reflector:353] events watcher timeout
[D 2021-07-02 15:33:13.806 JupyterHub reflector:278] Connecting events watcher
[D 2021-07-02 15:33:15.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.69ms
[D 2021-07-02 15:33:17.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.67ms
[D 2021-07-02 15:33:19.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.65ms
[D 2021-07-02 15:33:21.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.67ms
[D 2021-07-02 15:33:22.696 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.76ms
[D 2021-07-02 15:33:23.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.70ms
[D 2021-07-02 15:33:23.644 JupyterHub reflector:353] pods watcher timeout
[D 2021-07-02 15:33:23.645 JupyterHub reflector:278] Connecting pods watcher
[D 2021-07-02 15:33:23.819 JupyterHub reflector:353] events watcher timeout
[D 2021-07-02 15:33:23.819 JupyterHub reflector:278] Connecting events watcher
[D 2021-07-02 15:33:25.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.67ms
[D 2021-07-02 15:33:27.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.69ms
[D 2021-07-02 15:33:29.160 JupyterHub log:189] 200 GET /hub/health (@10.0.64.55) 0.65ms
[I 2021-07-02 15:33:29.308 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:35.191.3.216) 0.71ms <--- Clicked the "JupyterHub" icon on the top left
[I 2021-07-02 15:33:29.336 JupyterHub log:189] 302 GET /hub/login?next=%2Fhub%2F -> /hub/oauth_login?next=%2Fhub%2F (@::ffff:35.191.3.201) 0.78ms
[I 2021-07-02 15:33:29.362 JupyterHub oauth2:103] OAuth redirect: 'https://my-jupyter-url/hub/oauth_callback'
[D 2021-07-02 15:33:29.362 JupyterHub base:526] Setting cookie oauthenticator-state: {'httponly': True, 'expires_days': 1}
[I 2021-07-02 15:33:29.363 JupyterHub log:189] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://moove-prod.us.auth0.com/authorize?response_type=code&redirect_uri=https%3A%2F%2Fnotebooks.moove.ai%2Fhub%2Foauth_callback&client_id=qa6FQLUNbixFt9Qtu7IJesmwbeBsQy1W&state=[secret]&scope=openid+email+profile (@::ffff:35.191.3.222) 1.23ms
[D 2021-07-02 15:33:30.338 JupyterHub base:526] Setting cookie jupyterhub-session-id: {'httponly': True}
[D 2021-07-02 15:33:30.338 JupyterHub base:530] Setting cookie for my-jupyter@user: jupyterhub-hub-login
[D 2021-07-02 15:33:30.338 JupyterHub base:526] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2021-07-02 15:33:30.338 JupyterHub base:762] User logged in: my-jupyter@user <--- Successfully logged in
[I 2021-07-02 15:33:30.339 JupyterHub log:189] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/ (@::ffff:35.191.3.234) 489.04ms
[D 2021-07-02 15:33:30.392 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for my-jupyter@user:
[I 2021-07-02 15:33:30.396 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (my-jupyter@user@::ffff:35.191.3.216) 32.00ms
[D 2021-07-02 15:33:30.424 JupyterHub pages:209] Serving options form for my-jupyter@user
[I 2021-07-02 15:33:30.425 JupyterHub log:189] 200 GET /hub/spawn (my-jupyter@user@::ffff:35.191.3.198) 5.69ms
[D 2021-07-02 15:33:30.502 JupyterHub log:189] 200 GET /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 (@::ffff:35.191.3.201) 1.39ms

Do you have the logs from before that? Ideally starting from when you first load the JupyterHub login page.