Hi @manics I’m @seanturner026’s colleague. Here’s the information you requested. I tried to preserve as much as I could of the relevant pieces (hub, ingress, and proxy config) without leaving in any sensitive information. Let me know if there is any additional context I can provide for you if this is insufficient.
Configuration in values.yaml:
hub:
config:
GenericOAuthenticator:
...
JupyterHub:
authenticator_class: generic-oauth
existingSecret: hub-secrets
extraEnv:
JUPY_COOKIE_SECRET:
valueFrom:
secretKeyRef: ...
JUPYTERHUB_CRYPT_KEY:
valueFrom:
secretKeyRef: ...
JUPYTERHUB_XSRF_ANONYMOUS_IP_CIDRS: "0.0.0.0/0"
JUPYTERHUB_XSRF_ANONYMOUS_ID_HEADERS: ""
OAUTH_CLIENT_ID:
valueFrom:
secretKeyRef: ...
OAUTH_CLIENT_SECRET:
valueFrom:
secretKeyRef: ...
PGPASSWORD:
valueFrom:
secretKeyRef: ...
image:
name: ...
tag: ...
redirectToServer: False
service:
extraPorts:
- port: 9888
targetPort: 9888
name: announcement
services:
gallery:
name: gallery
oauth_no_confirm: True
url: http://hub-gallery.jupyterhub.svc.cluster.local:8080
annoncement:
name: announcement
oauth_no_confirm: True
url: http://hub.jupyterhub.svc.cluster.local:9888
command: ["python", "-m", "jupyterhub_announcement"]
...
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path: /hub/api
alb.ingress.kubernetes.io/certificate-arn: ...
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/group.name: ...
external-dns.alpha.kubernetes.io/hostname: ...
hosts:
...
...
proxy:
secretToken: ...
annotations: ...
chp:
networkPolicy:
enabled: false
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ...
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
type: NodePort
nodePorts:
http: 30080
https: 30443
Please note a few things:
- we are setting
JUPYTERHUB_XSRF_ANONYMOUS_IP_CIDRS and JUPYTERHUB_XSRF_ANONYMOUS_ID_HEADERS per the suggestion by mnirk above. I’ve confirmed these environment variables are present in the running hub pod as well.
- we are using
alb in AWS as our ingress, which seems to behave a bit differently than other ingresses (eg X-Forwarded-For header is enabled by default and ours is configured for “Append”, and headers do not appear to be customizable like they are for nginx)
Hub logs related to XSRF cookies. This one captures me encountering the issue, logging out, then logging back in which fixed the issue:
[D 2025-07-30 21:30:49.586 JupyterHub base:411] Refreshing auth for julian
[I 2025-07-30 21:30:49.587 JupyterHub log:192] 302 GET /hub/ -> /hub/home (julian@::ffff:IP_ADDRESS_1) 194.21ms
[D 2025-07-30 21:30:49.726 JupyterHub _xsrf_utils:161] xsrf id mismatch b'n82<redacted>6Mc=:192<redacted>dd' != b'l1x<redacted>86Vk=:1921f<redacted>8add'
[I 2025-07-30 21:30:49.726 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'l1x<redacted>86Vk=:1921f<redacted>8add' {'path': '/hub/'}
[D 2025-07-30 21:30:51.194 JupyterHub scopes:1013] Checking access to /hub/spawn/julian via scope servers!server=julian/
[D 2025-07-30 21:30:51.194 JupyterHub pages:208] Serving options form for julian
[D 2025-07-30 21:30:51.194 JupyterHub _xsrf_utils:161] xsrf id mismatch b'l1x<redacted>86Vk=:1921f<redacted>8add' != b'n82<redacted>v6Mc=:1921f<redacted>8add'
[I 2025-07-30 21:30:51.194 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'n82<redacted>v6Mc=:1921f<redacted>8add' {'path': '/hub/'}
[D 2025-07-30 21:31:03.964 JupyterHub _xsrf_utils:161] xsrf id mismatch b'n82<redacted>v6Mc=:1921f<redacted>8add' != b'-b8<redacted>lXBI=:1921f<redacted>8add'
[I 2025-07-30 21:31:03.964 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'-b8<redacted>lXBI=:1921f<redacted>8add' {'path': '/hub/'}
[W 2025-07-30 21:31:03.965 JupyterHub web:1873] 403 POST /hub/spawn/julian?_xsrf=MnwxOjB8MT<redacted>3YmMy (::ffff:IP_ADDRESS_1): XSRF cookie does not match POST argument
[D 2025-07-30 21:31:03.965 JupyterHub base:1542] Using default error template for 403
[W 2025-07-30 21:31:03.979 JupyterHub log:192] 403 POST /hub/spawn/julian?_xsrf=[secret] (julian@::ffff:IP_ADDRESS_1) 98.96ms
[D 2025-07-30 21:31:07.966 JupyterHub _xsrf_utils:161] xsrf id mismatch b'-b8<redacted>lXBI=:1921f<redacted>8add' != b'l1x<redacted>86Vk=:1921f<redacted>8add'
[I 2025-07-30 21:31:07.966 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'l1x<redacted>86Vk=:1921f<redacted>8add' {'path': '/hub/'}
[I 2025-07-30 21:31:09.682 JupyterHub login:46] User logged out: julian
[I 2025-07-30 21:31:09.765 JupyterHub log:192] 302 GET /hub/logout -> /hub/login (@::ffff:IP_ADDRESS_2) 165.50ms
[I 2025-07-30 21:31:09.818 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'None:-b8D8<redacted>XBI=' {'path': '/hub/', 'max_age': 3600}
[I 2025-07-30 21:31:09.818 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'None:-b8D8<redacted>XBI=' {'path': '/hub/', 'max_age': 3600}
....
[D 2025-07-30 22:07:41.961 JupyterHub roles:326] Assigning default role to User julian
[D 2025-07-30 22:07:42.003 JupyterHub base:681] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2025-07-30 22:07:42.003 JupyterHub base:685] Setting cookie for julian: jupyterhub-hub-login
[D 2025-07-30 22:07:42.003 JupyterHub base:681] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2025-07-30 22:07:42.003 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'l1x<redacted>86Vk=:1921f<redacted>8add' {'path': '/hub/'}
[I 2025-07-30 22:07:42.003 JupyterHub base:973] User logged in: julian
[I 2025-07-30 22:07:42.004 JupyterHub log:192] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/home (julian@::ffff:IP_ADDRESS_3) 980.23ms
[D 2025-07-30 22:07:42.212 JupyterHub _xsrf_utils:161] xsrf id mismatch b'l1x<redacted>86Vk=:1921f<redacted>8add' != b'044<redacted>d6f54:1921f<redacted>8add'
[I 2025-07-30 22:07:42.212 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'044<redacted>d6f54:1921f<redacted>8add' {'path': '/hub/'}
[D 2025-07-30 22:08:37.337 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-07-30 22:08:37.339 JupyterHub proxy:392] Checking routes
[D 2025-07-30 22:08:56.727 JupyterHub scopes:1013] Checking access to /hub/spawn/julian via scope servers!server=julian/
[D 2025-07-30 22:08:56.727 JupyterHub pages:208] Serving options form for julian
[D 2025-07-30 22:09:11.757 JupyterHub scopes:1013] Checking access to /hub/spawn/julian via scope servers!server=julian/
[D 2025-07-30 22:09:11.758 JupyterHub pages:256] Triggering spawn with supplied form options for julian
....
[I 2025-07-30 22:09:12.343 JupyterHub log:192] 302 POST /hub/spawn/julian?_xsrf=[secret] -> /hub/spawn-pending/julian?_xsrf=[secret] (julian@::ffff:IP_ADDRESS_2) 680.67ms
I’ve redacted potentially sensitive information, namely:
- full cookies (but left the begin/end so you can compare)
- IP addresses (matching ones sharing the same number)
One thing I noticed debugging this is that for failed requests (with the XSRF error) the browser does not include jupyterhub-session-id cookie in the session. Instead, there is ajs_anonymous_id. Not sure if that is relevant but one thing that stood out to me.
Another thing to point out is that I’m surprised we are seeing the cookie comparison happening at all, since I thought minrk’s suggestion would disable this. But maybe it is working as expected and that’s just not surfaced in logs.
Anyway, grateful for any advice or ideas you have here! This has been stumping us for a while now and, while its an easy fix with a logout/login, it is affecting a lot of our users and is generally just very annoying. We started noticing it when we upgraded to Jupyterhub 5.3.0 (helm version 4.2.0) from helm version 3.2.1.
Thanks very much!
