Another Redirect Loop post (helm)

JupyterHub
1

Hello All,

After digging through the wide variety of posts here about redirect loops I have exhausted my efforts to solve our redirect loop issue. The logs on the spawned pod do not show any errors (yes debug mode is enabled.) The only difference is we use an ALB ingress which I think is irrelevant as we can access the app. Below is the debug log output from the -hub pod.
The ALB points to the proxy-public as stated in almost every post I have read, this is using the default ingress from the helm chart with ALB annotations. I have added and removed almost every option imaginable. e.g. in the config.yaml below I have added and removed the baseUrl/defaultUrl without any change to the redirect issue. Any help would be greatly appreciated!
helm chart: 4.3.1
platform: eks

[D 2025-12-09 17:32:08.651 JupyterHub base:411] Refreshing auth for testuser
[I 2025-12-09 17:32:08.652 JupyterHub log:192] 302 GET /hub/user/testuser/ -> /user/testuser/?redirects=1 (testuser@::ffff:10.246.15.52) 3.31ms
[I 2025-12-09 17:32:08.735 JupyterHub log:192] 302 GET /user/testuser/?redirects=1 -> /hub/user/testuser/?redirects=1 (@10.246.1.222) 0.49ms
[W 2025-12-09 17:32:08.818 JupyterHub base:1883] Redirect loop detected on /hub/user/testuser/?redirects=1
[D 2025-12-09 17:32:09.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.49ms
[I 2025-12-09 17:32:10.223 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.14.114) 0.43ms
[I 2025-12-09 17:32:10.820 JupyterHub log:192] 302 GET /hub/user/testuser/?redirects=1 -> /user/testuser/?redirects=2 (testuser@::ffff:10.246.15.52) 2004.57ms
[I 2025-12-09 17:32:10.901 JupyterHub log:192] 302 GET /user/testuser/?redirects=2 -> /hub/user/testuser/?redirects=2 (@10.246.1.222) 0.48ms
[W 2025-12-09 17:32:10.984 JupyterHub base:1883] Redirect loop detected on /hub/user/testuser/?redirects=2
[D 2025-12-09 17:32:11.335 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.46ms
[D 2025-12-09 17:32:11.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[I 2025-12-09 17:32:11.544 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.8.34) 0.43ms
[D 2025-12-09 17:32:13.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.46ms
[I 2025-12-09 17:32:14.985 JupyterHub log:192] 302 GET /hub/user/testuser/?redirects=2 -> /user/testuser/?redirects=3 (testuser@::ffff:10.246.15.52) 4003.51ms
[I 2025-12-09 17:32:15.069 JupyterHub log:192] 302 GET /user/testuser/?redirects=3 -> /hub/user/testuser/?redirects=3 (@10.246.1.222) 0.48ms
[W 2025-12-09 17:32:15.156 JupyterHub base:1883] Redirect loop detected on /hub/user/testuser/?redirects=3
[D 2025-12-09 17:32:15.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[D 2025-12-09 17:32:16.629 JupyterHub reflector:390] pods watcher timeout
[D 2025-12-09 17:32:16.629 JupyterHub reflector:304] Connecting pods watcher
[D 2025-12-09 17:32:17.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.47ms
[I 2025-12-09 17:32:19.246 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.14.114) 0.44ms
[D 2025-12-09 17:32:19.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.41ms
[D 2025-12-09 17:32:21.336 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.46ms
[D 2025-12-09 17:32:21.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[I 2025-12-09 17:32:22.059 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.15.52) 0.44ms
[I 2025-12-09 17:32:22.924 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.8.34) 0.44ms
[I 2025-12-09 17:32:23.158 JupyterHub log:192] 302 GET /hub/user/testuser/?redirects=3 -> /user/testuser/?redirects=4 (testuser@::ffff:10.246.15.52) 8004.26ms
[I 2025-12-09 17:32:23.232 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.12.73) 0.44ms
[I 2025-12-09 17:32:23.238 JupyterHub log:192] 302 GET /user/testuser/?redirects=4 -> /hub/user/testuser/?redirects=4 (@10.246.1.222) 0.28ms
[W 2025-12-09 17:32:23.320 JupyterHub web:1932] 500 GET /hub/user/testuser/?redirects=4 (::ffff:10.246.15.52): Redirect loop detected.
[D 2025-12-09 17:32:23.321 JupyterHub base:1558] Using default error template for 500
[E 2025-12-09 17:32:23.321 JupyterHub log:184] {
      "X-Forwarded-For": "10.246.1.222,::ffff:10.246.15.52",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "Host": "jupyterhub.<redacted>",
      "X-Amzn-Trace-Id": "Root=1-69385d27-62dcf9cc0a87ff0d2d72c6cd",
      "Cache-Control": "max-age=0",
      "Upgrade-Insecure-Requests": "1",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
      "Sec-Gpc": "1",
      "Accept-Language": "en-US,en;q=0.7",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Dest": "document",
      "Sec-Ch-Ua": "\"Brave\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua-Platform": "\"Windows\"",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Priority": "u=0, i",
      "Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
      "X-Forwarded-Host": "jupyterhub.<redacted>",
      "Connection": "keep-alive"
    }
[E 2025-12-09 17:32:23.321 JupyterHub log:192] 500 GET /hub/user/testuser/?redirects=4 (testuser@::ffff:10.246.15.52) 3.57ms
[D 2025-12-09 17:32:23.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[D 2025-12-09 17:32:24.153 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/css/fontawesome.css (@10.246.1.222) 0.48ms
[D 2025-12-09 17:32:24.154 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/css/solid.css (@::ffff:10.246.15.52) 0.31ms
[D 2025-12-09 17:32:24.392 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@10.246.1.222) 0.48ms
[I 2025-12-09 17:32:25.223 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.12.73) 0.46ms
[D 2025-12-09 17:32:25.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[I 2025-12-09 17:32:26.549 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.8.34) 0.43ms
[D 2025-12-09 17:32:26.638 JupyterHub reflector:390] pods watcher timeout
[D 2025-12-09 17:32:26.638 JupyterHub reflector:304] Connecting pods watcher

config.yaml

        singleuser:
          defaultUrl: "/lab"
          extraEnv:
            NO_PROXY: "jupyterhub.<redacted>,localhost,127.0.0.1"
            no_proxy: "jupyterhub.<redacted>,localhost,127.0.0.1"
        debug:
          enabled: true
        ingress:
          enabled: true
          hosts:
            - jupyterhub.<redacted>
          ingressClassName: alb
          annotations:
            alb.ingress.kubernetes.io/scheme: internal
            alb.ingress.kubernetes.io/target-type: 'ip'
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
            alb.ingress.kubernetes.io/certificate-arn: "<redacted>"
            alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
            alb.ingress.kubernetes.io/ssl-redirect: '443'
            external-dns.alpha.kubernetes.io/hostname: <redacted>     
        proxy:
          service:
            type: ClusterIP
        hub:
          baseUrl: "/"
          config:
            Authenticator:
              admin_users:
                - testuser
        scheduling:
          userScheduler:
            enabled: true
            pdb:
              enabled: false

proxy logs:

2025-12-09T17:45:03.964Z [ConfigProxy] info: 200 GET /api/routes
2025-12-09T17:45:04.752Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:04.755Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:10.346Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:10.350Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:11.738Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:11.739Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:11.740Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:11.741Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:11.742Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:11.745Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:11.745Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:11.745Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:11.745Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:11.746Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:13.451Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:13.452Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:13.453Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:13.458Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:13.458Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:13.458Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:13.624Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:13.625Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:13.625Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:15.630Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:15.631Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:15.631Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:15.801Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:15.801Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:15.802Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:19.765Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:19.768Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:19.806Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:19.806Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:19.807Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:19.976Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:19.977Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:19.977Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:25.349Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:25.349Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:25.351Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:25.353Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:25.354Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:25.354Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:26.742Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:26.751Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:27.983Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:27.983Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:27.984Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T17:45:28.145Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.145Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.145Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/ to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.150Z [ConfigProxy] debug: Not recording activity for status 500 on /
2025-12-09T17:45:28.150Z [ConfigProxy] debug: Not recording activity for status 500 on /
2025-12-09T17:45:28.151Z [ConfigProxy] debug: Not recording activity for status 500 on /
2025-12-09T17:45:28.906Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/solid.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.907Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/solid.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.907Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/solid.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:28.908Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T17:45:28.908Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T17:45:28.909Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T17:45:34.779Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:34.779Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T17:45:34.779Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025: The Year in Review
2

Can you share the logs, starting from before the singleuser server is started?

3

Here you go! I stopped and started it again.

[D 2025-12-09 19:05:33.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.48ms
[D 2025-12-09 19:05:35.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.46ms
[D 2025-12-09 19:05:36.140 JupyterHub reflector:390] events watcher timeout
[D 2025-12-09 19:05:36.140 JupyterHub reflector:304] Connecting events watcher
[D 2025-12-09 19:05:36.521 JupyterHub scopes:1013] Checking access to /hub/api/users/testuser/servers/ via scope servers!server=testuser/
[D 2025-12-09 19:05:36.523 JupyterHub user:496] Creating <class 'kubespawner.spawner.KubeSpawner'> for testuser:
[D 2025-12-09 19:05:36.524 JupyterHub base:1097] Initiating spawn for testuser
[D 2025-12-09 19:05:36.524 JupyterHub base:1101] 0/64 concurrent spawns
[D 2025-12-09 19:05:36.524 JupyterHub base:1106] 0 active servers
[I 2025-12-09 19:05:36.538 JupyterHub provider:661] Creating oauth client jupyterhub-user-testuser
[D 2025-12-09 19:05:36.557 JupyterHub user:913] Calling Spawner.start for testuser
[I 2025-12-09 19:05:36.557 JupyterHub log:192] 202 POST /hub/api/users/testuser/servers/?_xsrf=[secret] (testuser@10.246.1.222) 38.68ms
[I 2025-12-09 19:05:36.557 JupyterHub spawner:2931] Attempting to create pvc claim-testuser, with timeout 3
[I 2025-12-09 19:05:36.578 JupyterHub spawner:2947] PVC claim-testuser already exists, so did not create new pvc.
[I 2025-12-09 19:05:36.579 JupyterHub spawner:2890] Attempting to create pod jupyter-testuser, with timeout 3
[D 2025-12-09 19:05:36.648 JupyterHub scopes:1013] Checking access to /hub/api/users via scope list:users
[I 2025-12-09 19:05:36.653 JupyterHub log:192] 200 GET /hub/api/users?include_stopped_servers=1&offset=0&limit=50&name_filter=&sort=id&state=[secret]&_xsrf=[secret] (testuser@::ffff:10.246.15.52) 8.12ms
[D 2025-12-09 19:05:37.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.47ms
[I 2025-12-09 19:05:38.592 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.15.52) 0.44ms
[I 2025-12-09 19:05:38.593 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.15.52) 0.24ms
[D 2025-12-09 19:05:39.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.47ms
[D 2025-12-09 19:05:40.091 JupyterHub spawner:3254] pod jupyterhub/jupyter-testuser events before launch: 2025-12-09T19:05:36.619886Z [Normal] Successfully assigned jupyterhub/jupyter-testuser to ip-10-246-13-14.ec2.internal
    2025-12-09T19:05:38Z [Normal] Container image "quay.io/jupyterhub/k8s-network-tools:4.3.1" already present on machine
    2025-12-09T19:05:38Z [Normal] Created container: block-cloud-metadata
    2025-12-09T19:05:38Z [Normal] Started container block-cloud-metadata
    2025-12-09T19:05:38Z [Normal] Container image "quay.io/jupyterhub/k8s-singleuser-sample:4.3.1" already present on machine
    2025-12-09T19:05:38Z [Normal] Created container: notebook
    2025-12-09T19:05:39Z [Normal] Started container notebook
[D 2025-12-09 19:05:40.097 JupyterHub spawner:1706] Polling subprocess every 30s
[D 2025-12-09 19:05:40.098 JupyterHub utils:298] Waiting 30s for server at http://10.246.13.135:8888/user/testuser/api
[I 2025-12-09 19:05:40.870 JupyterHub log:192] 200 GET /hub/api (@10.246.13.135) 0.53ms
[D 2025-12-09 19:05:40.950 JupyterHub base:366] Recording first activity for <APIToken('b965...', user='testuser', client_id='jupyterhub')>
[D 2025-12-09 19:05:40.956 JupyterHub scopes:1013] Checking access to /hub/api/users/testuser/activity via scope users:activity!user=testuser
[D 2025-12-09 19:05:40.958 JupyterHub users:1005] Activity for user testuser: 2025-12-09T19:05:40.857229Z
[D 2025-12-09 19:05:40.958 JupyterHub users:1023] Activity on server testuser/: 2025-12-09T19:05:40.857229Z
[I 2025-12-09 19:05:40.964 JupyterHub log:192] 200 POST /hub/api/users/testuser/activity (testuser@10.246.13.135) 15.34ms
[D 2025-12-09 19:05:41.336 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.46ms
[D 2025-12-09 19:05:41.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.44ms
[D 2025-12-09 19:05:41.731 JupyterHub utils:334] Server at http://10.246.13.135:8888/user/testuser/api responded in 1.63s
[D 2025-12-09 19:05:41.731 JupyterHub _version:73] jupyterhub and jupyterhub-singleuser both on version 5.4.1
[I 2025-12-09 19:05:41.731 JupyterHub base:1126] User testuser took 5.207 seconds to start
[I 2025-12-09 19:05:41.732 JupyterHub proxy:331] Adding user testuser to proxy /user/testuser/ => http://10.246.13.135:8888
[D 2025-12-09 19:05:41.732 JupyterHub proxy:925] Proxy: Fetching POST http://<redacted>-proxy-api:8001/api/routes/user/testuser
[I 2025-12-09 19:05:42.129 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.8.34) 0.44ms
[D 2025-12-09 19:05:42.504 JupyterHub reflector:390] pods watcher timeout
[D 2025-12-09 19:05:42.504 JupyterHub reflector:304] Connecting pods watcher
[I 2025-12-09 19:05:42.538 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.14.114) 0.40ms
[I 2025-12-09 19:05:42.810 JupyterHub log:192] 302 GET / -> /hub/ (@10.246.12.73) 0.45ms
[I 2025-12-09 19:05:42.811 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.246.14.114) 0.25ms
[D 2025-12-09 19:05:43.441 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.47ms
[D 2025-12-09 19:05:45.440 JupyterHub log:192] 200 GET /hub/health (@10.246.15.67) 0.45ms
[D 2025-12-09 19:05:46.156 JupyterHub reflector:390] events watcher timeout
[D 2025-12-09 19:05:46.156 JupyterHub reflector:304] Connecting events watcher
4

This log indicates that JupyterHub correctly added a route to the JupyterHub proxy, telling it to route /user/testuser to http://10.246.13.135:8888 which should be the spawned singluser pod. If the proxy is for some reason ignoring that route it would send /user/testuserto the hub, which doesn’t support that endpoint so redirects to /hub/…. which also doesn’t exist.

Can you share your proxy debug logs too? Ideally there should be something corresponding to the hub’s request to setup the route, and also some logs relating to the user accessing /user/testuser

5

Yeah, here is the logs from the proxy at that time.
proxy:

2025-12-09T19:05:29.383Z [ConfigProxy] debug: PROXY WEB /hub/api/users to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:36.644Z [ConfigProxy] debug: PROXY WEB /hub/api/users to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:38.592Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:38.593Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:41.733Z [ConfigProxy] info: Adding route /user/testuser -> http://10.246.13.135:8888
2025-12-09T19:05:41.733Z [ConfigProxy] info: Route added /user/testuser -> http://10.246.13.135:8888
2025-12-09T19:05:41.734Z [ConfigProxy] info: 201 POST /api/routes/user/testuser
2025-12-09T19:05:42.534Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:42.535Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:42.535Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:42.538Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:42.538Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:42.539Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:42.808Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:42.809Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:42.811Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:42.811Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:51.457Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/fontawesome.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:51.459Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T19:05:51.693Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:51.695Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T19:05:53.604Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:53.605Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:57.535Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-hub:8081
2025-12-09T19:05:57.538Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:05:57.820Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:05:57.823Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:06:03.964Z [ConfigProxy] info: 200 GET /api/routes

jupyterhub-testuser pod:

Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[W 2025-12-09 19:05:40.645 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2025-12-09 19:05:40.647 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2025-12-09 19:05:40.649 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2025-12-09 19:05:40.649 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.4.1
[I 2025-12-09 19:05:40.649 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
[I 2025-12-09 19:05:40.652 ServerApp] jupyterhub | extension was successfully linked.
[W 2025-12-09 19:05:40.652 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-12-09 19:05:40.654 ServerApp] jupyterlab | extension was successfully linked.
[W 2025-12-09 19:05:40.655 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-12-09 19:05:40.656 ServerApp] nbclassic | extension was successfully linked.
[I 2025-12-09 19:05:40.656 ServerApp] nbgitpuller | extension was successfully linked.
[I 2025-12-09 19:05:40.851 ServerApp] notebook_shim | extension was successfully linked.
[I 2025-12-09 19:05:40.862 ServerApp] notebook_shim | extension was successfully loaded.
[I 2025-12-09 19:05:40.864 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2025-12-09 19:05:40.865 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2025-12-09 19:05:40.871 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2025-12-09 19:05:40.871 ServerApp] jupyterhub | extension was successfully loaded.
[I 2025-12-09 19:05:40.873 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.12/site-packages/jupyterlab
[I 2025-12-09 19:05:40.874 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2025-12-09 19:05:40.875 LabApp] Extension Manager is 'pypi'.
[I 2025-12-09 19:05:40.883 ServerApp] Skipped non-installed server(s): basedpyright, bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyrefly, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2025-12-09 19:05:40.940 ServerApp] jupyterlab | extension was successfully loaded.
[I 2025-12-09 19:05:40.942 ServerApp] nbclassic | extension was successfully loaded.
[I 2025-12-09 19:05:40.943 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2025-12-09 19:05:40.943 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2025-12-09 19:05:40.943 ServerApp] Jupyter Server 2.17.0 is running at:
[I 2025-12-09 19:05:40.943 ServerApp] http://jupyter-testuser:8888/user/testuser/lab?token=...
[I 2025-12-09 19:05:40.943 ServerApp]     http://127.0.0.1:8888/user/testuser/lab?token=...
[I 2025-12-09 19:05:40.943 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2025-12-09 19:05:41.731 ServerApp] 200 GET /user/testuser/api (@10.246.15.175) 0.84ms

proxy log of the loop:

2025-12-09T19:23:17.793Z [ConfigProxy] debug: PROXY WEB /user/testuser/ to http://10.246.13.135:8888
2025-12-09T19:23:17.796Z [ConfigProxy] debug: Not recording activity for status 302 on /user/testuser
2025-12-09T19:23:17.966Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:17.968Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:17.974Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:17.975Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:18.138Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:18.139Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:20.144Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:20.145Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:20.306Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:20.306Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:24.312Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:24.312Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:24.472Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:24.472Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:24.515Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:24.516Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:24.519Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:24.519Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.034Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.035Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.036Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.037Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.038Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.039Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.041Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.041Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.042Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.042Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.042Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.042Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.672Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.673Z [ConfigProxy] debug: PROXY WEB / to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:28.675Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:28.675Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:32.477Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:32.477Z [ConfigProxy] debug: Not recording activity for status 302 on /
2025-12-09T19:23:32.642Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:32.642Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:32.647Z [ConfigProxy] debug: Not recording activity for status 500 on /
2025-12-09T19:23:32.647Z [ConfigProxy] debug: Not recording activity for status 500 on /
2025-12-09T19:23:33.558Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/solid.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:33.559Z [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/css/solid.css to http://<redacted>-jupyterhub-hub:8081
2025-12-09T19:23:33.560Z [ConfigProxy] debug: Not recording activity for status 304 on /
2025-12-09T19:23:33.560Z [ConfigProxy] debug: Not recording activity for status 304 on /
6

This looks fine

But the singleuser server or something else is incorrectly redirecting /user/testuser/ to /user/testuser which ends up going to the hub. Can you turn on debug logs in your singleuser server and share those logs again?

Can you also try setting NO_PROXY/no_proxy to disable the proxy for all IPs/hosts? By defaultJUPYTERHUB_API_URL=http://hub:8081/hub/api which doesn’t match

7

I have NO_PROXY, no_proxy config in there, what should the no_proxy entries for all hosts? Just 0.0.0.0/0,*? Thanks! I’ll have the logs shortly.

8

I have this in the config:

        singleuser:
          defaultUrl: "/lab"
          cmd: ["jupyterhub-singleuser", "--debug"]
          extraEnv:
            no_proxy: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1"
            NO_PROXY: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1" # Example
        hub:
          baseUrl: "/"
          config:
            Authenticator:
              admin_users:
                - testuser
          extraEnv:
            no_proxy: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1"
            NO_PROXY: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1"

Deployed that and still seeing the redirect loop.
Same exact errors as before.
Logs from the singleuser pod show little to no info after the first initial registration of the endpoint.

[D 2025-12-10 00:12:08.650 ServerApp] No user identified
[I 2025-12-10 00:12:08.651 ServerApp] 200 GET /user/testuser/api (@10.246.15.177) 0.90ms
[I 2025-12-10 00:12:14.379 ServerApp] 302 GET /user/testuser/ -> /user/testuser/lab? (@::ffff:10.246.15.52) 0.44ms
[D 2025-12-10 00:17:09.062 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:22:11.262 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:26:50.113 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:31:33.952 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:36:20.572 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:41:17.142 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
[D 2025-12-10 00:46:30.172 JupyterHubSingleUser] Notifying Hub of activity 2025-12-10T00:12:08.294435Z
9

What paths are part of the loop? I have not read up, but that is a key question, followed by trying to understand each redirect reason, and concluding what redirect is a misbehavior

10

The paths that are misbehaving are the /hub/user/testuser/lab gets stuck in a redirect loop.

11

Any updates or recommendations would be helpful, we really want to use this.

12

Sorry to be a pain, but it would be really helpful to see the debug logs for all components, covering the same time period, so we can see which components are (or aren’t) communicating with each other, and at what point.

Are you using the default Z2JH test image (as suggested by the config.yaml in your first post), or your own?

2 Likes
13

Yes, no other customizations other than catering to our environment, default images for all deployed parts. I can update the logs to the same time period for all of the above again to help debug as I am not seeing something obvious.

14

What redirected to that path? It isn’t a url that one should access. It should be without /hub to access a user-server.

Logs would show this i think

15 
2025-12-09T19:23:17.793Z [ConfigProxy] debug: PROXY WEB /user/testuser/ to http://10.246.13.135:8888
2025-12-09T19:23:17.796Z [ConfigProxy] debug: Not recording activity for status 302 on /user/testuser
2025-12-09T19:23:17.966Z [ConfigProxy] debug: PROXY WEB /hub/user/testuser/lab to http://<redacted>-jupyterhub-hub:8081

It seems /user/testuser redirected you to /hub/user/testuser.

Is the user server configured to serve everything under URL prefix /hub? Then that is a misconfiguration.

1 Like
16 
        singleuser:
          cmd: ["jupyterhub-singleuser", "--debug"]
          extraEnv:
            no_proxy: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1"
            NO_PROXY: "jupyterhub.<redacted>,10.0.0.0/8,localhost,127.0.0.1" # Example
        hub:
          baseUrl: "/"

Maybe try without this as well, not sure it doesnt cause trouble

17

I have figured it out! It was never an issue with the helm chart or application. I had a wrapper (mother chart) on top of the helm chart and bypassed that and it works like a charm. Thanks for all the help and sorry to waste your time!

2 Likes