Jupyterhub on k8s| 500 : Internal Server Error Redirect loop detected. Notebook has jupyterhub version unknown (likely < 0.8), but the Hub expects 4.1.5. Try installing jupyterhub==4.1.5 in the user environment if you continue to have problems

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Environment:
AKS

Nginx for reverse proxy(ingress)

After enabling subdomain by following this doc: Security — Zero to JupyterHub with Kubernetes documentation got 500 : Internal Server Error

Redirect loop detected. Notebook has jupyterhub version unknown (likely < 0.8), but the Hub expects 4.1.5. Try installing jupyterhub==4.1.5 in the user environment if you continue to have problems.

Noticing two issue:
1.After server starting re-directing to login page.
Hub logs:
[I 2025-12-03 00:36:55.217 JupyterHub users:776] Server ram is ready
[I 2025-12-03 00:36:55.218 JupyterHub log:192] 200 GET /hub/api/users/ram/server/progress?_xsrf=[secret] 15375.45ms
[D 2025-12-03 00:36:55.953 JupyterHub scopes:884] Checking access to /hub/spawn-pending/ram via scope servers
[D 2025-12-03 00:36:55.953 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/ram via servers
[I 2025-12-03 00:36:55.953 JupyterHub log:192] 302 GET /hub/spawn-pending/ram → https://ram.jupyerthubtest.net/user/ram/ 1.74ms
[I 2025-12-03 00:36:56.832 JupyterHub log:192] 302 GET /user/ram/ → /hub/user/ram/ (@10.244.0.18) 0.48ms
[D 2025-12-03 00:36:57.117 JupyterHub log:192] 200 GET /hub/health (@10.224.0.4) 0.45ms
[D 2025-12-03 00:36:57.368 JupyterHub log:192] 200 GET /hub/health (@10.224.0.4) 0.47ms
[I 2025-12-03 00:36:57.443 JupyterHub log:192] 302 GET /hub/user/ram/ → /hub/login?next=%2Fhub%2Fuser%2Fram%2F (@10.244.0.18) 0.50ms
[I 2025-12-03 00:36:58.132 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b’0a0e5eac9b554b8981b83f38a3a1e5db:C_Cyx0FzVo94PJJCmxNT52gMdYPTp8Wh1nR0fi2Bp78=’ {‘path’: ‘/hub/’, ‘max_age’: 3600}
[I 2025-12-03 00:36:58.133 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fuser%2Fram%2F (@10.244.0.18) 1.64ms
[D 2025-12-03 00:36:58.835 JupyterHub log:192] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=bf6089ed4698cb8270a8b0c8ad9508ff886a7a842278e98064d5c1790ca3a36d5d69d9f047ef196882554fc104da2c88eb5395f1ee8cf0f3f6ff8869408350fe (@10.244.0.18) 0.67ms
[D 2025-12-03 00:36:58.881 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=01598a5386176f0279952a3b9632a07e7fce9a12aa53108973c83be9ec3473e7a59354876fab64bfeb01892eb503870183707aa03f207d7a94845ca7980c3819 (@10.244.0.18) 0.75ms
[D 2025-12-03 00:36:58.954 JupyterHub log:192] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@10.244.0.18) 0.58ms
[D 2025-12-03 00:36:59.118 JupyterHub log:192] 200 GET /hub/health (@10.224.0.4) 0.40ms
[D 2025-12-03 00:36:59.202 JupyterHub log:192] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@10.244.0.18) 0.65ms
[D 2025-12-03 00:36:59.263 JupyterHub log:192] 200 GET /hub/logo (@10.244.0.18) 0.58ms

second issue:
after loggin with user “ram” getting

hub logs:

[I 2025-12-03 00:39:20.272 JupyterHub log:192] 302 GET /user/ram/?redirects=2 → /hub/user/ram/?redirects=2 (@10.244.0.18) 0.46ms
[D 2025-12-03 00:39:20.573 JupyterHub reflector:374] pods watcher timeout
[D 2025-12-03 00:39:20.573 JupyterHub reflector:289] Connecting pods watcher
[D 2025-12-03 00:39:20.841 JupyterHub reflector:374] events watcher timeout
[D 2025-12-03 00:39:20.841 JupyterHub reflector:289] Connecting events watcher
[D 2025-12-03 00:39:20.917 JupyterHub scopes:884] Checking access to /hub/user/ram/ via scope access:servers
[D 2025-12-03 00:39:20.917 JupyterHub scopes:697] Argument-based access to /hub/user/ram/ via access:servers
[W 2025-12-03 00:39:20.917 JupyterHub web:1873] 500 GET /hub/user/ram/?redirects=2 (10.244.0.18): Redirect loop detected. Notebook has jupyterhub version unknown (likely < 0.8), but the Hub expects 4.1.5. Try installing jupyterhub==4.1.5 in the user environment if you continue to have problems.
[D 2025-12-03 00:39:20.917 JupyterHub base:1471] No template for 500
[E 2025-12-03 00:39:20.918 JupyterHub log:184] {
“Cookie”: “jupyterhub-hub-login=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]”,
“Priority”: “u=0, i”,
“Accept-Language”: “en-US,en;q=0.9”,
“Accept-Encoding”: “gzip, deflate, br, zstd”,
“Sec-Ch-Ua-Platform”: ““Windows””,
“Sec-Ch-Ua-Mobile”: “?0”,
“Sec-Ch-Ua”: ““Chromium”;v=“142”, “Microsoft Edge”;v=“142”, “Not_A Brand”;v=“99””,
“Sec-Fetch-Dest”: “document”,
“Sec-Fetch-User”: “?1”,
“Sec-Fetch-Mode”: “navigate”,
“Sec-Fetch-Site”: “same-origin”,
“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7”,
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0”,
“Upgrade-Insecure-Requests”: “1”,
“Cache-Control”: “max-age=0”,
“X-Scheme”: “https”,
“X-Forwarded-Scheme”: “https”,
“X-Forwarded-Proto”: “https, https,http”,
“X-Forwarded-Port”: “443,80”,
“X-Forwarded-For”: “157.58.216.98, 157.58.216.98,::ffff:10.244.1.190”,
“X-Real-Ip”: “157.58.216.98, 157.58.216.98”,
“X-Request-Id”: “1b60b0f15f760358556f69480a468e63”,
“Connection”: “keep-alive”
}
[E 2025-12-03 00:39:20.918 JupyterHub log:192] 500 GET /hub/user/ram/?redirects=2 2.73ms

2

@manics please help, struck on the issue for few days now, thanks

3

Please can you show us your full Z2JH configuration, tell us which version of Z2JH you’re using, and tell us how your K8S cluster and surrounding infrastructure are setup?

1 Like
4

Below is the Z2JH helm chat version is version=3.3.7 running in AKS using nginx ingress.
Issue:
Only when subdomain_host: staging.example.com is set in helm chart, user is getting logged.
I believe 302 are the issue but no sure why user getting logged out.

Jupyerthub Logs:

[D 2025-12-10 02:57:28.261 JupyterHub spawner:2821] pod jupyterhub/jupyter-rmanickam events before launch: 2025-12-10T02:57:11.071690Z [Normal] Successfully assigned jupyterhub/jupyter-rmanickam to aks-nodepool1-32173651-vmss000000
2025-12-10T02:57:23Z [Normal] AttachVolume.Attach succeeded for volume “pvc-a480ff7f-0b2a-4adb-9636-e730a5b214c3”
2025-12-10T02:57:25Z [Normal] Container image “Quay already present on machine
2025-12-10T02:57:25Z [Normal] Created container: block-cloud-metadata
2025-12-10T02:57:25Z [Normal] Started container block-cloud-metadata
2025-12-10T02:57:26Z [Normal] Pulling image “notebookacr.azurecr.io/jupyter-spark-notebook:v4.1spark”
2025-12-10T02:57:26Z [Normal] Successfully pulled image “notebookacr.azurecr.io/jupyter-spark-notebook:v4.1spark” in 289ms (289ms including waiting). Image size: 959151390 bytes.
2025-12-10T02:57:26Z [Normal] Created container: notebook
2025-12-10T02:57:26Z [Normal] Started container notebook
[D 2025-12-10 02:57:28.270 JupyterHub spawner:1388] Polling subprocess every 30s
[D 2025-12-10 02:57:28.279 JupyterHub utils:286] Waiting 30s for server at http://10.244.0.133:8888/user/rmanickam/
[D 2025-12-10 02:57:28.281 JupyterHub base:362] Recording first activity for <APIToken(‘4353…’, user=‘rmanickam’, client_id=‘jupyterhub’)>
[D 2025-12-10 02:57:28.292 JupyterHub scopes:884] Checking access to /hub/api/users/rmanickam/activity via scope users:activity
[D 2025-12-10 02:57:28.292 JupyterHub scopes:697] Argument-based access to /hub/api/users/rmanickam/activity via users:activity
[D 2025-12-10 02:57:28.293 JupyterHub users:882] Activity for user rmanickam: 2025-12-10T02:57:28.145648Z
[D 2025-12-10 02:57:28.293 JupyterHub users:900] Activity on server rmanickam/: 2025-12-10T02:57:28.145648Z
[I 2025-12-10 02:57:28.302 JupyterHub log:192] 200 POST /hub/api/users/rmanickam/activity (rmanickam@10.244.0.133) 22.93ms
[D 2025-12-10 02:57:28.304 JupyterHub utils:303] Server at http://10.244.0.133:8888/user/rmanickam/ responded with 302
[D 2025-12-10 02:57:28.304 JupyterHub utils:324] Server at http://10.244.0.133:8888/user/rmanickam/ responded in 0.03s
[W 2025-12-10 02:57:28.304 JupyterHub _version:38] Single-user server has no version header, which means it is likely < 0.8. Expected 4.1.5
[I 2025-12-10 02:57:28.305 JupyterHub base:1090] User rmanickam took 17.476 seconds to start
[I 2025-12-10 02:57:28.305 JupyterHub proxy:331] Adding user rmanickam to proxy rmanickam.staging.example.com/user/rmanickam/ => http://10.244.0.133:8888
[D 2025-12-10 02:57:28.305 JupyterHub proxy:924] Proxy: Fetching POST http://proxy-api:8001/api/routes/rmanickam.staging.example.com/user/rmanickam
[I 2025-12-10 02:57:28.307 JupyterHub users:776] Server rmanickam is ready
[I 2025-12-10 02:57:28.307 JupyterHub log:192] 200 GET /hub/api/users/rmanickam/server/progress?_xsrf=[secret] (rmanickam@10.244.0.82) 16249.45ms
[D 2025-12-10 02:57:28.883 JupyterHub scopes:884] Checking access to /hub/spawn-pending/rmanickam via scope servers
[D 2025-12-10 02:57:28.883 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/rmanickam via servers
[I 2025-12-10 02:57:28.883 JupyterHub log:192] 302 GET /hub/spawn-pending/rmanickam → https://rmanickam.staging.example.com/user/rmanickam/ (rmanickam@10.244.0.82) 1.88ms
[I 2025-12-10 02:57:29.415 JupyterHub log:192] 302 GET /user/rmanickam/ → /hub/user/rmanickam/ (@10.244.0.82**) 0.47ms**
[I 2025-12-10 02:57:30.038 JupyterHub log:192] 302 GET /hub/user/rmanickam/ → /hub/login?next=%2Fhub%2Fuser%2Frmanickam%2F (@10.244.0.82**) 0.48ms**
[I 2025-12-10 02:57:30.146 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b’7cca1c1bddd5411b8e2d6643d9fb23f8:7uGxe9gNbBYHcbRFTh2aogvxTvVuTI7CF-jfk8WTutg=’ {‘path’: ‘/hub/’, ‘max_age’: 3600}
[I 2025-12-10 02:57:30.147 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fuser%2Frmanickam%2F (@10.244.0.82) 2.16ms

fullnameOverride: “”
nameOverride: “”

hub:

Use your custom service account (don’t create default)

serviceAccount:
create: false
name: jupyterhub-hub-sa

Enable workload identity label on hub pod

labels:
azure.workload.identity/use: “true”

Environment variables for Azure AD certificate authentication

extraEnv:
JUPYTERHUB_CLIENT_ID: “1234”
JUPYTERHUB_AAD_TENANT_ID: “4444”
JUPYTERHUB_OAUTH_CALLBACK_URL: “https://staging.example.com/hub/oauth_callback”
JUPYTERHUB_KEYVAULT_NAME: “Jupyterhub-kv”

Use the certificate name without file extension - Key Vault will provide the right format

JUPYTERHUB_CLIENT_CERTIFICATE: “JH-CERT”
JUPYTERHUB_USER_MANAGED_IDENTITY_CLIENTID: “232131”

Add debug flag to help troubleshoot certificate issues

JUPYTERHUB_DEBUG_CERTIFICATE: “true”

Mount custom authenticator from ConfigMap

extraVolumes:

  • name: custom-authenticator
    configMap:
    name: hub-custom-authenticator

extraVolumeMounts:

  • name: custom-authenticator
    mountPath: /usr/local/etc/jupyterhub/custom_authenticator.py
    subPath: custom_authenticator.py

JupyterHub configuration

config:
JupyterHub:
subdomain_host: staging.example.com
admin_access: false

Authenticator:
  enable_auth_state: true

AzureAdOAuthenticatorWithCertificate:
  tenant_id: "1234"
  oauth_callback_url: "https://staging.example.com/hub/oauth_callback"
  client_id: "4444"
  username_claim: "unique_name"
  scope: ["83ee3f53-c914-4612-a007-468fb8c182ac/.default"]
  allow_all: true  # Set to false and configure allowed_users or groups for production

Install packages and load custom authenticator module

extraConfig:
00-install-packages: |

Install required Python packages for Azure AD certificate authentication

import subprocess
import sys
import importlib

  print(f"Python executable: {sys.executable}")
  print(f"Python version: {sys.version}")
  print(f"sys.path: {sys.path}")
  
  print("\nInstalling required Python packages...")
  
  packages = [
      'oauthenticator',
      'PyJWT>=2.8.0',
      'cryptography>=3.1',
      'azure-identity>=1.15.0',
      'azure-keyvault-secrets>=4.6.0',
      'pkce>=1.0.3',
      'jupyterhub-idle-culler>=1.2.1'
  ]
  
  for package in packages:
      print(f"Installing {package}...")
      # Use pip3 with --target or add PIP_USER=false to install to system location
      # First try to install to system location
      subprocess.check_call(['pip3', 'install', '--no-warn-script-location', package])
  
  print("\nPackage installation complete!")
  print("Checking installed packages...")
  subprocess.check_call(['pip3', 'list', '--format=columns'])
  
  # Add user site-packages to sys.path explicitly
  import site
  user_site = site.getusersitepackages()
  if user_site not in sys.path:
      sys.path.insert(0, user_site)
      print(f"\nAdded user site-packages to sys.path: {user_site}")
  
  print(f"Updated sys.path: {sys.path[:3]}...")  # Show first 3 entries
  
  # Invalidate import caches to pick up newly installed packages
  importlib.invalidate_caches()
  
  # Verify azure packages are available
  print("\nVerifying installed packages...")
  try:
      import azure.identity
      print(f"✓ azure-identity version: {azure.identity.__version__}")
  except ImportError as e:
      print(f"✗ Failed to import azure.identity: {e}")
  
  try:
      import azure.keyvault.secrets
      print(f"✓ azure-keyvault-secrets imported successfully")
  except ImportError as e:
      print(f"✗ Failed to import azure.keyvault.secrets: {e}")
  
  # Now add path and set authenticator AFTER packages are installed and verified
  sys.path.insert(0, '/usr/local/etc/jupyterhub')
  
  # Import and set the authenticator class
  print("\nLoading custom authenticator...")
  from custom_authenticator import AzureAdOAuthenticatorWithCertificate
  c.JupyterHub.authenticator_class = AzureAdOAuthenticatorWithCertificate
  
  print("✓ Custom Azure AD authenticator loaded and configured successfully!")

Idle Culler Service - Automatically stops inactive notebook servers

Culls user servers that have been idle for 2 hours (7200 seconds)

cull:
enabled: true
timeout: 7200 # 2 hours in seconds
every: 600 # Check every 10 minutes
users: false # Don’t remove users, just stop their servers
removeNamedServers: false
maxAge: 0 # Don’t cull based on age, only idle time

proxy:
service:
type: ClusterIP

singleuser:

Use your custom user service account

serviceAccountName: jupyterhub-user-sa
image:
name: notebookacr.azurecr.io/jupyter-spark-notebook
tag: v4.1spark
pullPolicy: Always

Jupyter configuration

defaultUrl: “/lab”
extraEnv:

Set Spark configuration for auto-initialization

PYSPARK_PYTHON: “/opt/conda/bin/python”
PYSPARK_DRIVER_PYTHON: “/opt/conda/bin/python”

Spark auto-initialization via IPython startup script

lifecycleHooks:
postStart:
exec:
command:

  • “bash”
  • “-c”
  • |
    echo “Setting up Spark auto-initialization…”
    mkdir -p /home/jovyan/.ipython/profile_default/startup
        cat > /home/jovyan/.ipython/profile_default/startup/00-spark-init.py << 'EOF'
        # Auto-initialize Spark session for JupyterHub users
        import os
        print("=== Spark Auto-Initialization ===")
        username = os.environ.get('JUPYTERHUB_USER', 'defaultuser')
        print("Initializing Spark for user:", username)

        try:
            from pyspark.sql import SparkSession
            spark = SparkSession.builder.appName('SparkApp-' + username).getOrCreate()
            sc = spark.sparkContext
            print("✓ Spark session initialized successfully")
            print("✓ Spark version:", spark.version)
            print("✓ Application name:", sc.appName)
            print("✓ Variables available: spark, sc")
            print("=" * 40)
        except Exception as e:
            print("⚠ Spark initialization failed:", str(e))
            print("You can manually create Spark session if needed")
        EOF
        
        chown -R jovyan:jovyan /home/jovyan/.ipython
        echo "✓ Spark auto-initialization setup completed"

cmd: [‘/opt/conda/bin/python’, ‘-m’, ‘jupyterhub.singleuser’]

Disable default RBAC creation - use your custom ones

rbac:
create: false

Production resources and settings

scheduling:
userScheduler:
enabled: false

prePuller:
hook:
enabled: false
continuous:
enabled: false

debug:
enabled: true

Nginx: ingress.yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: jupyterhub-ingress

namespace: jupyterhub

annotations:

kubernetes.azure.com/tls-cert-keyvault-uri: https://jupyterhub-kv.vault.azure.net/certificates/JH-CERT

nginx.ingress.kubernetes.io/ssl-redirect: "true"

nginx.ingress.kubernetes.io/configuration-snippet: |

  proxy_set_header X-Real-IP $remote_addr;

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  proxy_set_header X-Forwarded-Proto $scheme;

  proxy_set_header Host $host;

spec:

ingressClassName: webapprouting.kubernetes.azure.com

tls:

- hosts:

    - "staging.example.com"

  secretName: keyvault-jupyterhub-ingress

rules:

\# Primary JupyterHub endpoint

- host: "staging.example.com"

  http:

    paths:

      - path: /

        pathType: Prefix

        backend:

          service:

            name: proxy-public

            port:

              number: 80

\# Wildcard for user subdomains (staging.example.com, etc.)

- host: "\*.staging.example.com"

  http:

    paths:

      - path: /

        pathType: Prefix

        backend:

          service:

            name: proxy-public

            port:

              number: 80

\# Wildcard for user subdomains (staging.example.com, etc.)

- host: "staging.example.com"

  http:

    paths:

      - path: /

        pathType: Prefix

        backend:

          service:

            name: proxy-public

            port:

              number: 80