Hub readiness probe/Successive pod launches fail after upgrading to version 4.0.2+ (custom image)

JupyterHub
,
1

I’m upgrading from version 3.0.0 to version 5.3.0 of Jupyterhub using the Z2JH helm chart. From upgrading to version 4.0.2 and beyond, the Jupyterhub pod starts but never becomes ready.
There is also behavior where the pods can be launched and started initially, but fail to start on successive launches.

The following is my configuration:
Helm Chart:

# Full file is a .gotmpl template; sensitive values are redacted.
jupyterhub:
  hub:
    # Core configuration from values file
    config:
      KubeSpawner:
        slug_scheme: escape
        user_namespace_template: "jupyter-{username}"
      Authenticator:
        allow_all: true
        admin_users:
          - dta001
          - pjamason
          - n2akiti
          # ... and other admins
      # NOTE: GoogleOAuthenticator is configured but not used, as SAML is set in extraConfig.

    # Using a Recreate strategy for upgrades
    deploymentStrategy:
      type: Recreate

    # Using an external PostgreSQL database
    db:
      type: postgres
      upgrade: true
      url: postgresql+psycopg2://jupyterhub:[url]@{{ .Release.Name }}-postgresql.jupyterhub.svc.cluster.local:5432/jupyterhub
      password: [password]

    # Loading the main python configuration from a separate file
    extraConfig:
      config: |
        {{ readFile "config.py" | indent 8 }}

    # Custom environment variables
    extraEnv:
      AWSED_BASE_URL: {{ .Environment.Values.awsed_qa.endpoint }}/
      AWSED_API_KEY: “[key]”
      SAML_HTTPS_OVERRIDE: "yes"

    extraVolumes:
      - name: jupyterhub-saml-auth
        nfs:
          server: {{ .Environment.Values.nfs.server }}
          path: /export/apps/jupyterhub/saml/etc
      # ... other NFS volume definitions
      - name: datahub-config
        nfs:
          server: {{ .Environment.Values.nfs.server }}
          path: /export/apps/jupyterhub/dsmlp/datahub-config-qa
    extraVolumeMounts:
      - name: theme
        mountPath: /usr/local/share/jupyterhub/templates
        subPath: templates
      # ... other volume mounts
      - name: datahub-config
        mountPath: /etc/datahub

    # Custom Hub container image
    image:
      name: r-its-ats-docker.repository.ucsd.edu/ats/datahub
      tag: fix-testing-errors
      pullPolicy: Always
      pullSecrets:
       - name: "artifactory-datahub-qa"

    # Custom services
    services:
      disk-quota-service:
        url: 'http://{{ .Release.Name }}-disk-quota-service.jupyterhub.svc.cluster.local:5000'
        api_token: “[token]”
      manual-resetter:
        admin: true
        url: 'http://{{ .Release.Name }}-manual-resetter.jupyterhub.svc.cluster.local:5001'
        api_token: “[token]”
        
    readinessProbe:
      enabled: true
      initialDelaySeconds: 0
      periodSeconds: 2
      failureThreshold: 1000
      timeoutSeconds: 1

  # Proxy configured to use Recreate strategy
  proxy:
    secretToken: “secret token”
    deploymentStrategy:
      type: Recreate

  # Debug and culler settings
  cull:
    enabled: true
    timeout: 3600
    every: 600
  debug:
    enabled: true

# ... other sections are using default chart values and have been omitted.

The custom config file injected into the .gotmpl file:

import kubespawner
from kubernetes import config, client
from datahub_config import AwsedClient, fetch_profiles # Custom import, important
from tornado.log import app_log
import os
import logging
from jupyterhub.handlers.base import BaseHandler
from tornado import template
import requests

# --- Core Spawner and Auth Configuration (Highly Relevant) ---

"""Set user template system"""
class PatchedSpawner(kubespawner.KubeSpawner):
    # Custom spawner class is a major customization.
    async def _ensure_namespace(self):
        return

c.JupyterHub.spawner_class = PatchedSpawner

c.KubeSpawner.enable_user_namespaces = True
c.KubeSpawner.pod_name_template = 'dsmlp-jupyter-{username}'
c.KubeSpawner.user_namespace_template = '{username}'
c.KubeSpawner.start_timeout = 300
c.KubeSpawner.slug_scheme = "escape"

# --- SAML Authenticator Configuration ---
def extract_username(acs_handler, attributes):
    return attributes['urn:mace:ucsd.edu:sso:ad:username'][0]

c.SAMLAuthenticator.saml_settings_path = '/etc/saml'
c.SAMLAuthenticator.login_service = 'your UC San Diego account (username@ucsd.edu)'
c.SAMLAuthenticator.extract_username = extract_username
c.JupyterHub.authenticator_class = 'jupyterhub_saml_auth.SAMLAuthenticator'
c.OAuthenticator.allow_all = True # Note: this is also set

# --- API Clients and Role Loading ---
config.load_incluster_config()
v1 = client.CoreV1Api()
awsed_client = AwsedClient()

c.JupyterHub.load_roles = [
    {
        "name": "user",
        "description": "Allow users to access services in addition to default perms",
        "scopes": ["self", "list:services", "list:users", "read:services", "access:services"],
    },
    {
        "name": "jupyter-idle-culler",
        "scopes": ["list:users", "read:users:activity"],
        "services": ["jupyterhub-idle-culler"]
    },
    {
        "name": "server",
        "scopes": ["users:activity!user", "access:servers!server"],
    }
]

# --- Custom Hooks (Run during user spawning) ---
def modify_pod_hook(spawner, pod):
    selected = spawner.user_options
    app_log.info('selected', selected)
    return pod

c.KubeSpawner.modify_pod_hook = modify_pod_hook

c.KubeSpawner.profile_list = lambda spawner: fetch_profiles(
    spawner=spawner,
    client=awsed_client,
    v1=v1,
    jupyterhub_service_name=os.getenv("HELM_RELEASE_NAME") + "-jupyterhub-hub.jupyterhub.svc.cluster.local"
)


# --- Custom API Handlers (Potential for startup delays) ---
class StatusHandler(BaseHandler):
    # Implementation omitted for brevity...
    pass

class DatasetsHandler(BaseHandler):
    # This handler makes an external network request to AWSED_BASE_URL.
    # Implementation omitted for brevity...
    pass

class DeploymentHandler(BaseHandler):
    # This handler is very complex and makes network requests to an internal
    # 'deployment-controller' service during its initialization.
    # Implementation omitted for brevity...
    pass

class FeedbackHandler(BaseHandler):
    # Implementation omitted for brevity...
    pass

# Registering the custom handlers.
c.JupyterHub.extra_handlers = [
    (r'/status', StatusHandler),
    (r'/datasets', DatasetsHandler),
    (r'/deployment', DeploymentHandler),
    (r'/feedback', FeedbackHandler)
]
2

Can you share your full debug logs for the failed hub pod?

3

In the logs it shows a redirect loop due to a version mismatch in versions, but I believe that this is a red herring, since I’m able to launch user pods fine in some instances. After an arbitrary number of user pod launches, the user is unable to start any new pods.

$ k get pods -n jupyterhub
datahub-qa-disk-quota-service-68d5864678-m8h9k    1/1     Running   0               4m50s
datahub-qa-jupyterhub-hub-7cb4f7767-tk7kd         1/1     Running   0               4m50s
datahub-qa-jupyterhub-proxy-f87dd7d7f-x2jrt       1/1     Running   0               4m50s
datahub-qa-manual-resetter-74687567dc-mf9fr       1/1     Running   0               4m50s
datahub-qa-postgresql-0                           1/1     Running   0               4m50s

Describe Output:

Name:                 datahub-qa-jupyterhub-hub-7cb4f7767-tk7kd
Namespace:            jupyterhub
Priority:             10
Priority Class Name:  normal
Service Account:      datahub-qa-jupyterhub-hub
Node:                 its-dsmlp-n604.ucsd.edu/172.17.70.234
Start Time:           Tue, 30 Sep 2025 08:28:09 -0700
Labels:               app.kubernetes.io/instance=datahub-qa
                      app.kubernetes.io/managed-by=Helm
                      app.kubernetes.io/name=jupyterhub
                      app.kubernetes.io/version=5.3.0
                      helm.sh/chart=jupyterhub-4.2.0
                      hub.jupyter.org/network-access-proxy-api=true
                      hub.jupyter.org/network-access-proxy-http=true
                      hub.jupyter.org/network-access-singleuser=true
                      pod-template-hash=7cb4f7767
Annotations:          checksum/config-map: 0c627c7fa571e38c291b814a3ae9d1e30ce7d4815e1193a05d747241a69fd9f7
                      checksum/secret: c45ad3d8fe2dc12b172fa6b05dffc2d68874e8ca3d5bb413a9baa05679494886
                      cni.projectcalico.org/containerID: 9ff56c9206713d026eba19df2d1705ddab7b1b83491452981b4d42a973cb865e
                      cni.projectcalico.org/podIP: 10.41.212.66/32
                      cni.projectcalico.org/podIPs: 10.41.212.66/32
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   10.41.212.66
IPs:
  IP:           10.41.212.66
Controlled By:  ReplicaSet/datahub-qa-jupyterhub-hub-7cb4f7767
Containers:
  hub:
    Container ID:  cri-o://4563fe19382b1c6cee18a1e4ab88c743d12985ca70349f4e34b52fc58b10e352
    Image:         r-its-ats-docker.repository.ucsd.edu/ats/datahub:fix-testing-errors
    Image ID:      r-its-ats-docker.repository.ucsd.edu/ats/datahub@sha256:bc8bdeb935918e201078189f2f9049bf04681a504963608e511c1f2d87ba257d
    Port:          8081/TCP
    Host Port:     0/TCP
    Args:
      jupyterhub
      --config
      /usr/local/etc/jupyterhub/jupyterhub_config.py
      --debug
      --upgrade-db
    State:          Running
      Started:      Tue, 30 Sep 2025 08:28:27 -0700
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30
    Readiness:      http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000
    Environment:
      PYTHONUNBUFFERED:        1
      HELM_RELEASE_NAME:       datahub-qa
      POD_NAMESPACE:           jupyterhub (v1:metadata.namespace)
      CONFIGPROXY_AUTH_TOKEN:  <set to the key 'hub.config.ConfigurableHTTPProxy.auth_token' in secret 'datahub-qa-jupyterhub-hub'>  Optional: false
      AWSED_API_KEY:           AWSEd api_key=PKWEGXZhrsrNMZdexTuQVsJ6
      AWSED_BASE_URL:          https://awsed-qa.ucsd.edu/api/
      SAML_HTTPS_OVERRIDE:     yes
    Mounts:
      /etc/datahub from datahub-config (rw)
      /etc/saml from jupyterhub-saml-auth (rw)
      /home/jovyan/.jupyter/custom from theme (rw,path="static")
      /home/jovyan/jupyterhub-saml-auth from jupyterhub-saml-auth-app (rw)
      /srv/nbmessages/admin from nbmessage-board (rw)
      /usr/local/etc/jupyterhub/config/ from config (rw)
      /usr/local/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
      /usr/local/etc/jupyterhub/secret/ from secret (rw)
      /usr/local/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
      /usr/local/share/jupyterhub/datasets from theme (rw,path="datasets")
      /usr/local/share/jupyterhub/static/custom from theme (rw,path="static")
      /usr/local/share/jupyterhub/status from theme (rw,path="status")
      /usr/local/share/jupyterhub/templates from theme (rw,path="templates")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s6m2p (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      datahub-qa-jupyterhub-hub
    Optional:  false
  secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  datahub-qa-jupyterhub-hub
    Optional:    false
  jupyterhub-saml-auth:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    its-dsmlp-fs04.ucsd.edu
    Path:      /export/apps/jupyterhub/saml/etc
    ReadOnly:  false
  jupyterhub-saml-auth-app:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    its-dsmlp-fs04.ucsd.edu
    Path:      /export/apps/jupyterhub/saml/jupyterhub-saml-auth
    ReadOnly:  false
  nbmessage-board:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    its-dsmlp-fs04.ucsd.edu
    Path:      /export/apps/nbmessage-board
    ReadOnly:  false
  theme:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    its-dsmlp-fs04.ucsd.edu
    Path:      /export/apps/jupyterhub/dsmlp/jupyterhub-theme-qa
    ReadOnly:  false
  datahub-config:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    its-dsmlp-fs04.ucsd.edu
    Path:      /export/apps/jupyterhub/dsmlp/datahub-config-qa
    ReadOnly:  false
  kube-api-access-s6m2p:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              dsmlp-apps=true
Tolerations:                 hub.jupyter.org/dedicated=core:NoSchedule
                             hub.jupyter.org_dedicated=core:NoSchedule
                             node-type=apps:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  5m22s                default-scheduler  Successfully assigned jupyterhub/datahub-qa-jupyterhub-hub-7cb4f7767-tk7kd to its-dsmlp-n604.ucsd.edu
  Normal   Pulling    5m21s                kubelet            Pulling image "r-its-ats-docker.repository.ucsd.edu/ats/datahub:fix-testing-errors"
  Normal   Pulled     5m5s                 kubelet            Successfully pulled image "r-its-ats-docker.repository.ucsd.edu/ats/datahub:fix-testing-errors" in 16.722s (16.722s including waiting). Image size: 1445000290 bytes.
  Normal   Created    5m4s                 kubelet            Created container: hub
  Normal   Started    5m4s                 kubelet            Started container hub
  Warning  Unhealthy  5m1s (x4 over 5m3s)  kubelet            Readiness probe failed: Get "http://10.41.212.66:8081/hub/health": dial tcp 10.41.212.66:8081: connect: connection refused

Main Hub Pod Logs (On Failure):

[D 2025-09-30 15:28:28.733 JupyterHub application:908] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /app
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: config
[D 2025-09-30 15:28:29.615 JupyterHub application:929] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[W 2025-09-30 15:28:29.733 JupyterHub app:1715] JupyterHub.extra_handlers is deprecated in JupyterHub 3.1. Please use JupyterHub services to register additional HTTP endpoints.
[I 2025-09-30 15:28:29.733 JupyterHub app:3354] Running JupyterHub version 5.3.0
[I 2025-09-30 15:28:29.733 JupyterHub app:3384] Using Authenticator: jupyterhub_saml_auth.authenticator.SAMLAuthenticator-0.4.0
[I 2025-09-30 15:28:29.734 JupyterHub app:3384] Using Spawner: builtins.PatchedSpawner
[I 2025-09-30 15:28:29.734 JupyterHub app:3384] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.3.0
/usr/local/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
  validate_schema(_schema)
[D 2025-09-30 15:28:29.736 JupyterHub app:2006] Connecting to db: postgresql+psycopg2://jupyterhub:[redacted]@datahub-qa-postgresql.jupyterhub.svc.cluster.local:5432/jupyterhub
[D 2025-09-30 15:28:29.854 JupyterHub orm:1509] database schema version found: 4621fec11365
[D 2025-09-30 15:28:29.934 JupyterHub orm:1509] database schema version found: 4621fec11365
[D 2025-09-30 15:28:30.048 JupyterHub app:2346] Loading roles into database
[D 2025-09-30 15:28:30.048 JupyterHub app:2355] Loading role user
[D 2025-09-30 15:28:30.048 JupyterHub app:2357] Overriding default role user
[D 2025-09-30 15:28:30.053 JupyterHub app:2355] Loading role jupyter-idle-culler
[D 2025-09-30 15:28:30.055 JupyterHub app:2355] Loading role server
[D 2025-09-30 15:28:30.055 JupyterHub app:2357] Overriding default role server
[I 2025-09-30 15:28:30.345 JupyterHub app:2927] Creating service jupyterhub-idle-culler without oauth.
[I 2025-09-30 15:28:30.369 JupyterHub app:2889] Creating service disk-quota-service with oauth_client_id=service-disk-quota-service
[I 2025-09-30 15:28:30.393 JupyterHub provider:663] Updating oauth client service-disk-quota-service
[I 2025-09-30 15:28:30.422 JupyterHub app:2889] Creating service manual-resetter with oauth_client_id=service-manual-resetter
[I 2025-09-30 15:28:30.433 JupyterHub provider:663] Updating oauth client service-manual-resetter
[D 2025-09-30 15:28:30.473 JupyterHub app:2679] Not duplicating token <APIToken('efae...', service='disk-quota-service', client_id='jupyterhub')>
[D 2025-09-30 15:28:30.479 JupyterHub app:2679] Not duplicating token <APIToken('5a0a...', service='manual-resetter', client_id='jupyterhub')>
[D 2025-09-30 15:28:30.480 JupyterHub app:2693] Purging expired APITokens
[D 2025-09-30 15:28:30.485 JupyterHub app:2693] Purging expired OAuthCodes
[D 2025-09-30 15:28:30.492 JupyterHub app:2693] Purging expired Shares
[D 2025-09-30 15:28:30.494 JupyterHub app:2693] Purging expired ShareCodes
[D 2025-09-30 15:28:30.499 JupyterHub app:2467] Loading role assignments from config
[I 2025-09-30 15:28:30.547 JupyterHub cache:154] Creating cache['type'] = in-memory
[D 2025-09-30 15:28:30.553 JupyterHub app:2978] Initializing spawners
[D 2025-09-30 15:28:30.565 JupyterHub app:3128] Loaded users:
    
[I 2025-09-30 15:28:30.565 JupyterHub app:3424] Initialized 0 spawners in 0.012 seconds
[I 2025-09-30 15:28:30.583 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-09-30 15:28:30.584 JupyterHub metrics:425] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2025-09-30 15:28:30.586 JupyterHub metrics:425] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I 2025-09-30 15:28:30.586 JupyterHub app:3711] Not starting proxy
[D 2025-09-30 15:28:30.587 JupyterHub proxy:925] Proxy: Fetching GET http://datahub-qa-jupyterhub-proxy-api:8001/api/routes
[D 2025-09-30 15:28:30.597 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-09-30 15:28:30.597 JupyterHub app:3747] Hub API listening on http://:8081/hub/
[I 2025-09-30 15:28:30.597 JupyterHub app:3749] Private Hub API connect url http://datahub-qa-jupyterhub-hub:8081/hub/
[I 2025-09-30 15:28:30.597 JupyterHub app:3623] Starting managed service jupyterhub-idle-culler
[I 2025-09-30 15:28:30.597 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2025-09-30 15:28:30.599 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2025-09-30 15:28:30.600 JupyterHub spawner:1693] Polling subprocess every 30s
[I 2025-09-30 15:28:30.600 JupyterHub app:3632] Adding external service disk-quota-service at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000
[D 2025-09-30 15:28:30.601 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/
[D 2025-09-30 15:28:30.613 JupyterHub utils:314] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded with 302
[D 2025-09-30 15:28:30.613 JupyterHub utils:333] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded in 0.01s
[I 2025-09-30 15:28:30.613 JupyterHub app:3632] Adding external service manual-resetter at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001
[D 2025-09-30 15:28:30.614 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/
[D 2025-09-30 15:28:30.628 JupyterHub utils:314] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded with 302
[D 2025-09-30 15:28:30.629 JupyterHub utils:333] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded in 0.01s
[D 2025-09-30 15:28:30.629 JupyterHub proxy:389] Fetching routes to check
[D 2025-09-30 15:28:30.629 JupyterHub proxy:925] Proxy: Fetching GET http://datahub-qa-jupyterhub-proxy-api:8001/api/routes
[D 2025-09-30 15:28:30.634 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-09-30 15:28:30.634 JupyterHub proxy:392] Checking routes
[I 2025-09-30 15:28:30.634 JupyterHub proxy:477] Adding route for Hub: / => http://datahub-qa-jupyterhub-hub:8081
[W 2025-09-30 15:28:30.636 JupyterHub proxy:445] Adding missing route for disk-quota-service (Server(url=http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/, bind_url=http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/))
[W 2025-09-30 15:28:30.637 JupyterHub proxy:445] Adding missing route for manual-resetter (Server(url=http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/, bind_url=http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/))
[D 2025-09-30 15:28:30.638 JupyterHub proxy:925] Proxy: Fetching POST http://datahub-qa-jupyterhub-proxy-api:8001/api/routes/
[I 2025-09-30 15:28:30.639 JupyterHub proxy:312] Adding service disk-quota-service to proxy /services/disk-quota-service/ => http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000
[D 2025-09-30 15:28:30.639 JupyterHub proxy:925] Proxy: Fetching POST http://datahub-qa-jupyterhub-proxy-api:8001/api/routes/services/disk-quota-service
[I 2025-09-30 15:28:30.640 JupyterHub proxy:312] Adding service manual-resetter to proxy /services/manual-resetter/ => http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001
[D 2025-09-30 15:28:30.640 JupyterHub proxy:925] Proxy: Fetching POST http://datahub-qa-jupyterhub-proxy-api:8001/api/routes/services/manual-resetter
[I 2025-09-30 15:28:30.654 JupyterHub app:3780] JupyterHub is now running, internal Hub API at http://datahub-qa-jupyterhub-hub:8081/hub/
[D 2025-09-30 15:28:30.655 JupyterHub app:3347] It took 1.934 seconds for the Hub to start
[D 2025-09-30 15:28:30.930 JupyterHub base:366] Recording first activity for <APIToken('faa3...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2025-09-30 15:28:30.937 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 11.77ms
[D 2025-09-30 15:28:30.946 JupyterHub scopes:1013] Checking access to /hub/api/users via scope list:users
[I 2025-09-30 15:28:30.966 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 24.33ms
[D 2025-09-30 15:39:24.685 JupyterHub spawner:3653] Applying KubeSpawner override for profile 'TEST_NBGRADER - DTA [FA23]'
[D 2025-09-30 15:39:24.685 JupyterHub spawner:3631] .. overriding KubeSpawner value namespace=n2akiti
[D 2025-09-30 15:39:24.685 JupyterHub spawner:3631] .. overriding KubeSpawner value args=['--', '/opt/k8s-support/bin/initenv-createhomedir.sh', 'jupyterhub-singleuser', '--KernelRestarter.restart_limit=0']
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value cmd=/opt/k8s-support/bin/tini-wrapper
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value cpu_guarantee=1
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value cpu_limit=2
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value image_spec=ghcr.io/ucsd-ets/rstudio-notebook:2023.4-stable
[W 2025-09-30 15:39:24.686 JupyterHub spawner:1994] KubeSpawner.image_spec is deprecated in 0.10. Use KubeSpawner.image
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value image_pull_policy=Always
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value mem_guarantee=2G
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value mem_limit=4G
[D 2025-09-30 15:39:24.686 JupyterHub spawner:3631] .. overriding KubeSpawner value extra_resource_guarantees={'nvidia.com/gpu': 0}
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value priority_class_name=normal
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value extra_resource_limits={'nvidia.com/gpu': 0}
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value extra_labels={'dsmlp/user': 'n2akiti', 'dsmlp/course': 'TEST_NBGRADER'}
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value environment={'JUPYTERHUB_ACTIVITY_URL': 'http://datahub-qa-jupyterhub-hub.jupyterhub.svc.cluster.local:8081/hub/api/users/n2akiti/activity', 'JUPYTERHUB_API_URL': 'http://datahub-qa-jupyterhub-hub.jupyterhub.svc.cluster.local:8081/hub/api', 'NB_USER': 'n2akiti', 'XDG_CACHE_HOME': '/home/n2akiti/.cache/', 'USER': 'n2akiti', 'LOGNAME': 'n2akiti', 'JUPYTERHUB_USER': 'n2akiti'}
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value init_containers=[]
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value uid=1374478
[D 2025-09-30 15:39:24.687 JupyterHub spawner:3631] .. overriding KubeSpawner value lifecycle_hooks={}
[I 2025-09-30 15:39:24.692 JupyterHub spawner:3153] Pod is being modified via modify_pod_hook
[I 2025-09-30 15:39:24.692 JupyterHub <string>:152] selected
[I 2025-09-30 15:39:24.692 JupyterHub spawner:2890] Attempting to create pod dsmlp-jupyter-n2akiti, with timeout 30
[I 2025-09-30 15:39:25.131 JupyterHub log:192] 302 POST /hub/spawn/n2akiti?_xsrf=[secret] -> /hub/spawn-pending/n2akiti?_xsrf=[secret] (n2akiti@172.17.86.170) 1017.16ms
[D 2025-09-30 15:39:25.199 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/n2akiti via scope servers!server=n2akiti/
[I 2025-09-30 15:39:25.199 JupyterHub pages:400] n2akiti is pending spawn
[I 2025-09-30 15:39:25.202 JupyterHub log:192] 200 GET /hub/spawn-pending/n2akiti?_xsrf=[secret] (n2akiti@172.17.86.170) 18.78ms
[D 2025-09-30 15:39:25.253 JupyterHub scopes:1013] Checking access to /hub/api/users/n2akiti/server/progress via scope read:servers!server=n2akiti/
[D 2025-09-30 15:39:25.262 JupyterHub spawner:2672] progress generator: dsmlp-jupyter-n2akiti
[D 2025-09-30 15:39:25.851 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 1.14ms
[D 2025-09-30 15:39:26.926 JupyterHub reflector:390] pods watcher timeout
[D 2025-09-30 15:39:26.926 JupyterHub reflector:304] Connecting pods watcher
[D 2025-09-30 15:39:27.028 JupyterHub reflector:390] events watcher timeout
[D 2025-09-30 15:39:27.028 JupyterHub reflector:304] Connecting events watcher
[D 2025-09-30 15:39:27.852 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 1.12ms
[D 2025-09-30 15:39:29.474 JupyterHub spawner:3254] pod n2akiti/dsmlp-jupyter-n2akiti events before launch: 2025-09-30T15:39:25Z [Normal] Successfully assigned n2akiti/dsmlp-jupyter-n2akiti to its-dsmlp-n27.ucsd.edu
    2025-09-30T15:39:26Z [Normal] Container image "ghcr.io/ucsd-ets/k8s-support:updated-passwd" already present on machine
    2025-09-30T15:39:26Z [Normal] Created container: init-support
    2025-09-30T15:39:26Z [Normal] Started container init-support
    2025-09-30T15:39:27Z [Normal] Pulling image "ghcr.io/ucsd-ets/rstudio-notebook:2023.4-stable"
    2025-09-30T15:39:28Z [Normal] Successfully pulled image "ghcr.io/ucsd-ets/rstudio-notebook:2023.4-stable" in 639ms (639ms including waiting). Image size: 6083511462 bytes.
    2025-09-30T15:39:28Z [Normal] Created container: notebook
    2025-09-30T15:39:28Z [Normal] Started container notebook
[D 2025-09-30 15:39:29.480 JupyterHub spawner:1693] Polling subprocess every 30s
[D 2025-09-30 15:39:29.485 JupyterHub utils:297] Waiting 30s for server at http://10.41.226.230:8888/user/n2akiti/api
[D 2025-09-30 15:39:29.852 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 1.91ms
[D 2025-09-30 15:39:29.853 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 2.12ms
[D 2025-09-30 15:39:30.655 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/
[D 2025-09-30 15:39:30.655 JupyterHub proxy:925] Proxy: Fetching GET http://datahub-qa-jupyterhub-proxy-api:8001/api/routes
[D 2025-09-30 15:39:30.659 JupyterHub utils:314] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded with 302
[D 2025-09-30 15:39:30.660 JupyterHub utils:333] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded in 0.00s
[D 2025-09-30 15:39:30.660 JupyterHub app:2970] External service disk-quota-service running at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000
[D 2025-09-30 15:39:30.661 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/
[D 2025-09-30 15:39:30.662 JupyterHub proxy:392] Checking routes
[D 2025-09-30 15:39:30.674 JupyterHub utils:314] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded with 302
[D 2025-09-30 15:39:30.674 JupyterHub utils:333] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded in 0.01s
[D 2025-09-30 15:39:30.674 JupyterHub app:2970] External service manual-resetter running at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001
[I 2025-09-30 15:39:30.883 JupyterHub log:192] 200 GET /hub/api (@10.41.226.230) 1.40ms
[D 2025-09-30 15:39:30.912 JupyterHub base:366] Recording first activity for <APIToken('cb5a...', user='n2akiti', client_id='jupyterhub')>
[D 2025-09-30 15:39:30.922 JupyterHub scopes:1013] Checking access to /hub/api/users/n2akiti/activity via scope users:activity!user=n2akiti
[D 2025-09-30 15:39:30.931 JupyterHub users:1006] Activity for user n2akiti: 2025-09-30T15:39:28.933146Z
[D 2025-09-30 15:39:30.931 JupyterHub users:1024] Activity on server n2akiti/: 2025-09-30T15:39:28.933146Z
[I 2025-09-30 15:39:30.936 JupyterHub log:192] 200 POST /hub/api/users/n2akiti/activity (n2akiti@::ffff:172.17.70.237) 32.08ms
[D 2025-09-30 15:39:31.091 JupyterHub utils:333] Server at http://10.41.226.230:8888/user/n2akiti/api responded in 1.61s
[I 2025-09-30 15:39:31.092 JupyterHub base:1126] User n2akiti took 6.964 seconds to start
[I 2025-09-30 15:39:31.092 JupyterHub proxy:331] Adding user n2akiti to proxy /user/n2akiti/ => http://10.41.226.230:8888
[D 2025-09-30 15:39:31.092 JupyterHub proxy:925] Proxy: Fetching POST http://datahub-qa-jupyterhub-proxy-api:8001/api/routes/user/n2akiti
[I 2025-09-30 15:39:31.097 JupyterHub users:899] Server n2akiti is ready
[I 2025-09-30 15:39:31.099 JupyterHub log:192] 200 GET /hub/api/users/n2akiti/server/progress?_xsrf=[secret] (n2akiti@172.17.86.170) 5857.94ms
[D 2025-09-30 15:39:31.123 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/n2akiti via scope servers!server=n2akiti/
[I 2025-09-30 15:39:31.124 JupyterHub log:192] 302 GET /hub/spawn-pending/n2akiti?_xsrf=[secret] -> /user/n2akiti/ (n2akiti@172.17.86.170) 15.21ms
[I 2025-09-30 15:39:31.134 JupyterHub log:192] 302 GET /user/n2akiti/ -> /hub/user/n2akiti/ (@172.17.86.170) 2.08ms
[I 2025-09-30 15:39:31.159 JupyterHub log:192] 302 GET /hub/user/n2akiti/ -> /user/n2akiti/?redirects=1 (n2akiti@172.17.86.170) 13.90ms
[I 2025-09-30 15:39:31.213 JupyterHub log:192] 302 GET /user/n2akiti/?redirects=1 -> /hub/user/n2akiti/?redirects=1 (@172.17.86.170) 2.12ms
[W 2025-09-30 15:39:31.239 JupyterHub base:1867] Redirect loop detected on /hub/user/n2akiti/?redirects=1
[D 2025-09-30 15:39:31.851 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 1.25ms
[I 2025-09-30 15:39:33.241 JupyterHub log:192] 302 GET /hub/user/n2akiti/?redirects=1 -> /user/n2akiti/?redirects=2 (n2akiti@172.17.86.170) 2015.45ms
[I 2025-09-30 15:39:33.271 JupyterHub log:192] 302 GET /user/n2akiti/?redirects=2 -> /hub/user/n2akiti/?redirects=2 (@172.17.86.170) 2.18ms
[W 2025-09-30 15:39:33.291 JupyterHub web:1873] 500 GET /hub/user/n2akiti/?redirects=2 (172.17.86.170): Redirect loop detected. Notebook has jupyterhub version 1.4.1, but the Hub expects 5.3.0. Try installing jupyterhub==5.3.0 in the user environment if you continue to have problems.
[D 2025-09-30 15:39:33.292 JupyterHub base:1542] Using default error template for 500
[E 2025-09-30 15:39:33.323 JupyterHub log:184] {
      "Connection": "close",
      "X-Real-Ip": "172.17.86.170",
      "X-Forwarded-For": "172.17.86.170",
      "X-Forwarded-Host": "datahub.ucsd.edu",
      "X-Forwarded-Port": "443",
      "X-Forwarded-Proto": "https",
      "Host": "datahub.ucsd.edu",
      "Cache-Control": "max-age=0",
      "Upgrade-Insecure-Requests": "1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Sec-Ch-Ua": "\"Chromium\";v=\"140\", \"Not=A?Brand\";v=\"24\", \"Google Chrome\";v=\"140\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua-Platform": "\"macOS\"",
      "Referer": "https://datahub.ucsd.edu/hub/spawn-pending/n2akiti?_xsrf=MnwxOjB8MTA6MTc1OTI0NjY2NHw1Ol94c3JmfDg4Ok1ERTBObVZsWm1RME1EY3pOR0ptTlRrd1lUZ3lNMk5oWlRVMU1XVTJaR0k2WlRWbE1EVTBaRGhoTUdVeE5HSTROR0poTlRnMk9UWmpaRGMzTVdZM05qRT18MTUxZDczMWYyNjA1ZTViNWQ4YzY5NjVjOTcxN2JhNmY5YTE5ZGVjNmY3NzZlZDA1NzIwODY0N2E1MGY3OWQ3Nw",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Accept-Language": "en-US,en;q=0.9,es;q=0.8",
      "Priority": "u=0, i",
      "Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; service-disk-quota-service-oauth-state=[secret]; jupyterhub-session-id=[secret]; _gcl_au=[secret]; _rdt_uuid=[secret]; _uetvid=[secret]; _ga_8BP9YL1JLD=[secret]; nmstat=[secret]; _gid=[secret]; _ga_DQLWSKCKE6=[secret]; _ga_PWJGRGMV0T=[secret]; _ga_NBFHGPEBGT=[secret]; _ga=[secret]; _ga_E3FLY5PZSP=[secret]; jupyterhub_release=[secret]"
    }
[E 2025-09-30 15:39:33.323 JupyterHub log:192] 500 GET /hub/user/n2akiti/?redirects=2 (n2akiti@172.17.86.170) 44.55ms
[D 2025-09-30 15:39:33.851 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 0.92ms
4

The JupyterHub package inside your image is too old. There’s some backwards compatibility (e.g. 4.x should work), but not 1.x.

1 Like
5

Even when I attempt to launch a notebook with version 4.x.x or higher, I still get the same issue. I believe that there may be an underlying issue with the configuration/XSRF tokens. Even when I launch a notebook that is version 1.x.x, I’m able to launch it successfully initially, its just that successive launches fail.

I 2025-10-06 14:49:29.416 JupyterHub log:192] 200 POST /hub/api/users/n2akiti/activity (n2akiti@10.46.173.22) 29.34ms
[D 2025-10-06 14:49:29.854 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 2.65ms
[D 2025-10-06 14:49:29.855 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 3.16ms
[D 2025-10-06 14:49:30.655 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/
[D 2025-10-06 14:49:30.655 JupyterHub proxy:925] Proxy: Fetching GET http://datahub-qa-jupyterhub-proxy-api:8001/api/routes
[D 2025-10-06 14:49:30.659 JupyterHub utils:314] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded with 302
[D 2025-10-06 14:49:30.660 JupyterHub utils:333] Server at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000/services/disk-quota-service/ responded in 0.00s
[D 2025-10-06 14:49:30.660 JupyterHub app:2970] External service disk-quota-service running at http://datahub-qa-disk-quota-service.jupyterhub.svc.cluster.local:5000
[D 2025-10-06 14:49:30.661 JupyterHub utils:297] Waiting 1s for server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/
[D 2025-10-06 14:49:30.662 JupyterHub proxy:392] Checking routes
[D 2025-10-06 14:49:30.674 JupyterHub utils:314] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded with 302
[D 2025-10-06 14:49:30.675 JupyterHub utils:333] Server at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001/services/manual-resetter/ responded in 0.01s
[D 2025-10-06 14:49:30.675 JupyterHub app:2970] External service manual-resetter running at http://datahub-qa-manual-resetter.jupyterhub.svc.cluster.local:5001
[D 2025-10-06 14:49:31.126 JupyterHub spawner:3254] pod n2akiti/dsmlp-jupyter-n2akiti events before launch: 2025-10-06T14:49:24Z [Normal] Successfully assigned n2akiti/dsmlp-jupyter-n2akiti to its-dsmlp-n26.ucsd.edu
    2025-10-06T14:49:25Z [Normal] Container image "ghcr.io/ucsd-ets/k8s-support:updated-passwd" already present on machine
    2025-10-06T14:49:25Z [Normal] Created container: init-support
    2025-10-06T14:49:25Z [Normal] Started container init-support
    2025-10-06T14:49:25Z [Normal] Pulling image "ghcr.io/ucsd-ets/datascience-notebook:2025.1-stable"
    2025-10-06T14:49:26Z [Normal] Successfully pulled image "ghcr.io/ucsd-ets/datascience-notebook:2025.1-stable" in 983ms (983ms including waiting). Image size: 7626807014 bytes.
    2025-10-06T14:49:26Z [Normal] Created container: notebook
    2025-10-06T14:49:26Z [Normal] Started container notebook
[D 2025-10-06 14:49:31.135 JupyterHub spawner:1693] Polling subprocess every 30s
[D 2025-10-06 14:49:31.139 JupyterHub utils:297] Waiting 30s for server at http://10.46.173.22:8888/user/n2akiti/api
[D 2025-10-06 14:49:31.191 JupyterHub utils:333] Server at http://10.46.173.22:8888/user/n2akiti/api responded in 0.05s
[I 2025-10-06 14:49:31.191 JupyterHub base:1126] User n2akiti took 8.481 seconds to start
[I 2025-10-06 14:49:31.191 JupyterHub proxy:331] Adding user n2akiti to proxy /user/n2akiti/ => http://10.46.173.22:8888
[D 2025-10-06 14:49:31.191 JupyterHub proxy:925] Proxy: Fetching POST http://datahub-qa-jupyterhub-proxy-api:8001/api/routes/user/n2akiti
[I 2025-10-06 14:49:31.195 JupyterHub users:899] Server n2akiti is ready
[I 2025-10-06 14:49:31.196 JupyterHub log:192] 200 GET /hub/api/users/n2akiti/server/progress?_xsrf=[secret] (n2akiti@172.17.86.170) 7414.74ms
[D 2025-10-06 14:49:31.239 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/n2akiti via scope servers!server=n2akiti/
[I 2025-10-06 14:49:31.240 JupyterHub log:192] 302 GET /hub/spawn-pending/n2akiti?_xsrf=[secret] -> /user/n2akiti/ (n2akiti@172.17.86.170) 17.38ms
[I 2025-10-06 14:49:31.249 JupyterHub log:192] 302 GET /user/n2akiti/ -> /hub/user/n2akiti/ (@172.17.86.170) 1.03ms
[I 2025-10-06 14:49:31.279 JupyterHub log:192] 302 GET /hub/user/n2akiti/ -> /user/n2akiti/?redirects=1 (n2akiti@172.17.86.170) 13.35ms
[I 2025-10-06 14:49:31.289 JupyterHub log:192] 302 GET /user/n2akiti/?redirects=1 -> /hub/user/n2akiti/?redirects=1 (@172.17.86.170) 1.65ms
[W 2025-10-06 14:49:31.311 JupyterHub base:1867] Redirect loop detected on /hub/user/n2akiti/?redirects=1
[D 2025-10-06 14:49:31.851 JupyterHub log:192] 200 GET /hub/health (@172.17.70.234) 1.29ms
[I 2025-10-06 14:49:33.313 JupyterHub log:192] 302 GET /hub/user/n2akiti/?redirects=1 -> /user/n2akiti/?redirects=2 (n2akiti@172.17.86.170) 2013.87ms
[I 2025-10-06 14:49:33.328 JupyterHub log:192] 302 GET /user/n2akiti/?redirects=2 -> /hub/user/n2akiti/?redirects=2 (@172.17.86.170) 2.19ms
[W 2025-10-06 14:49:33.367 JupyterHub web:1873] 500 GET /hub/user/n2akiti/?redirects=2 (172.17.86.170): Redirect loop detected. Notebook has jupyterhub version 4.1.5, but the Hub expects 5.3.0. Try installing jupyterhub==5.3.0 in the user environment if you continue to have problems.
[D 2025-10-06 14:49:33.368 JupyterHub base:1542] Using default error template for 500
6

It looks like the hub creates the proxy route /user/n2akiti/, but the proxy isn’t forwarding traffic to the pod. Can you share your proxy logs, and the singleuser logs for the same period?

7

These are the logs for the proxy. They show two user sessions that are created in succession where the first user pod launches successfully, but the second results in a redirect error:

18:08:30.660 [ConfigProxy] info: 200 GET /api/routes 
18:09:06.649 [ConfigProxy] debug: PROXY WEB / to http://datahub-qa-jupyterhub-hub:8081
18:09:06.656 [ConfigProxy] debug: Not recording activity for status 302 on /
18:09:06.671 [ConfigProxy] debug: PROXY WEB /hub/ to http://datahub-qa-jupyterhub-hub:8081
18:09:06.674 [ConfigProxy] debug: PROXY WEB /hub/ to http://datahub-qa-jupyterhub-hub:8081
18:09:06.677 [ConfigProxy] debug: PROXY WEB /hub/ to http://datahub-qa-jupyterhub-hub:8081
18:09:06.702 [ConfigProxy] debug: Not recording activity for status 302 on /
18:09:06.703 [ConfigProxy] debug: Not recording activity for status 302 on /
18:09:06.712 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://datahub-qa-jupyterhub-hub:8081
18:09:07.308 [ConfigProxy] debug: Not recording activity for status 304 on /
18:09:07.417 [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://datahub-qa-jupyterhub-hub:8081
18:09:07.420 [ConfigProxy] debug: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://datahub-qa-jupyterhub-hub:8081
18:09:07.432 [ConfigProxy] debug: Not recording activity for status 304 on /
18:09:07.432 [ConfigProxy] debug: Not recording activity for status 304 on /
18:10:30.660 [ConfigProxy] info: 200 GET /api/routes 
...
18:39:31.830 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://datahub-qa-jupyterhub-hub:8081
18:39:31.834 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://datahub-qa-jupyterhub-hub:8081
18:39:32.862 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:32.864 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:32.941 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/n2akiti to http://datahub-qa-jupyterhub-hub:8081
18:39:33.187 [ConfigProxy] debug: PROXY WEB /hub/api/users/n2akiti/server/progress to http://datahub-qa-jupyterhub-hub:8081
18:39:33.191 [ConfigProxy] debug: PROXY WEB /hub/api/users/n2akiti/server/progress to http://datahub-qa-jupyterhub-hub:8081
18:39:39.318 [ConfigProxy] debug: PROXY WEB /hub/api/users/n2akiti/activity to http://datahub-qa-jupyterhub-hub:8081
18:39:40.178 [ConfigProxy] info: Adding route /user/n2akiti -> http://10.32.73.213:8888
18:39:40.179 [ConfigProxy] info: Route added /user/n2akiti -> http://10.32.73.213:8888
18:39:40.179 [ConfigProxy] info: 201 POST /api/routes/user/n2akiti 
18:39:40.214 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/n2akiti to http://datahub-qa-jupyterhub-hub:8081
18:39:40.218 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/n2akiti to http://datahub-qa-jupyterhub-hub:8081
18:39:40.237 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:40.238 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:40.271 [ConfigProxy] debug: PROXY WEB /user/n2akiti/ to http://10.32.73.213:8888
18:39:40.277 [ConfigProxy] debug: Not recording activity for status 302 on /user/n2akiti
18:39:40.293 [ConfigProxy] debug: PROXY WEB /user/n2akiti/tree to http://10.32.73.213:8888
18:39:40.377 [ConfigProxy] debug: Not recording activity for status 302 on /user/n2akiti
18:39:40.401 [ConfigProxy] debug: PROXY WEB /hub/api/oauth2/authorize to http://datahub-qa-jupyterhub-hub:8081
18:39:40.451 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:40.452 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:40.453 [ConfigProxy] debug: Not recording activity for status 302 on /
18:39:40.490 [ConfigProxy] debug: PROXY WEB /user/n2akiti/oauth_callback to http://10.32.73.213:8888
18:39:40.692 [ConfigProxy] debug: Not recording activity for status 302 on /user/n2akiti
18:39:40.728 [ConfigProxy] debug: PROXY WEB /user/n2akiti/tree to http://10.32.73.213:8888
18:39:41.877 [ConfigProxy] debug: PROXY WEB /user/n2akiti/custom/custom.css to http://10.32.73.213:8888
18:39:41.887 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:41.909 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/9136.bundle.js to http://10.32.73.213:8888
18:39:42.052 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/9605.bundle.js to http://10.32.73.213:8888
18:39:42.054 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/911.bundle.js to http://10.32.73.213:8888
18:39:42.995 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/7506.bundle.js to http://10.32.73.213:8888
18:39:43.297 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/5649.bundle.js to http://10.32.73.213:8888
18:39:43.299 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/2955.bundle.js to http://10.32.73.213:8888
18:39:43.524 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/notebook/5263.bundle.js to http://10.32.73.213:8888
18:39:44.427 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-plotly/static/remoteEntry.c6c768d682b3638efd6b.js to http://10.32.73.213:8888
18:39:44.429 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab_pygments/static/remoteEntry.5cbb9d2323598fbda535.js to http://10.32.73.213:8888
18:39:44.431 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-rise/static/remoteEntry.64949938345da5069400.js to http://10.32.73.213:8888
18:39:44.436 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:44.439 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:44.662 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/nbdime-jupyterlab/static/remoteEntry.370de6f77fd53f20b297.js to http://10.32.73.213:8888
18:39:44.664 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyter-matplotlib/static/remoteEntry.93cdb8c92eff90417376.js to http://10.32.73.213:8888
18:39:44.667 [ConfigProxy] debug: PROXY WEB /user/n2akiti/static/favicons/favicon.ico to http://10.32.73.213:8888
18:39:44.670 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:44.674 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:44.676 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.147 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-notebook/lab-extension/static/remoteEntry.d04be841d48ffe2a9bdd.js to http://10.32.73.213:8888
18:39:45.149 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter/nbgrader/static/remoteEntry.8752e07acaa5b8733105.js to http://10.32.73.213:8888
18:39:45.150 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-server/resource-usage/static/remoteEntry.aa2cf7c0ce7c93d19b36.js to http://10.32.73.213:8888
18:39:45.154 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.158 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.159 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.384 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/remoteEntry.9b08bf4afadafb4454e8.js to http://10.32.73.213:8888
18:39:45.386 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/fasta-extension/static/remoteEntry.b15a25cb741a6c7381f8.js to http://10.32.73.213:8888
18:39:45.388 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/geojson-extension/static/remoteEntry.6a76d3e37f02d3977b44.js to http://10.32.73.213:8888
18:39:45.391 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.395 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.397 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.429 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/latex/static/remoteEntry.6c3ab77b7da0ff4c4903.js to http://10.32.73.213:8888
18:39:45.431 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/remoteEntry.9f387e5e108e458f62c3.js to http://10.32.73.213:8888
18:39:45.433 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterhub/jupyter-server-proxy/static/remoteEntry.1d058ab1c1661989f918.js to http://10.32.73.213:8888
18:39:45.437 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.440 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.442 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.478 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab_pygments/static/747.67662283a5707eeb4d4c.js to http://10.32.73.213:8888
18:39:45.485 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.488 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js to http://10.32.73.213:8888
18:39:45.490 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-rise/static/898.9e08659f55f7a47080b1.js to http://10.32.73.213:8888
18:39:45.495 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.499 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.570 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-rise/static/776.976b2555be76525c5186.js to http://10.32.73.213:8888
18:39:45.573 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/291.cff5ef71b29e18850479.js to http://10.32.73.213:8888
18:39:45.580 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.584 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.587 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.618 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/595.998263bfa9875eeeb29b.js to http://10.32.73.213:8888
18:39:45.624 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.632 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-plotly/static/133.d03e1dd786ee2967a117.js to http://10.32.73.213:8888
18:39:45.634 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyterlab-plotly/static/423.d0d3e2912c33c7566484.js to http://10.32.73.213:8888
18:39:45.639 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.641 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.666 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/jupyter-matplotlib/static/480.18f23d468bae372d1c77.js to http://10.32.73.213:8888
18:39:45.671 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.706 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/nbdime-jupyterlab/static/426.bddb0c2da3cfe0585528.js to http://10.32.73.213:8888
18:39:45.708 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/nbdime-jupyterlab/static/324.0cfa28d7690db13a18b6.js to http://10.32.73.213:8888
18:39:45.714 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.716 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.732 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/nbdime-jupyterlab/static/971.472368f83dff20e4fce9.js to http://10.32.73.213:8888
18:39:45.738 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.784 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-notebook/lab-extension/static/568.9cd06af416befd593828.js to http://10.32.73.213:8888
18:39:45.787 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-notebook/lab-extension/static/93.eae3497dd223d842d198.js to http://10.32.73.213:8888
18:39:45.789 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/946.0b568842df5b992322e5.js to http://10.32.73.213:8888
18:39:45.793 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.798 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.829 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-server/resource-usage/static/489.99e3a74eff38179fd644.js to http://10.32.73.213:8888
18:39:45.831 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-server/resource-usage/static/311.dc54ab57a76db15e51e8.js to http://10.32.73.213:8888
18:39:45.833 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-server/resource-usage/static/643.00ea3bded09f7fdc4270.js to http://10.32.73.213:8888
18:39:45.837 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.840 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.946 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter/nbgrader/static/603.98d762b1e275ea7b932d.js to http://10.32.73.213:8888
18:39:45.948 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter/nbgrader/static/544.b87078694185d178af2e.js to http://10.32.73.213:8888
18:39:45.949 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/66.0acb8c62ba019743ba5b.js to http://10.32.73.213:8888
18:39:45.953 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.956 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:45.992 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/881.42c4dde967f7d347bc86.js to http://10.32.73.213:8888
18:39:46.004 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.009 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.065 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/338.e6e982cc684fe40f637d.js to http://10.32.73.213:8888
18:39:46.070 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.072 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.078 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/378.ba25417c17d90ae2da47.js to http://10.32.73.213:8888
18:39:46.083 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.117 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/405.a3174e55e00ed9436e42.js to http://10.32.73.213:8888
18:39:46.119 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/122.5ba163cf07ee5ab4b45f.js to http://10.32.73.213:8888
18:39:46.128 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.130 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.220 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/27.96eaab30453ba73d0911.js to http://10.32.73.213:8888
18:39:46.221 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/266.45b6171124f7c87df871.js to http://10.32.73.213:8888
18:39:46.223 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/967.2aaf8ca7c37a92081b62.js to http://10.32.73.213:8888
18:39:46.228 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.229 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.259 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/612.8bf475b8be2d5fd16884.js to http://10.32.73.213:8888
18:39:46.261 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/242.ca107b156a41aa38af3c.js to http://10.32.73.213:8888
18:39:46.262 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/728.552705d5c0c271b81970.js to http://10.32.73.213:8888
18:39:46.269 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:46.566 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/575.879317cc5dff08e50695.js to http://10.32.73.213:8888
18:39:46.606 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/git/static/747.30dabbfed4f4994e24bc.js to http://10.32.73.213:8888
18:39:46.612 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:47.503 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/fasta-extension/static/410.a7e6b5eae966dbeb7f67.js to http://10.32.73.213:8888
18:39:47.753 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/fasta-extension/static/216.4fdb3cb423aec0a0b294.js to http://10.32.73.213:8888
18:39:47.759 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:47.774 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/fasta-extension/static/568.b32ebcf7155378c65c70.js to http://10.32.73.213:8888
18:39:47.781 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.343 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/geojson-extension/static/62.5e34d4093a41ab340a24.js to http://10.32.73.213:8888
18:39:48.344 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/fasta-extension/static/643.929c653e8b3ed3e6bb69.js to http://10.32.73.213:8888
18:39:48.345 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/geojson-extension/static/969.fb561faf72b11e7b2f4e.js to http://10.32.73.213:8888
18:39:48.351 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.353 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.441 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/geojson-extension/static/643.bdb928a9116846bf6939.js to http://10.32.73.213:8888
18:39:48.443 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/latex/static/970.4573cdd728360eba7360.js to http://10.32.73.213:8888
18:39:48.445 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterlab/latex/static/631.8468ad1483e75e001a90.js to http://10.32.73.213:8888
18:39:48.448 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.490 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/113.e4cfda62b59ddbe550d3.js to http://10.32.73.213:8888
18:39:48.564 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/extensions/@jupyterhub/jupyter-server-proxy/static/961.5a30de36bec162032e99.js to http://10.32.73.213:8888
18:39:48.567 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.569 [ConfigProxy] debug: Not recording activity for status 304 on /user/n2akiti
18:39:48.692 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/kernelspecs to http://10.32.73.213:8888
18:39:48.694 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/me to http://10.32.73.213:8888
18:39:48.710 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings to http://10.32.73.213:8888
18:39:48.712 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/kernels to http://10.32.73.213:8888
18:39:48.715 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/sessions to http://10.32.73.213:8888
18:39:48.716 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/terminals to http://10.32.73.213:8888
18:39:48.740 [ConfigProxy] debug: PROXY WS /user/n2akiti/api/events/subscribe to http://10.32.73.213:8888
18:39:48.773 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/kernelspecs to http://10.32.73.213:8888
18:39:48.775 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/me to http://10.32.73.213:8888
18:39:49.247 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/translations/default to http://10.32.73.213:8888
18:39:49.248 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/metrics/v1 to http://10.32.73.213:8888
18:39:49.441 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lsp/status to http://10.32.73.213:8888
18:39:49.545 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/translations to http://10.32.73.213:8888
18:39:49.547 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/contents to http://10.32.73.213:8888
18:39:49.554 [ConfigProxy] debug: PROXY WEB /user/n2akiti/server-proxy/servers-info to http://10.32.73.213:8888
18:39:49.600 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings to http://10.32.73.213:8888
18:39:49.601 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings to http://10.32.73.213:8888
18:39:49.616 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/codemirror-extension:plugin to http://10.32.73.213:8888
18:39:49.618 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/notebook-extension:panel to http://10.32.73.213:8888
18:39:49.619 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/metrics/v1 to http://10.32.73.213:8888
18:39:49.727 [ConfigProxy] debug: PROXY WEB /user/n2akiti/api/contents to http://10.32.73.213:8888
18:39:49.926 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/tooltip-extension:notebooks to http://10.32.73.213:8888
18:39:49.928 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/tooltip-extension:consoles to http://10.32.73.213:8888
18:39:49.930 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/toc-extension:registry to http://10.32.73.213:8888
18:39:49.931 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/notebook-extension:tools to http://10.32.73.213:8888
18:39:49.934 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/notebook-extension:completer to http://10.32.73.213:8888
18:39:49.935 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/metadataform-extension:metadataforms to http://10.32.73.213:8888
18:39:49.936 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/markdownviewer-extension:plugin to http://10.32.73.213:8888
18:39:49.937 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/fileeditor-extension:completer to http://10.32.73.213:8888
18:39:49.937 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/debugger-extension:main to http://10.32.73.213:8888
18:39:49.938 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/celltags-extension:plugin to http://10.32.73.213:8888
18:39:49.939 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/settings/@jupyterlab/cell-toolbar-extension:plugin to http://10.32.73.213:8888
18:39:50.186 [ConfigProxy] debug: PROXY WEB /user/n2akiti/lab/api/themes/@jupyterlab/theme-light-extension/index.css to http://10.32.73.213:8888
18:39:56.922 [ConfigProxy] info: Removing route /user/n2akiti
18:39:56.923 [ConfigProxy] info: 204 DELETE /api/routes/user/n2akiti 
18:40:11.864 [ConfigProxy] debug: PROXY WEB /hub/api to http://datahub-qa-jupyterhub-hub:8081
18:40:11.868 [ConfigProxy] debug: PROXY WEB /hub/api to http://datahub-qa-jupyterhub-hub:8081
18:40:11.872 [ConfigProxy] debug: PROXY WEB /hub/api to http://datahub-qa-jupyterhub-hub:8081
18:40:11.907 [ConfigProxy] debug: PROXY WEB /hub/api/users/n2akiti/activity to http://datahub-qa-jupyterhub-hub:8081
18:40:11.910 [ConfigProxy] debug: PROXY WEB /hub/api/users/n2akiti/activity to http://datahub-qa-jupyterhub-hub:8081
18:40:13.943 [ConfigProxy] info: Adding route /user/n2akiti -> http://10.32.73.214:8888
18:40:13.943 [ConfigProxy] info: Route added /user/n2akiti -> http://10.32.73.214:8888
18:40:13.944 [ConfigProxy] info: 201 POST /api/routes/user/n2akiti 
18:40:23.001 [ConfigProxy] info: 200 GET /api/routes

These are the logs for the single user pod that did not launch successfully:

Defaulted container "notebook" out of: notebook, init-support (init)
Entered start.sh with args: /opt/k8s-support/bin/tini-wrapper -- /opt/k8s-support/bin/initenv-createhomedir.sh jupyterhub-singleuser --KernelRestarter.restart_limit=0 --JupyterHubSingleUser.default_url=/lab
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1374478 gid: 0
Done running hooks in: /usr/local/bin/start-notebook.d
There is no entry in /etc/passwd for our UID=1374478. Attempting to fix...
Renaming old jovyan user to nayvoj (1000:100)
Added new n2akiti user (1374478:0). Fixed UID!
WARNING: user is n2akiti but home is /home/jovyan. You must run as root to rename the home directory!
WARNING: no write access to /home/jovyan. Try starting the container with group 'users' (100), e.g. using "--group-add=users".
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1374478 gid: 0
Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
Done running hooks in: /usr/local/bin/before-notebook.d
Executing the command: /opt/k8s-support/bin/tini-wrapper -- /opt/k8s-support/bin/initenv-createhomedir.sh jupyterhub-singleuser --KernelRestarter.restart_limit=0 --JupyterHubSingleUser.default_url=/lab
[WARN  tini (2)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
[I 2025-10-07 18:40:11.095 SingleUserLabApp mixins:541] Starting jupyterhub single-user server version 4.1.5
[I 2025-10-07 18:40:11.095 SingleUserLabApp mixins:555] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.2.4
[I 2025-10-07 18:40:11.095 SingleUserLabApp mixins:555] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.14.2
[D 2025-10-07 18:40:11.101 SingleUserLabApp application:194] Searching ['/home/n2akiti/.jupyter', '/home/n2akiti/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[... repetitive config file searching removed ...]
[W 2025-10-07 18:40:11.104 SingleUserLabApp configurable:214] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2025-10-07 18:40:11.508 SingleUserLabApp manager:348] jupyter_lsp | extension was successfully linked.
[I 2025-10-07 18:40:11.509 SingleUserLabApp manager:348] jupyter_resource_usage | extension was successfully linked.
[... many similar "extension was successfully linked" messages removed ...]
[I 2025-10-07 18:40:11.705 SingleUserLabApp manager:348] notebook_shim | extension was successfully linked.
[W 2025-10-07 18:40:11.709 SingleUserLabApp serverapp:2108] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
/opt/conda/lib/python3.11/site-packages/jupyter_server/serverapp.py:2237: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/n2akiti/logout
[I 2025-10-07 18:40:11.724 SingleUserLabApp manager:368] notebook_shim | extension was successfully loaded.
[... many similar "extension was successfully loaded" messages removed ...]
[I 2025-10-07 18:40:11.848 SingleUserLabApp manager:368] notebook | extension was successfully loaded.
[I 2025-10-07 18:40:11.851 SingleUserLabApp mixins:629] Starting jupyterhub-singleuser server version 4.1.5
[W 2025-10-07 18:40:11.889 SingleUserLabApp _version:69] jupyterhub version 5.3.0 != jupyterhub-singleuser version 4.1.5. This could cause failure to authenticate and result in redirect loops!
[I 2025-10-07 18:40:11.889 SingleUserLabApp serverapp:3005] Serving notebooks from local directory: /home/n2akiti
[I 2025-10-07 18:40:11.889 SingleUserLabApp serverapp:3005] Jupyter Server 2.14.2 is running at:
[I 2025-10-07 18:40:11.889 SingleUserLabApp serverapp:3005] http://dsmlp-jupyter-n2akiti:8888/user/n2akiti/tree?token=382cf843ff19f08657e1838c0f98f6458eecc6b7123eb34f
[I 2025-10-07 18:40:11.889 SingleUserLabApp serverapp:3005]     http://127.0.0.1:8888/user/n2akiti/tree?token=382cf843ff19f08657e1838c0f98f6458eecc6b7123eb34f
[I 2025-10-07 18:40:11.889 SingleUserLabApp serverapp:3006] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2025-10-07 18:40:11.894 SingleUserLabApp serverapp:3068] 
    
    To access the server, open this file in a browser:
        file:///home/n2akiti/.local/share/jupyter/runtime/jpserver-58-open.html
    Or copy and paste one of these URLs:
        http://dsmlp-jupyter-n2akiti:8888/user/n2akiti/tree?token=382cf843ff19f08657e1838c0f98f6458eecc6b7123eb34f
        http://127.0.0.1:8888/user/n2akiti/tree?token=382cf843ff19f08657e1838c0f98f6458eecc6b7123eb34f
[I 2025-10-07 18:40:11.895 SingleUserLabApp mixins:523] Updating Hub with activity every 300 seconds
[... hundreds of repetitive debug messages checking for language servers removed ...]
[I 2025-10-07 18:40:14.051 SingleUserLabApp manager:306] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2025-10-07 18:40:13.935 SingleUserLabApp log:192] 200 GET /user/n2akiti/api (@10.41.212.76) 2.34ms
[D 2025-10-07 18:40:14.296 SingleUserLabApp serverextension:44] [lsp] The following Language Servers will be available: {}

Thank you

8

Hi, I was wondering if there was any updates regarding the issue I’m facing?