Cannot Get Beyond Login and 503 Error

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Three times I have gone through the steps of Zero to JupyterHub with Kubernetes. Yet I cannot get any farther than the login screen followed by a 503 error.

I have gone back and forth with DigitalOcean support and they cannot detect anything amiss.

For some time I thought that perhaps it was the configuration settings — perhaps not enough storage or capacity. But D.O. support doesn’t see that as a problem.

I could supply all kinds of information — but everything is green in the kubernetes dashboard; kubectl get pod -A shows all 28 pods running; nothing is wrong in D.O. dashboard — the all three pods in the node pool are good to go, the loadbalancer appears fine, and a PVC is hooked up; the configuration file I’ve left blank, using all the default configurations.

For what it is worth, on that configuration score, this is the helm command I ran to start it all off:

helm upgrade --cleanup-on-fail \
  --install myhelmrel3 jupyterhub/jupyterhub \
  --namespace myjhubnamespace \
  --create-namespace \
  --version=1.2.0 \
  --values config.yaml

After which I get the standard success message.

I’ve run this on different browser types — Chrome, Canary Chrome, Brave, Firefox. I’ve removed the ad blockers.

Still I get:

Screen Shot 2022-06-15 at 2.40.17 PM
Screen Shot 2022-06-15 at 2.40.17 PM788×801 24 KB

followed almost invariably by either Service Unavailable or:

Screen Shot 2022-06-15 at 2.54.04 PM
Screen Shot 2022-06-15 at 2.54.04 PM1055×801 70.8 KB

I’d be grateful for any and all suggestions.

Thanks!

2

Hi! Please can you:

Thanks!

3

Thank you for responding.

The kubernetes config file reads:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: L...c...K
    server: https://3f006d0d-ad87-4f8c-82ec-a58f3ba723f9.k8s.ondigitalocean.com
  name: do-nyc3-trkostudax-jupyter-kubernetes
contexts:
- context:
    cluster: do-nyc3-trkostudax-jupyter-kubernetes
    user: do-nyc3-trkostudax-jupyter-kubernetes-admin
  name: do-nyc3-trkostudax-jupyter-kubernetes
current-context: do-nyc3-trkostudax-jupyter-kubernetes
kind: Config
preferences: {}
users:
- name: do-nyc3-trkostudax-jupyter-kubernetes-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kubernetes
      - cluster
      - kubeconfig
      - exec-credential
      - --version=v1beta1
      - --context=default
      - 3f006d0d-ad87-4f8c-82ec-a58f3ba723f9
      command: doctl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

I ran a helm upgrade command with the chart config.yaml set so:

debug:
  enabled: true

Running kubectl logs hub-8c9b54cb7-t4675 --namespace myjhubnamespace provided the following output:

[D 2022-06-16 17:10:25.353 JupyterHub application:730] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[D 2022-06-16 17:10:26.205 JupyterHub application:752] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[I 2022-06-16 17:10:26.323 JupyterHub app:2479] Running JupyterHub version 1.5.0
[I 2022-06-16 17:10:26.324 JupyterHub app:2509] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.5.0
[I 2022-06-16 17:10:26.324 JupyterHub app:2509] Using Spawner: kubespawner.spawner.KubeSpawner-1.1.0
[I 2022-06-16 17:10:26.325 JupyterHub app:2509] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.5.0
[D 2022-06-16 17:10:26.330 JupyterHub app:1721] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-06-16 17:10:26.369 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[D 2022-06-16 17:10:26.390 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[W 2022-06-16 17:10:26.403 JupyterHub app:1828] No admin users, admin interface will be unavailable.
[W 2022-06-16 17:10:26.404 JupyterHub app:1829] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2022-06-16 17:10:26.404 JupyterHub app:1858] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-06-16 17:10:26.597 JupyterHub app:2010] Purging expired APITokens
[D 2022-06-16 17:10:26.627 JupyterHub app:2010] Purging expired OAuthAccessTokens
[D 2022-06-16 17:10:26.633 JupyterHub app:2010] Purging expired OAuthCodes
[D 2022-06-16 17:10:26.670 JupyterHub app:2133] Initializing spawners
[D 2022-06-16 17:10:26.674 JupyterHub app:2266] Loaded users:
    
[I 2022-06-16 17:10:26.675 JupyterHub app:2546] Initialized 0 spawners in 0.005 seconds
[I 2022-06-16 17:10:26.693 JupyterHub app:2758] Not starting proxy
[D 2022-06-16 17:10:26.694 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:10:26.717 JupyterHub app:2794] Hub API listening on http://:8081/hub/
[I 2022-06-16 17:10:26.718 JupyterHub app:2796] Private Hub API connect url http://hub:8081/hub/
[I 2022-06-16 17:10:26.718 JupyterHub app:2809] Starting managed service cull-idle
[I 2022-06-16 17:10:26.719 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-06-16 17:10:26.723 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2022-06-16 17:10:26.736 JupyterHub spawner:1179] Polling subprocess every 30s
[D 2022-06-16 17:10:26.738 JupyterHub proxy:342] Fetching routes to check
[D 2022-06-16 17:10:26.739 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:10:26.749 JupyterHub proxy:347] Checking routes
[I 2022-06-16 17:10:26.750 JupyterHub app:2869] JupyterHub is now running at http://:8000
[D 2022-06-16 17:10:26.756 JupyterHub app:2472] It took 1.416 seconds for the Hub to start
[D 2022-06-16 17:10:27.012 JupyterHub base:283] Recording first activity for <APIToken('17c2...', service='cull-idle')>
[I 2022-06-16 17:10:27.057 JupyterHub log:189] 200 GET /hub/api/users (cull-idle@::1) 53.39ms
[D 2022-06-16 17:10:28.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.59ms
[D 2022-06-16 17:10:30.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.34ms
[D 2022-06-16 17:10:32.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 6.31ms
[D 2022-06-16 17:10:34.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.46ms
[D 2022-06-16 17:10:36.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.11ms
[D 2022-06-16 17:10:38.562 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.14ms
[D 2022-06-16 17:10:40.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.88ms
[D 2022-06-16 17:10:42.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.56ms
[D 2022-06-16 17:10:44.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.15ms
[D 2022-06-16 17:10:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.03ms
[D 2022-06-16 17:10:48.561 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.65ms
[D 2022-06-16 17:10:50.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.99ms
[D 2022-06-16 17:10:52.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.20ms
[I 2022-06-16 17:10:52.921 JupyterHub log:189] 302 GET /_ignition/execute-solution -> /hub/_ignition/execute-solution (@::ffff:10.108.0.2) 1.86ms
[W 2022-06-16 17:10:53.253 JupyterHub log:189] 404 GET /hub/_ignition/execute-solution (@::ffff:104.131.61.168) 83.09ms
[D 2022-06-16 17:10:54.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.03ms
[D 2022-06-16 17:10:56.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.99ms
[D 2022-06-16 17:10:58.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.43ms
[D 2022-06-16 17:11:00.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.60ms
[D 2022-06-16 17:11:02.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.28ms
[D 2022-06-16 17:11:04.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.94ms
[D 2022-06-16 17:11:06.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.23ms
[D 2022-06-16 17:11:08.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.32ms
[D 2022-06-16 17:11:10.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.65ms
[D 2022-06-16 17:11:12.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.06ms
[D 2022-06-16 17:11:14.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.42ms
[D 2022-06-16 17:11:16.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.90ms
[D 2022-06-16 17:11:18.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.21ms
[D 2022-06-16 17:11:20.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.32ms
[D 2022-06-16 17:11:22.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.64ms
[D 2022-06-16 17:11:24.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.25ms
[D 2022-06-16 17:11:26.557 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 17:11:26.752 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:11:26.761 JupyterHub proxy:347] Checking routes
[D 2022-06-16 17:11:28.557 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.81ms
[D 2022-06-16 17:11:30.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.18ms
[D 2022-06-16 17:11:32.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.37ms
[D 2022-06-16 17:11:34.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.40ms
[D 2022-06-16 17:11:36.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.04ms
[D 2022-06-16 17:11:38.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.98ms
[D 2022-06-16 17:11:40.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.15ms
[D 2022-06-16 17:11:42.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.33ms
[D 2022-06-16 17:11:44.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.94ms
[D 2022-06-16 17:11:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.03ms
[D 2022-06-16 17:11:48.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 17:11:50.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.25ms
[D 2022-06-16 17:11:52.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.27ms
[D 2022-06-16 17:11:54.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.04ms
[D 2022-06-16 17:11:56.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.06ms
[D 2022-06-16 17:11:58.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.31ms
[D 2022-06-16 17:12:00.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.23ms
[D 2022-06-16 17:12:02.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.47ms
[D 2022-06-16 17:12:04.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.90ms
[D 2022-06-16 17:12:06.555 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.65ms
[D 2022-06-16 17:12:08.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.14ms
[D 2022-06-16 17:12:10.555 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.99ms
[D 2022-06-16 17:12:12.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.19ms
[D 2022-06-16 17:12:14.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.67ms
[D 2022-06-16 17:12:16.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.86ms
[D 2022-06-16 17:12:18.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.67ms
[D 2022-06-16 17:12:20.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.92ms
[D 2022-06-16 17:12:22.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.36ms
[D 2022-06-16 17:12:24.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 17:12:26.557 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.41ms
[D 2022-06-16 17:12:26.753 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:12:26.765 JupyterHub proxy:347] Checking routes
[D 2022-06-16 17:12:28.559 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.69ms
[D 2022-06-16 17:12:30.563 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.86ms
[D 2022-06-16 17:12:32.562 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 17:12:34.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.26ms
[D 2022-06-16 17:12:36.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.12ms
[D 2022-06-16 17:12:38.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.51ms
[D 2022-06-16 17:12:40.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.16ms
[D 2022-06-16 17:12:42.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.21ms
[D 2022-06-16 17:12:44.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.58ms
[D 2022-06-16 17:12:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 17:12:48.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.93ms
[D 2022-06-16 17:12:50.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.69ms
[D 2022-06-16 17:12:52.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.28ms
[D 2022-06-16 17:12:54.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.00ms
[D 2022-06-16 17:12:56.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.27ms
[D 2022-06-16 17:12:58.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.73ms
[D 2022-06-16 17:13:00.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.69ms
[D 2022-06-16 17:13:02.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.91ms
[D 2022-06-16 17:13:04.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.97ms
[D 2022-06-16 17:13:06.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.78ms
[D 2022-06-16 17:13:08.549 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.96ms
[D 2022-06-16 17:13:10.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.92ms
[D 2022-06-16 17:13:12.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.09ms
[D 2022-06-16 17:13:14.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.41ms
[D 2022-06-16 17:13:16.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.92ms
[D 2022-06-16 17:13:18.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.62ms
[D 2022-06-16 17:13:20.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.23ms
[D 2022-06-16 17:13:22.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.93ms
[D 2022-06-16 17:13:24.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.01ms
[D 2022-06-16 17:13:26.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.02ms
[D 2022-06-16 17:13:26.751 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:13:26.763 JupyterHub proxy:347] Checking routes
[D 2022-06-16 17:13:28.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.87ms
[D 2022-06-16 17:13:30.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.15ms
[D 2022-06-16 17:13:32.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.37ms
[D 2022-06-16 17:13:34.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.28ms
[D 2022-06-16 17:13:36.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.87ms
[D 2022-06-16 17:13:38.561 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 10.94ms
[D 2022-06-16 17:13:40.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.17ms
[D 2022-06-16 17:13:42.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.41ms
[D 2022-06-16 17:13:44.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.38ms
[D 2022-06-16 17:13:46.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.40ms
[D 2022-06-16 17:13:48.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.10ms
[D 2022-06-16 17:13:50.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.25ms
[D 2022-06-16 17:13:52.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.33ms
[D 2022-06-16 17:13:54.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.18ms
[D 2022-06-16 17:13:56.560 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 10.70ms
[D 2022-06-16 17:13:58.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.00ms
[D 2022-06-16 17:14:00.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.67ms
[D 2022-06-16 17:14:02.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.32ms
[D 2022-06-16 17:14:04.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.47ms
[D 2022-06-16 17:14:06.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 17:14:08.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.77ms
[D 2022-06-16 17:14:10.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.23ms
[D 2022-06-16 17:14:12.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.16ms
[D 2022-06-16 17:14:14.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.91ms
[D 2022-06-16 17:14:16.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.99ms
[D 2022-06-16 17:14:18.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.16ms
[D 2022-06-16 17:14:20.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.63ms
[D 2022-06-16 17:14:22.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.10ms
[D 2022-06-16 17:14:24.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.99ms
[D 2022-06-16 17:14:26.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.67ms
[D 2022-06-16 17:14:26.751 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:14:26.766 JupyterHub proxy:347] Checking routes
[D 2022-06-16 17:14:28.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.22ms
[D 2022-06-16 17:14:30.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.12ms
[D 2022-06-16 17:14:32.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.96ms
[D 2022-06-16 17:14:34.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.85ms
[D 2022-06-16 17:14:36.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.06ms
[D 2022-06-16 17:14:38.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.34ms
[D 2022-06-16 17:14:40.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.97ms
[D 2022-06-16 17:14:42.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.25ms
[D 2022-06-16 17:14:44.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.83ms
[D 2022-06-16 17:14:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 17:14:48.555 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.06ms
[D 2022-06-16 17:14:50.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.45ms
[D 2022-06-16 17:14:52.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.27ms
[D 2022-06-16 17:14:54.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.59ms
[D 2022-06-16 17:14:56.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.87ms
[D 2022-06-16 17:14:58.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.31ms
[D 2022-06-16 17:15:00.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.98ms
[D 2022-06-16 17:15:02.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.10ms
[D 2022-06-16 17:15:04.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 7.43ms
[D 2022-06-16 17:15:06.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.22ms
[D 2022-06-16 17:15:08.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.91ms
[D 2022-06-16 17:15:10.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.15ms
[D 2022-06-16 17:15:12.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.86ms
[D 2022-06-16 17:15:14.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.36ms
[D 2022-06-16 17:15:16.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.68ms
[D 2022-06-16 17:15:18.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 17:15:20.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.28ms
[D 2022-06-16 17:15:22.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.32ms
[D 2022-06-16 17:15:24.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.97ms
[D 2022-06-16 17:15:26.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.49ms
[D 2022-06-16 17:15:26.752 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 17:15:26.765 JupyterHub proxy:347] Checking routes
[D 2022-06-16 17:15:28.539 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.81ms
[D 2022-06-16 17:15:28.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.52ms
[D 2022-06-16 17:15:30.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.57ms
[D 2022-06-16 17:15:32.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.00ms
[D 2022-06-16 17:15:34.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.81ms
[D 2022-06-16 17:15:36.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.24ms
[D 2022-06-16 17:15:38.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 10.42ms
[D 2022-06-16 17:15:38.562 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.53ms
[D 2022-06-16 17:15:40.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.08ms
[D 2022-06-16 17:15:42.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.51ms
[D 2022-06-16 17:15:44.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.53ms
[D 2022-06-16 17:15:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.98ms
[D 2022-06-16 17:15:48.537 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.97ms
[D 2022-06-16 17:15:48.558 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.43ms
[D 2022-06-16 17:15:50.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.01ms
[D 2022-06-16 17:15:52.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.68ms
[D 2022-06-16 17:15:54.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 17:15:56.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.26ms
[D 2022-06-16 17:15:58.538 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.56ms
[D 2022-06-16 17:15:58.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.43ms
[D 2022-06-16 17:16:00.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.20ms
[D 2022-06-16 17:16:02.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.30ms
[D 2022-06-16 17:16:04.558 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 7.27ms
[D 2022-06-16 17:16:06.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 17:16:08.537 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.86ms
[D 2022-06-16 17:16:08.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.34ms
[D 2022-06-16 17:16:10.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 17:16:12.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.08ms
[D 2022-06-16 17:16:14.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.74ms
[D 2022-06-16 17:16:16.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.73ms

Hate to dump all this. And if any other thing will help debug, please let me know.

What am I missing?

Thank you, again.

4

Can you share the hub logs corresponding to a login attempt?

5 
[D 2022-06-16 19:37:23.754 JupyterHub application:730] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[D 2022-06-16 19:37:24.855 JupyterHub application:752] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[I 2022-06-16 19:37:24.940 JupyterHub app:2479] Running JupyterHub version 1.5.0
[I 2022-06-16 19:37:24.940 JupyterHub app:2509] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.5.0
[I 2022-06-16 19:37:24.940 JupyterHub app:2509] Using Spawner: kubespawner.spawner.KubeSpawner-1.1.0
[I 2022-06-16 19:37:24.941 JupyterHub app:2509] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.5.0
[D 2022-06-16 19:37:24.943 JupyterHub app:1721] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-06-16 19:37:24.972 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[D 2022-06-16 19:37:24.988 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[W 2022-06-16 19:37:24.998 JupyterHub app:1828] No admin users, admin interface will be unavailable.
[W 2022-06-16 19:37:24.999 JupyterHub app:1829] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2022-06-16 19:37:24.999 JupyterHub app:1858] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-06-16 19:37:25.131 JupyterHub app:2010] Purging expired APITokens
[D 2022-06-16 19:37:25.139 JupyterHub app:2010] Purging expired OAuthAccessTokens
[D 2022-06-16 19:37:25.148 JupyterHub app:2010] Purging expired OAuthCodes
[D 2022-06-16 19:37:25.175 JupyterHub app:2133] Initializing spawners
[D 2022-06-16 19:37:25.179 JupyterHub app:2266] Loaded users:
    
[I 2022-06-16 19:37:25.185 JupyterHub app:2546] Initialized 0 spawners in 0.010 seconds
[I 2022-06-16 19:37:25.191 JupyterHub app:2758] Not starting proxy
[D 2022-06-16 19:37:25.192 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 19:37:25.207 JupyterHub app:2794] Hub API listening on http://:8081/hub/
[I 2022-06-16 19:37:25.208 JupyterHub app:2796] Private Hub API connect url http://hub:8081/hub/
[I 2022-06-16 19:37:25.208 JupyterHub app:2809] Starting managed service cull-idle
[I 2022-06-16 19:37:25.209 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-06-16 19:37:25.214 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2022-06-16 19:37:25.242 JupyterHub spawner:1179] Polling subprocess every 30s
[D 2022-06-16 19:37:25.243 JupyterHub proxy:342] Fetching routes to check
[D 2022-06-16 19:37:25.244 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 19:37:25.250 JupyterHub proxy:347] Checking routes
[I 2022-06-16 19:37:25.251 JupyterHub app:2869] JupyterHub is now running at http://:8000
[D 2022-06-16 19:37:25.254 JupyterHub app:2472] It took 1.514 seconds for the Hub to start
[D 2022-06-16 19:37:25.509 JupyterHub base:283] Recording first activity for <APIToken('57e1...', service='cull-idle')>
[I 2022-06-16 19:37:25.586 JupyterHub log:189] 200 GET /hub/api/users (cull-idle@::1) 87.34ms
[D 2022-06-16 19:37:26.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.37ms
[D 2022-06-16 19:37:28.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.91ms
[D 2022-06-16 19:37:30.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.08ms
[D 2022-06-16 19:37:32.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.07ms
[D 2022-06-16 19:37:34.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.83ms
[D 2022-06-16 19:37:36.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 3.04ms
[D 2022-06-16 19:37:38.555 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.35ms
[D 2022-06-16 19:37:40.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 19:37:42.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 19:37:44.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 19:37:46.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.59ms
[D 2022-06-16 19:37:48.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.14ms
[D 2022-06-16 19:37:50.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.09ms
[D 2022-06-16 19:37:52.555 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.25ms
[D 2022-06-16 19:37:54.556 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 4.52ms
[D 2022-06-16 19:37:56.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 19:37:58.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.28ms
[D 2022-06-16 19:38:00.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.88ms
[D 2022-06-16 19:38:01.366 JupyterHub pages:652] No template for 503
[I 2022-06-16 19:38:01.519 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fhub%2F (@10.244.0.81) 154.12ms
[D 2022-06-16 19:38:01.855 JupyterHub log:189] 200 GET /hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@::ffff:10.108.0.3) 7.90ms
[D 2022-06-16 19:38:01.948 JupyterHub log:189] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@::ffff:10.108.0.2) 10.98ms
[D 2022-06-16 19:38:01.955 JupyterHub log:189] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=f3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@::ffff:10.108.0.2) 5.17ms
[D 2022-06-16 19:38:02.078 JupyterHub log:189] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@::ffff:10.108.0.2) 4.11ms
[D 2022-06-16 19:38:02.235 JupyterHub log:189] 200 GET /hub/logo (@::ffff:10.108.0.2) 4.76ms
[D 2022-06-16 19:38:02.371 JupyterHub log:189] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:10.108.0.2) 2.09ms
[D 2022-06-16 19:38:02.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.91ms
[D 2022-06-16 19:38:04.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.07ms
[D 2022-06-16 19:38:06.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.44ms
[D 2022-06-16 19:38:08.558 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.30ms
[D 2022-06-16 19:38:10.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.68ms
[D 2022-06-16 19:38:12.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.02ms
[D 2022-06-16 19:38:14.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.67ms
[D 2022-06-16 19:38:16.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.50ms
[D 2022-06-16 19:38:18.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.13ms
[D 2022-06-16 19:38:20.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 19:38:22.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.16ms
[D 2022-06-16 19:38:24.554 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.54ms
[D 2022-06-16 19:38:25.252 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[I 2022-06-16 19:38:25.261 JupyterHub proxy:347] Checking routes
[D 2022-06-16 19:38:26.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.18ms
[D 2022-06-16 19:38:28.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.46ms
[D 2022-06-16 19:38:30.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.68ms
[D 2022-06-16 19:38:32.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.05ms
[D 2022-06-16 19:38:34.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.11ms
[D 2022-06-16 19:38:36.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.33ms
[D 2022-06-16 19:38:38.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.94ms
[D 2022-06-16 19:38:40.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.32ms
[D 2022-06-16 19:38:42.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.11ms
[D 2022-06-16 19:38:44.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.00ms
[D 2022-06-16 19:38:46.557 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 5.94ms
[D 2022-06-16 19:38:48.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.20ms
[D 2022-06-16 19:38:50.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.45ms
[D 2022-06-16 19:38:52.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.27ms
[D 2022-06-16 19:38:54.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 2.29ms
[D 2022-06-16 19:38:56.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.89ms
[D 2022-06-16 19:38:58.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.01ms
[D 2022-06-16 19:39:00.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 0.71ms
[D 2022-06-16 19:39:02.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.44ms
[D 2022-06-16 19:39:04.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.50ms
[D 2022-06-16 19:39:06.550 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.11ms
[D 2022-06-16 19:39:08.557 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.75ms
[D 2022-06-16 19:39:10.552 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.29ms
[D 2022-06-16 19:39:12.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.14ms
[D 2022-06-16 19:39:14.551 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.60ms
[D 2022-06-16 19:39:16.553 JupyterHub log:189] 200 GET /hub/health (@10.244.0.48) 1.89ms
[D 2022-06-16 19:39:18.349 JupyterHub base:328] Refreshing auth for tim
[D 2022-06-16 19:39:18.350 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for tim:
[D 2022-06-16 19:39:18.364 JupyterHub spawner:194] Starting executor thread pool with 5 workers
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f84ce549e20>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f84ce41d040>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f84ce41d190>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
[E 2022-06-16 19:43:18.617 JupyterHub web:1789] Uncaught exception GET /hub/home (::ffff:104.131.61.168)
    HTTPServerRequest(protocol='http', host='159.203.149.218', method='GET', uri='/hub/home', version='HTTP/1.1', remote_ip='::ffff:104.131.61.168')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py", line 84, in get
        default_server=user.spawner,
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 368, in spawner
        return self.spawners['']
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 186, in __getitem__
        self[key] = self.spawner_factory(key)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 361, in _new_spawner
        spawner = spawner_class(**spawn_kwargs)
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 209, in __init__
        self._start_watching_pods()
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2273, in _start_watching_pods
        return self._start_reflector(
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2231, in _start_reflector
        self.__class__.reflectors[key] = ReflectorClass(
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py", line 206, in __init__
        self.start()
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py", line 378, in start
        self._list_and_update()
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py", line 227, in _list_and_update
        initial_resources = getattr(self.api, self.list_method_name)(**kwargs)
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 15302, in list_namespaced_pod
        return self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 15413, in list_namespaced_pod_with_http_info
        return self.api_client.call_api(
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 348, in call_api
        return self.__call_api(resource_path, method,
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api
        response_data = self.request(
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 373, in request
        return self.rest_client.GET(url,
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 239, in GET
        return self.request("GET", url,
      File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 212, in request
        r = self.pool_manager.request(method, url,
      File "/usr/local/lib/python3.8/dist-packages/urllib3/request.py", line 74, in request
        return self.request_encode_url(
      File "/usr/local/lib/python3.8/dist-packages/urllib3/request.py", line 96, in request_encode_url
        return self.urlopen(method, url, **extra_kw)
      File "/usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py", line 375, in urlopen
        response = conn.urlopen(method, u.request_uri, **kw)
      File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 783, in urlopen
        return self.urlopen(
      File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 783, in urlopen
        return self.urlopen(
      File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 783, in urlopen
        return self.urlopen(
      File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
        retries = retries.increment(
      File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 574, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.245.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f84ce41d280>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)'))
    
[D 2022-06-16 19:43:18.643 JupyterHub base:1285] No template for 500
[D 2022-06-16 19:43:18.708 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for tim:

In the Kubernetes dashboard, the events under the hub appeared so:

Screen Shot 2022-06-16 at 2.50.30 PM
Screen Shot 2022-06-16 at 2.50.30 PM1625×255 85 KB

6

It looks like JupyterHub is having trouble talking to the Kubernetes API.

To help debug this could you:

  • Try the latest dev version of Z2JH, currently 1.1.3-n636.h4694cca8 (note this will install JupyterHub 2)
  • Disable all network policies, especially hub.networkPolicy.enabled and singleuser.networkPolicy.enabled,
7

cry:Thank you so much for your help. But I’m afraid that didn’t work.

I followed — as best as I could figure – the steps you recommended.

I switched to the most recent development version of JupyterHub (version=1.1.3-n636.h4694cca8) and instituted this in the config.yaml:

hub:
  networkPolicy:
    enabled: false

singleuser:
  networkPolicy:
    enabled: false

That still didn’t work. So I torn down and rebuilt again. Then, within that rebuild, I increased the pool size. Nothing.

While I worked on the initial recommendations, I received this from DigitalOcean support personnel:

DigitalOcean Support Message

I [i.e., the tech at D.O.] have retried to access the Jupyter service by accessing the following URL on the browser “159.203.149.218” [my comment: now torn down] and able to see the login screen. After this, I tried to login with “hello” user where I was prompted for 503 error message as you have described earlier. During this process, I observed that “hub-8c9b54cb7-t4675” pod restarted due to OOM error events (137 error code). You can observe that by reviewing the “137” event error message for the last terminated state.

# k describe pods -n myjhubnamespace hub-8c9b54cb7-t4675

Name:         hub-8c9b54cb7-t4675
Namespace:    myjhubnamespace
Priority:     0
Node:         worker-pool-cpelw/10.108.0.4
....
....
    State:          Running
      Started:      Fri, 17 Jun 2022 16:07:02 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 17 Jun 2022 15:45:52 +0000
      Finished:     Fri, 17 Jun 2022 16:07:01 +0000

Moreover, the login requests are being timed out from the backend application when the login request is made which looks to be the application issue. If the requests were being blocked on the [LoadBalancer], the “hello” user login request log entry would never have been shown. Please refer to the following logs:


[D 2022-06-17 15:42:40.822 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for hello:
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f4b657f6be0>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f4b65c328b0>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f4b65c32a00>, 'Connection to 10.245.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/myjhubnamespace/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
[E 220617 15:45:46 ioloop:761] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fc8c4dc84f0>>, <Future finished exception=HTTP 599: Operation timed out after 20001 milliseconds with 0 bytes received>)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 741, in _run_callback
        ret = callback()
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 765, in _discard_future_result
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 769, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 120, in cull_idle
        resp = yield fetch(req)
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
        value = future.result()
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 775, in run
        yielded = self.gen.send(value)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 113, in fetch
        return (yield client.fetch(req))
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
        value = future.result()
    tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20001 milliseconds with 0 bytes received

From reviewing these logs, the requests are not being accepted from the end application.

I would suggest raising this issue with the application provider in this forum: Issues · jupyterhub/zero-to-jupyterhub-k8s · GitHub

Although there are ample memory resources available on the worker node where the hub-8c9b54cb7-t4675 pod is scheduled, it is continuously being restarted due to OOM errors. You could try spinning up a larger node to test whether the issue gets resolved. [See my earlier comments on how I did that.] However, it looks to be memory leak issue on the application end. If the issue does not get resolved after increasing , please raise this issue with the relevant application provider as the error logs are showing that backend application is not accepting any connections resulting in request timeout.

I am sorry for this long post, but I hesitate to raise a bug issue on the repository given my lack of experience with JupyterHub deployment and Kubernetes. But I thought I should pass this along.

Thanks again. I will keep hacking away. But any thoughts, even after thoughts, would be appreciated.

Post Script: If nothing else, I’ve learned a ton about Kubernetes. Nothing like endless frustration to feed the mind — if a bit harmful to the soul. :sob:

8

Hi. The out-of-memory error sounds like a possible bug, however I don’t think it’s the root cause of your problems.

suggests the hub is unable to make an API request to Kubernetes.

I’d focus on this- try and find out what’s prevent the hub from connecting to the K8S API. For example, does Digital Ocean have some additional security features to limit access?

9

Thank you for taking a look and wading through this one.

In the interim, I continued to try to debug the DigitalOcean deployment, but nothing worked.

Having exhausted that, I turned to Google. After a lot of billing, IAM, and other configuration details (some of which I may make a pull request to detail), I was finally able to deploy this morning.

But working through Google Cloud’s IAM configurations, and having then seen your message, I realized DO had no such hurdle to implementing their version of managed Kubernetes. I assume that’s the security features you were considering. I went back to DO and searched through its docs — but I can find nothing on IAM — not in docs, not in the API. They do have authentication details, but not IAM.

Whereas gcloud commands threw errors without such configuration, I never experienced that on DO.

Anyway, thank you for your help. But for now I’m using Google. I’ll be sure to return here for help — an inevitability.

10

I had the same issue. The hub could not connect to the Kubernetes API. Maybe it’s because we use Cilium as a kube-proxy replacement, I don’t know.

I got it fixed by turning off the network policy:

hub:
  networkPolicy:
    enabled: false
11

Thank you. I’ve had to set this aside for a while — schools opens soon and other matters imposed themselves. But I will soon return to it and take your suggestion into account.