Getting 404 Not found error while trying to deploy jupyterhub in kubernetes

JupyterHub
1

Hi, I am naive to jupyterhub and try to configure jupyterhub in gke using github actions pipeline. I can see that hub and proxy pods are creating successfully. Also, I have created a nginx ingress controller but not sure if I configured it correctly. Is there any to validate this. Please find below ingress details -

kubectl describe ingress jupyterhub
Name:             jupyterhub
Labels:           app=jupyterhub
                  app.kubernetes.io/instance=jupyterhub
                  chart=jupyterhub-2.0.0
                  component=ingress
                  heritage=Helm
                  release=jupyterhub
Namespace:        default
Address:
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes test.somedomain.net
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  test.somedomain.net
                            /nb/?(*)   proxy-public:http (10.192.2.114:8000)
Annotations:                cert-manager.io/cluster-issuer: letsencrypt-staging
                            kubernetes.io/ingress.class: nginx
Events:
  Type     Reason     Age                  From                       Message
  ----     ------     ----                 ----                       -------
  Warning  BadConfig  4m42s (x5 over 19m)  cert-manager-ingress-shim  Skipped a TLS block: spec.tls[0].secretName: Required value

hub pod logs can be visible here.

╰─ kubectl logs hub-5fbfd59f68-c47w8

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2022-11-26 14:38:49.554 JupyterHub app:2775] Running JupyterHub version 3.0.0
[I 2022-11-26 14:38:49.555 JupyterHub app:2805] Using Authenticator: jupyterhub.auth.DummyAuthenticator-3.0.0
[I 2022-11-26 14:38:49.555 JupyterHub app:2805] Using Spawner: kubespawner.spawner.KubeSpawner-4.2.0
[I 2022-11-26 14:38:49.555 JupyterHub app:2805] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.0.0
[I 2022-11-26 14:38:49.649 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-11-26 14:38:49.759 JupyterHub app:2844] Initialized 0 spawners in 0.002 seconds
[I 2022-11-26 14:38:49.761 JupyterHub app:3057] Not starting proxy
[W 2022-11-26 14:38:49.763 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[W 2022-11-26 14:38:49.911 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[W 2022-11-26 14:38:50.126 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[W 2022-11-26 14:38:50.334 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[I 2022-11-26 14:38:51.502 JupyterHub app:3093] Hub API listening on http://:8081/hub/
[I 2022-11-26 14:38:51.502 JupyterHub app:3095] Private Hub API connect url http://hub:8081/hub/
[I 2022-11-26 14:38:51.502 JupyterHub app:3104] Starting managed service jupyterhub-idle-culler
[I 2022-11-26 14:38:51.502 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-11-26 14:38:51.505 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2022-11-26 14:38:51.513 JupyterHub proxy:480] Adding route for Hub: / => http://hub:8081
[I 2022-11-26 14:38:51.515 JupyterHub app:3162] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[I 2022-11-26 14:38:51.633 JupyterHub log:186] 200 GET /hub/api/ (jupyterhub-idle-culler@127.0.0.1) 12.44ms
[I 2022-11-26 14:38:51.643 JupyterHub log:186] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 8.86ms
[I 2022-11-26 14:41:17.546 JupyterHub log:186] 302 GET / -> /hub/ (@::ffff:10.10.0.12) 0.76ms
[I 2022-11-26 14:41:17.868 JupyterHub log:186] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:10.192.2.1) 0.75ms
[I 2022-11-26 14:41:18.432 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:10.192.2.1) 27.59ms
[I 2022-11-26 14:43:17.829 JupyterHub log:186] 302 GET / -> /hub/ (@::ffff:10.10.0.14) 0.73ms
[I 2022-11-26 14:47:27.465 JupyterHub log:186] 302 GET / -> /hub/ (@::ffff:10.10.0.12) 0.71ms
[I 2022-11-26 14:47:27.916 JupyterHub log:186] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:10.192.2.1) 0.72ms
[I 2022-11-26 14:48:51.758 JupyterHub log:186] 200 GET /hub/api/ (jupyterhub-idle-culler@127.0.0.1) 46.56ms
[I 2022-11-26 14:48:51.764 JupyterHub log:186] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 4.33ms

I can see from log that hub is trying to access from default url -

JupyterHub app:3093] Hub API listening on http://:8081/hub/
JupyterHub app:3095] Private Hub API connect url http://hub:8081/hub/

Where as I am passing this value in confi.yaml file but sure why its not working

hub:      
  baseUrl: /nb

Any lead to resolve this.

2

Hi! Could you show the full command you used to deploy JupyterHub, along with your full configuration (redact any secret parameters).

Based on that output you’ve added some customisations, and there are some warnings. Have you looked into those?

As a general rule it’s best to get Z2JH working with a minimal config, then add on your customisations.

3

Thanks for the response.

that warning is for tls as I am not using tls at this moment.