Kubernetes- api_request to the proxy failed with status code 599, retrying (JupyterHub)

JupyterHub
,
1

Hi,

I have a k8s cluster created with kubeadm tool. I am trying to deploy daskhub with Helm Chart and it includes jupyterhub also. “hub” pod is crashing continuously. When I check the hub logs I got the error like below.

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 00-add-dask-gateway-values
Setting DASK_GATEWAY__ADDRESS http://proxy-public/services/dask-gateway
Adding dask-gateway service URL
[I 2021-11-22 10:49:59.230 JupyterHub app:2459] Running JupyterHub version 1.4.2
[I 2021-11-22 10:49:59.230 JupyterHub app:2489] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.4.2
[I 2021-11-22 10:49:59.230 JupyterHub app:2489] Using Spawner: kubespawner.spawner.KubeSpawner-1.1.0
[I 2021-11-22 10:49:59.230 JupyterHub app:2489] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.2
[I 2021-11-22 10:49:59.243 alembic.runtime.migration migration:164] Context impl SQLiteImpl.
[I 2021-11-22 10:49:59.243 alembic.runtime.migration migration:167] Will assume non-transactional DDL.
[I 2021-11-22 10:49:59.251 alembic.runtime.migration migration:556] Running stamp_revision  -> 4dc2d5a8c53c
[I 2021-11-22 10:49:59.261 alembic.runtime.migration migration:164] Context impl SQLiteImpl.
[I 2021-11-22 10:49:59.261 alembic.runtime.migration migration:167] Will assume non-transactional DDL.
[W 2021-11-22 10:49:59.302 JupyterHub app:1808] No admin users, admin interface will be unavailable.
[W 2021-11-22 10:49:59.302 JupyterHub app:1809] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-11-22 10:49:59.302 JupyterHub app:1838] Not using allowed_users. Any authenticated user will be allowed.
[I 2021-11-22 10:49:59.341 JupyterHub provider:574] Creating oauth client service-dask-gateway
[I 2021-11-22 10:49:59.351 JupyterHub app:1961] Adding API token for service: dask-gateway
[I 2021-11-22 10:49:59.372 JupyterHub app:2526] Initialized 0 spawners in 0.004 seconds
[I 2021-11-22 10:49:59.373 JupyterHub app:2738] Not starting proxy
[W 2021-11-22 10:50:19.397 JupyterHub proxy:851] api_request to the proxy failed with status code 599, retrying...
[W 2021-11-22 10:50:39.421 JupyterHub proxy:851] api_request to the proxy failed with status code 599, retrying...
[E 2021-11-22 10:50:39.421 JupyterHub app:2969]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2967, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2742, in start
        await self.proxy.get_all_routes()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 898, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 862, in api_request
        result = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 184, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Repeated api_request to proxy path "" failed.

And also when I describe “hub” pod I see the log like below.

 Warning  BackOff  3m16s (x304 over 77m)  kubelet  Back-off restarting failed container

All pods listed in the below.

image
image806×142 6.7 KB

I also use weave-net network plugin as a CNI.

I am stuck with this since past 2 days and its just not coming up.

2

Have you tried setting up JupyterHub on it’s own following Zero to JupyterHub with Kubernetes — Zero to JupyterHub with Kubernetes documentation ?

There are several interacting components, so it’ll be easier to work through things one by one.

If you still have problems please:

  • describe your k8s cluster, e.g. how was it setup
  • show us your full Z2JH config with secrets redacted
  • tell us any other information about how you’ve customised your deployment

Thanks!

3

Hi,

Firstly thank you for your response @manics. My problem still going on. You can find information about cluster and z2jh config in the below.

→ k8s cluster created by kubeadm tool. It consist 3 node, one for master and other 2 works as worker-node. Used OS is Centos8.Used network plugin is weave-net. And kube-version is v1.21.5. For the providing LoadBalancer usage I use MetalLB.

→ Z2JH config is below

Name:         hub
Namespace:    dhub
Labels:       app=jupyterhub
              app.kubernetes.io/managed-by=Helm
              chart=jupyterhub-1.1.3
              component=hub
              heritage=Helm
              release=dhub
Annotations:  meta.helm.sh/release-name: dhub
              meta.helm.sh/release-namespace: dhub

Type:  Opaque

Data
====
hub.config.ConfigurableHTTPProxy.auth_token:  64 bytes
hub.config.CryptKeeper.keys:                  64 bytes
hub.config.JupyterHub.cookie_secret:          64 bytes
hub.services.dask-gateway.apiToken:           64 bytes
values.yaml:                                  11440 bytes

→ I just update values.yaml for disable storage as like below.

singleuser.storage.type : none
4

Hi,

I used to weave-net as a network plugin in k8s. I reset the cluster and install calico CNI plugin then hub start to run in k8s.