Jupytor Hub Giving Error Readiness probe failed: Get "http://XX.XXX.X.100:8081/hub/health": dial tcp XX.XXX.X.100:8081:

Hi Team,

I am deploying Jupytor Hub on Azure , I am getting this errror Readiness probe failed: Get “http://XX.XXX.X.100:8081/hub/health”: dial tcp XX.XXX.X.100:8081:

Hub Pod Deatils

Name:         hub-676958d57c-rw6x7
Namespace:    jupyter
Priority:     0
Node:         aks-agentpool-39040204-vmss000000/10.xx.xxx.71
Start Time:   Sun, 06 Mar 2022 20:45:33 +0530
Labels:       app=jupyterhub
              component=hub
              hub.jupyter.org/network-access-proxy-api=true
              hub.jupyter.org/network-access-proxy-http=true
              hub.jupyter.org/network-access-singleuser=true
              pod-template-hash=676958d57c
              release=jupyter
Annotations:  checksum/config-map: b98e33ead4b7888ba1a11b133969e5f897fbb18dfb573e8
              checksum/secret: 618500c9d6d58edfa92fff2d432a74d93fc5294c46cad022
Status:       Running
IP:           XX.XXX.X.100
IPs:
  IP:           XX.XXX.X.100
Controlled By:  ReplicaSet/hub-676958d57c
Containers:
  hub:
    Container ID:  containerd://f4f63e4abf11a4a1c8a8941cf42faba9e02879df4
    Image:         jupyterhub/k8s-hub:1.1.3-n326.h1619dc2d
    Image ID:      docker.io/jupyterhub/k8s-hub@sha256:750b6abf6907db82cb0da37ed45ed48674ace0e4a87f62cfe99840
    Port:          8081/TCP
    Host Port:     0/TCP
    Args:
      jupyterhub
      --config
      /usr/local/etc/jupyterhub/jupyterhub_config.py
      --upgrade-db
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 07 Mar 2022 19:05:41 +0530
      Finished:     Mon, 07 Mar 2022 19:06:17 +0530
    Ready:          False
    Restart Count:  240
    Liveness:       http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30
    Readiness:      http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000
    Environment:
      PYTHONUNBUFFERED:        1
      HELM_RELEASE_NAME:       jupyter
      POD_NAMESPACE:           jupyter (v1:metadata.namespace)
      CONFIGPROXY_AUTH_TOKEN:  <set to the key 'hub.config.ConfigurableHTTPProxy.auth_token' in secret 'hub'>  Optional: false
    Mounts:
      /srv/jupyterhub from pvc (rw)
      /usr/local/etc/jupyterhub/config/ from config (rw)
      /usr/local/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
      /usr/local/etc/jupyterhub/secret/ from secret (rw)
      /usr/local/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rw5jg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      hub
    Optional:  false
  secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hub
    Optional:    false
  pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  hub-db-dir
    ReadOnly:   false
  kube-api-access-rw5jg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 hub.jupyter.org/dedicated=core:NoSchedule
                             hub.jupyter.org_dedicated=core:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  46m (x4481 over 22h)    kubelet  Readiness probe failed: Get "http://XX.XXX.X.100:8081/hub/health": dial tcp XX.XXX.X.100:8081: connect: connection refused
  Warning  BackOff    6m14s (x5671 over 22h)  kubelet  Back-off restarting failed container
  Normal   Pulled     78s (x241 over 22h)     kubelet  Container image "jupyterhub/k8s-hub:1.1.3-n326.h1619dc2d" already present on machine

Hub Logs

[I 2022-03-07 13:35:42.081 JupyterHub app:2766] Running JupyterHub version 2.1.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.DummyAuthenticator-2.1.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Spawner: kubespawner.spawner.KubeSpawner-2.0.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.1.1
[I 2022-03-07 13:35:42.151 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-03-07 13:35:42.172 JupyterHub roles:444] Added role token to token <APIToken('6331...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2022-03-07 13:35:42.231 JupyterHub app:2835] Initialized 0 spawners in 0.002 seconds
[I 2022-03-07 13:35:42.233 JupyterHub app:3048] Not starting proxy
[W 2022-03-07 13:35:52.251 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:35:55.359 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:35:58.721 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:02.351 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:05.820 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:11.840 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:17.089 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[E 2022-03-07 13:36:17.089 JupyterHub app:3282]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 3280, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 3052, in start
        await self.proxy.get_all_routes()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 887, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 851, in api_request
        result = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Repeated api_request to proxy path "" failed.

Proxy Logs

13:21:24.540 [ConfigProxy] e[31merrore[39m: 503 POST /mifs/.;/services/LogService connect ECONNREFUSED XXX.XXX.XX.164:8081
13:21:29.547 [ConfigProxy] e[31merrore[39m: Failed to get custom error page: Error: getaddrinfo EAI_AGAIN hub
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
  errno: -3001,
  code: 'EAI_AGAIN',
  syscall: 'getaddrinfo',
  hostname: 'hub'
}

Hi! Please could you tell us

  • how you setup Kubernetes on Azure
  • show us your Z2JH configuration with secrets redacted

Hi Manics,

Kubernetes cluster has been configured by Infra team using pipelines with predefined policies .

For Z2JH Config Fil

prePuller:
hook:
enabled: false

Command which i am running helm upgrade --cleanup-on-fail jupyter jupyterhub/jupyterhub --namespace jupyter --version=1.1.3-n343.h204ec021 --values config.yaml

Do you know what these predefined policies are? Maybe they’re interfering with the Z2JH deployment?

Hi Manics,

I have checked the policies , Nothing specific in it and tried disabling the policies by using z2jh configuration as well.

hub:
networkPolicy:
enabled: false

Any Specific configuration is required for handling this or any way to check what causing this issue.

Regards,
Akash

Hi Manics,

During Analysis i have found that when pods are in different nodes then its hub is not able to connect to Proxy Pod.

In Hub i have added No_Proxy Environment variable , Still Hub is trying to connect to Proxy .

Can you please guide in case of Organization Proxy Policies, What is the ideal way to do the Configuration.

Kubernetes should transparently handle traffic across different nodes. The JupyterHub proxy is an integral part of JupyterHub, the noproxy environment variables you’re referring are typically used for directing http/https traffic through a corporate proxy, so aren’t relevant.

If you’re having trouble communicating between pods on different nodes it sounds like you’ve got a problem with your Kubernetes network configuration rather than JupyterHub.