Hi Team,
I am deploying Jupytor Hub on Azure , I am getting this errror Readiness probe failed: Get “http://XX.XXX.X.100:8081/hub/health”: dial tcp XX.XXX.X.100:8081:
Hub Pod Deatils
Name: hub-676958d57c-rw6x7
Namespace: jupyter
Priority: 0
Node: aks-agentpool-39040204-vmss000000/10.xx.xxx.71
Start Time: Sun, 06 Mar 2022 20:45:33 +0530
Labels: app=jupyterhub
component=hub
hub.jupyter.org/network-access-proxy-api=true
hub.jupyter.org/network-access-proxy-http=true
hub.jupyter.org/network-access-singleuser=true
pod-template-hash=676958d57c
release=jupyter
Annotations: checksum/config-map: b98e33ead4b7888ba1a11b133969e5f897fbb18dfb573e8
checksum/secret: 618500c9d6d58edfa92fff2d432a74d93fc5294c46cad022
Status: Running
IP: XX.XXX.X.100
IPs:
IP: XX.XXX.X.100
Controlled By: ReplicaSet/hub-676958d57c
Containers:
hub:
Container ID: containerd://f4f63e4abf11a4a1c8a8941cf42faba9e02879df4
Image: jupyterhub/k8s-hub:1.1.3-n326.h1619dc2d
Image ID: docker.io/jupyterhub/k8s-hub@sha256:750b6abf6907db82cb0da37ed45ed48674ace0e4a87f62cfe99840
Port: 8081/TCP
Host Port: 0/TCP
Args:
jupyterhub
--config
/usr/local/etc/jupyterhub/jupyterhub_config.py
--upgrade-db
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 07 Mar 2022 19:05:41 +0530
Finished: Mon, 07 Mar 2022 19:06:17 +0530
Ready: False
Restart Count: 240
Liveness: http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30
Readiness: http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000
Environment:
PYTHONUNBUFFERED: 1
HELM_RELEASE_NAME: jupyter
POD_NAMESPACE: jupyter (v1:metadata.namespace)
CONFIGPROXY_AUTH_TOKEN: <set to the key 'hub.config.ConfigurableHTTPProxy.auth_token' in secret 'hub'> Optional: false
Mounts:
/srv/jupyterhub from pvc (rw)
/usr/local/etc/jupyterhub/config/ from config (rw)
/usr/local/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
/usr/local/etc/jupyterhub/secret/ from secret (rw)
/usr/local/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rw5jg (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: hub
Optional: false
secret:
Type: Secret (a volume populated by a Secret)
SecretName: hub
Optional: false
pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: hub-db-dir
ReadOnly: false
kube-api-access-rw5jg:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=core:NoSchedule
hub.jupyter.org_dedicated=core:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 46m (x4481 over 22h) kubelet Readiness probe failed: Get "http://XX.XXX.X.100:8081/hub/health": dial tcp XX.XXX.X.100:8081: connect: connection refused
Warning BackOff 6m14s (x5671 over 22h) kubelet Back-off restarting failed container
Normal Pulled 78s (x241 over 22h) kubelet Container image "jupyterhub/k8s-hub:1.1.3-n326.h1619dc2d" already present on machine
Hub Logs
[I 2022-03-07 13:35:42.081 JupyterHub app:2766] Running JupyterHub version 2.1.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.DummyAuthenticator-2.1.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Spawner: kubespawner.spawner.KubeSpawner-2.0.1
[I 2022-03-07 13:35:42.081 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.1.1
[I 2022-03-07 13:35:42.151 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-03-07 13:35:42.172 JupyterHub roles:444] Added role token to token <APIToken('6331...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2022-03-07 13:35:42.231 JupyterHub app:2835] Initialized 0 spawners in 0.002 seconds
[I 2022-03-07 13:35:42.233 JupyterHub app:3048] Not starting proxy
[W 2022-03-07 13:35:52.251 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:35:55.359 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:35:58.721 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:02.351 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:05.820 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:11.840 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[W 2022-03-07 13:36:17.089 JupyterHub proxy:840] api_request to the proxy failed with status code 599, retrying...
[E 2022-03-07 13:36:17.089 JupyterHub app:3282]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 3280, in launch_instance_async
await self.start()
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 3052, in start
await self.proxy.get_all_routes()
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 887, in get_all_routes
resp = await self.api_request('', client=client)
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 851, in api_request
result = await exponential_backoff(
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Repeated api_request to proxy path "" failed.
Proxy Logs
13:21:24.540 [ConfigProxy] e[31merrore[39m: 503 POST /mifs/.;/services/LogService connect ECONNREFUSED XXX.XXX.XX.164:8081
13:21:29.547 [ConfigProxy] e[31merrore[39m: Failed to get custom error page: Error: getaddrinfo EAI_AGAIN hub
at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
errno: -3001,
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: 'hub'
}