Issue:- I am trying to deploy jupyterhub with helm chart version 3.0.0 on AKS(kubernetes version 1.27.1). During the deployment I see error with pod “hook-image-awaiter”.
After checking logs it looks like pods are repeatedly trying to access the Kubernetes API server but are timing out when attempting to connect to kubernetes.default.svc.
Troubleshooting
As part of troubleshooting I found no issues with kubernetes DNS as the service kubernetes.default.svc is resolving to the internal IP of kubernetes API server.
Also I found no restrictions on the jupyterhub namespace that I created in terms of network policy.
Below are the logs I found with respect to image-awaiter pod.
Logs
PS C:\Users\test\workspace\app1-jupyterhub> kubectl logs hook-image-awaiter-6drgt --namespace jupyter
2023/12/27 14:42:22 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller
2023/12/27 14:44:36 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:44:36 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller: retrying in 5s (5 left)
2023/12/27 14:46:52 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:46:52 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller: retrying in 10s (4 left)
2023/12/27 14:49:11 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:49:11 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller: retrying in 20s (3 left)
2023/12/27 14:51:40 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:51:40 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller: retrying in 30s (2 left)
2023/12/27 14:54:20 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:54:20 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller: retrying in 30s (1 left)
2023/12/27 14:57:00 [ERR] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller request failed: Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
2023/12/27 14:57:00 GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller giving up after 6 attempt(s): Get “https://kubernetes.default.svc:443/apis/apps/v1/namespaces/jupyter/daemonsets/hook-image-puller”: dial tcp 10.223.3.1:443: connect: connection timed out
PS C:\Users\test\workspace\app1-jupyterhub>
namespace details
C:\Users\test>kubectl get all --namespace jupyter
NAME READY STATUS RESTARTS AGE
pod/hook-image-awaiter-6drgt 0/1 Error 0 18h
pod/hook-image-awaiter-fjnst 0/1 Error 0 19h
pod/hook-image-awaiter-fl6d8 0/1 Error 0 18h
pod/hook-image-awaiter-j4bpg 0/1 Error 0 18h
pod/hook-image-awaiter-k2568 0/1 Error 0 19h
pod/hook-image-awaiter-p5bsd 0/1 Error 0 17h
pod/hook-image-awaiter-tlmzb 0/1 Error 0 18h
pod/hook-image-puller-575c2 1/1 Running 0 19h
pod/hook-image-puller-8t4pq 1/1 Running 0 19h
pod/hook-image-puller-kxmmd 1/1 Running 0 19h
pod/hook-image-puller-rkhh9 1/1 Running 0 19h
pod/hook-image-puller-vchft 1/1 Running 0 19h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/hook-image-puller 5 5 5 5 5 test/application-name=app1 19h
NAME COMPLETIONS DURATION AGE
job.batch/hook-image-awaiter 0/1 19h 19h
C:\Users\test>
Requesting community to help on this issue please.
Thanks!