I had running jupter hub on microk8s for long time.
Today I updated my values with new image tag.
After helm install hub’s pod is in crash loopback state.
describe:
Name: hub-9dc74985d-qp66z
Namespace: ai-dev
Priority: 0
Node: srv-01/192.168.200.10
Start Time: Tue, 21 Sep 2021 15:40:59 +0000
Labels: app=jupyterhub
component=hub
hub.jupyter.org/network-access-proxy-api=true
hub.jupyter.org/network-access-proxy-http=true
hub.jupyter.org/network-access-singleuser=true
pod-template-hash=9dc74985d
release=jupyter-hub
Annotations: checksum/config-map: 9b9bd0f236799671513755b40004ffab7e37adde1b0bd0deb5d6dfeb008a5336
checksum/secret: e55d2708d6e3f3a5dfd1f1f43dd9ee3b65243814811df832ba785acd6584d1ca
cni.projectcalico.org/podIP: 10.1.103.209/32
cni.projectcalico.org/podIPs: 10.1.103.209/32
Status: Running
IP: 10.1.103.209
IPs:
IP: 10.1.103.209
Controlled By: ReplicaSet/hub-9dc74985d
Containers:
hub:
Container ID: containerd://84725b24103b80c85162a26d72bf978c88ff3ab9a5550dd273fc472717d06610
Image: jupyterhub/k8s-hub:1.0.1
Image ID: docker.io/jupyterhub/k8s-hub@sha256:4276da110f866d89ed946fe14399e40ce7e1bb543b27b7445edcd9093f6c9359
Port: 8081/TCP
Host Port: 0/TCP
Args:
jupyterhub
--config
/usr/local/etc/jupyterhub/jupyterhub_config.py
--upgrade-db
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 135
Started: Tue, 21 Sep 2021 15:41:22 +0000
Finished: Tue, 21 Sep 2021 15:41:22 +0000
Ready: False
Restart Count: 2
Liveness: http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30
Readiness: http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000
Environment:
PYTHONUNBUFFERED: 1
HELM_RELEASE_NAME: jupyter-hub
POD_NAMESPACE: ai-dev (v1:metadata.namespace)
CONFIGPROXY_AUTH_TOKEN: <set to the key 'hub.config.ConfigurableHTTPProxy.auth_token' in secret 'hub'> Optional: false
Mounts:
/srv/jupyterhub from pvc (rw)
/usr/local/etc/jupyterhub/config/ from config (rw)
/usr/local/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
/usr/local/etc/jupyterhub/secret/ from secret (rw)
/usr/local/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-24866 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: hub
Optional: false
secret:
Type: Secret (a volume populated by a Secret)
SecretName: hub
Optional: false
pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: hub-db-dir
ReadOnly: false
kube-api-access-24866:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=core:NoSchedule
hub.jupyter.org_dedicated=core:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32s default-scheduler Successfully assigned ai-dev/hub-9dc74985d-qp66z to srv-01
Normal Pulled 10s (x3 over 32s) kubelet Container image "jupyterhub/k8s-hub:1.0.1" already present on machine
Normal Created 10s (x3 over 31s) kubelet Created container hub
Normal Started 10s (x3 over 31s) kubelet Started container hub
Warning Unhealthy 9s kubelet Readiness probe failed: Get "http://10.1.103.209:8081/hub/health": dial tcp 10.1.103.209:8081: connect: connection refused
Warning BackOff 3s (x7 over 29s) kubelet Back-off restarting failed container
I’m wandering how to solve this issue.
Calico controller looks ok.