Hi everyone,
First, thanks to all of you who have put so much time and effort into this project - amazing stuff!
I am pretty new to Kubernetes, Docker, etc and I have been following the Zero2Jupyterhub instructions for setting up a Jupyterhub server on Kubenetes running on Google Kubernetes Engine. I am able to get it running with the defaults in the tutorial, but when I attempt to pull an image from my own (public) docker hub repository it fails. I’ve done quite a bit of work to isolate the problem but I’m currently stuck.
Currently I’m using the following command to set it up on a GKE cluster
helm upgrade --cleanup-on-fail \
--install jhub jupyterhub/jupyterhub \
--namespace jhub \
--create-namespace \
--values config.yaml
My config.yaml file is
singleuser:
image:
name: mydockerrepository/scipy-notebook
tag: latest
As far as I know, mydockerrepository/scipy-notebook is identical to jupyter/scipy-notebook. In fact, all I did was docker pull the image from jupyter, retag it, and upload it to my docker hub. Everything works when I use name: jupyter/scipy-notebook but when I use the image from repository I get this error (–debug flag on)
client.go:491: [debug] Watching for changes to Job hook-image-awaiter with timeout of 5m0s
client.go:519: [debug] Add/Modify event for hook-image-awaiter: ADDED
client.go:558: [debug] hook-image-awaiter: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:558: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: failed pre-install: timed out waiting for the condition
helm.go:81: [debug] failed pre-install: timed out waiting for the condition
kubectl describe pod shows two hook-image-puller-xxxxx pods that have thrown errors. The rest of the pods look ok.
Namespace: jhub
Priority: 0
Node: gke-daskhub-default-pool-f768713a-8bp7/10.128.0.25
Start Time: Thu, 09 Dec 2021 20:02:41 +0000
Labels: app=jupyterhub
component=hook-image-puller
controller-revision-hash=79c65dcb8
pod-template-generation=1
release=jhub
Annotations: <none>
Status: Pending
IP: 10.44.1.18
IPs:
IP: 10.44.1.18
Controlled By: DaemonSet/hook-image-puller
Init Containers:
image-pull-metadata-block:
Container ID: containerd://a025259a2788fd56562ca711bce2cf74c30c4b64b8447dcd312853e961ee6b7d
Image: jupyterhub/k8s-network-tools:1.2.0
Image ID: docker.io/jupyterhub/k8s-network-tools@sha256:a6fa68b84748dcf01085016fd2475e84a38d4b5f0940d010c0ae3044e50ee28d
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
echo "Pulling complete"
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 09 Dec 2021 20:02:42 +0000
Finished: Thu, 09 Dec 2021 20:02:42 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts: <none>
image-pull-singleuser:
Container ID: containerd://2928fbe5caf23ce52d3242d060b5f5849da4fd15ba8434e3f623f285c1380f94
Image: mydockerhub/scipy-notebook:latest
Image ID: docker.io/mydockerhub/scipy-notebook@sha256:4c8fb9935617b31e085c17ba8cfa9cae8259a13c259eac72bc0ba061eef5876d
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
echo "Pulling complete"
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 09 Dec 2021 20:08:38 +0000
Finished: Thu, 09 Dec 2021 20:08:38 +0000
Ready: False
Restart Count: 6
Environment: <none>
Mounts: <none>
Containers:
pause:
Container ID:
Image: k8s.gcr.io/pause:3.5
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes: <none>
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=user:NoSchedule
hub.jupyter.org_dedicated=user:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m11s default-scheduler Successfully assigned jhub/hook-image-puller-fl5hl to gke-daskhub-default-pool-f768713a-8bp7
Normal Pulled 9m11s kubelet Container image "jupyterhub/k8s-network-tools:1.2.0" already present on machine
Normal Created 9m11s kubelet Created container image-pull-metadata-block
Normal Started 9m11s kubelet Started container image-pull-metadata-block
Normal Pulled 9m10s kubelet Successfully pulled image "mydockerhub/scipy-notebook:latest" in 484.14224ms
Normal Pulled 9m9s kubelet Successfully pulled image "mydockerhub/scipy-notebook:latest" in 497.308531ms
Normal Pulled 8m54s kubelet Successfully pulled image "mydockerhub/scipy-notebook:latest" in 496.67836ms
Normal Pulling 8m29s (x4 over 9m10s) kubelet Pulling image "mydockerhub/scipy-notebook:latest"
Normal Created 8m28s (x4 over 9m10s) kubelet Created container image-pull-singleuser
Normal Started 8m28s (x4 over 9m10s) kubelet Started container image-pull-singleuser
Normal Pulled 8m28s kubelet Successfully pulled image "ngroebner/scipy-notebook:latest" in 554.499501ms
Warning BackOff 4m10s (x25 over 9m8s) kubelet Back-off restarting failed container
As far as I can tell, the only thing that is different is that the image is hosted on my docker hub. The image is public and I am able to docker pull this image without problem.
Any ideas?
Thanks!