Jupyterhub k8s failing | Error: ImagePullBackOff

JupyterHub Zero to JupyterHub on Kubernetes
, , ,
1

Hello Folks,

We were able to deploy Jupyterhub on our private shared cluster in GCP using default chart zero-to-jupyterhub-k8s.

I see installation was successful, however we could see jupyterhub pod is showing as “ImagePullBackoff” and logs shows as it is failed to pull image “jupyterhub/k8s-hub:set-by-chartpress”.

kubectl get all
NAME                         READY   STATUS             RESTARTS   AGE
pod/hub-5d9dc4b59b-mlnhh     1/2     ImagePullBackOff   0          10m
pod/proxy-77f45c8497-vrjhv   2/2     Running            0          10m

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/hub            ClusterIP      198.19.232.40   <none>          8081/TCP       10m
service/proxy-api      ClusterIP      198.19.236.82   <none>          8001/TCP       10m
service/proxy-public   LoadBalancer   198.19.230.51   35.203.76.242   80:32121/TCP   10m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hub     0/1     1            0           11m
deployment.apps/proxy   1/1     1            1           11m      

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hub-5d9dc4b59b     1         1         0       11m
replicaset.apps/proxy-77f45c8497   1         1         1       11m

NAME                                READY   AGE
statefulset.apps/user-placeholder   0/0     11m

 Normal   SuccessfulAttachVolume  2m24s                  attachdetach-controller                                  AttachVolume.Attach succeeded for volume "pvc-8e7644fe-de41-400d-838b-25183b3e61b4"
  Normal   Pulled                  2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-init
  Normal   Started                 2m21s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-init
  Normal   Pulled                  2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-proxy
  Normal   Started                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-proxy
  Normal   BackOff                 2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Back-off pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ImagePullBackOff
  Normal   Pulling                 2m8s (x2 over 2m21s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Failed to pull image "jupyterhub/k8s-hub:set-by-chartpress": rpc error: code = NotFound desc = failed to pull 
and unpack image "docker.io/jupyterhub/k8s-hub:set-by-chartpress": failed to resolve reference "docker.io/jupyterhub/k8s-hub:set-by-chartpress": docker.io/jupyterhub/k8s-hub:set-by-chartpress: not found
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ErrImagePull

Is this related to this chart - jupyterhub/k8s-hub not availble? Any help is greatly appreciated.

2

This suggests you’re installing Z2JH from the GitHub source code. This requires you to build all containers, and update some templated values, by following the development instructions:

If you just want to install the published Helm Chart it’s a lot easier, see:

1 Like
3

Thank you. I had it fixed after replacing the default “set-by-chartpress” to respective docker latest image versions in values file.
However I noticed terminal is not responding as expected, couldn’t reach internet as well.

image
image796×319 15.6 KB

4

This is caused by the notebook image not allowing non-root users to use ping:

$ sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 1   0

Only user IDs 0 to 1 are allowed.

You may use wget etc. for testing internet connectivity.

1 Like