Jupyterhub k8s failing | Error: ImagePullBackOff

Hello Folks,

We were able to deploy Jupyterhub on our private shared cluster in GCP using default chart zero-to-jupyterhub-k8s.

I see installation was successful, however we could see jupyterhub pod is showing as “ImagePullBackoff” and logs shows as it is failed to pull image “jupyterhub/k8s-hub:set-by-chartpress”.

kubectl get all
NAME                         READY   STATUS             RESTARTS   AGE
pod/hub-5d9dc4b59b-mlnhh     1/2     ImagePullBackOff   0          10m
pod/proxy-77f45c8497-vrjhv   2/2     Running            0          10m

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/hub            ClusterIP      198.19.232.40   <none>          8081/TCP       10m
service/proxy-api      ClusterIP      198.19.236.82   <none>          8001/TCP       10m
service/proxy-public   LoadBalancer   198.19.230.51   35.203.76.242   80:32121/TCP   10m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hub     0/1     1            0           11m
deployment.apps/proxy   1/1     1            1           11m      

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hub-5d9dc4b59b     1         1         0       11m
replicaset.apps/proxy-77f45c8497   1         1         1       11m

NAME                                READY   AGE
statefulset.apps/user-placeholder   0/0     11m
 Normal   SuccessfulAttachVolume  2m24s                  attachdetach-controller                                  AttachVolume.Attach succeeded for volume "pvc-8e7644fe-de41-400d-838b-25183b3e61b4"
  Normal   Pulled                  2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-init
  Normal   Started                 2m21s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-init
  Normal   Pulled                  2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-proxy
  Normal   Started                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-proxy
  Normal   BackOff                 2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Back-off pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ImagePullBackOff
  Normal   Pulling                 2m8s (x2 over 2m21s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Failed to pull image "jupyterhub/k8s-hub:set-by-chartpress": rpc error: code = NotFound desc = failed to pull 
and unpack image "docker.io/jupyterhub/k8s-hub:set-by-chartpress": failed to resolve reference "docker.io/jupyterhub/k8s-hub:set-by-chartpress": docker.io/jupyterhub/k8s-hub:set-by-chartpress: not found
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ErrImagePull

Is this related to this chart - jupyterhub/k8s-hub not availble? Any help is greatly appreciated.

This suggests you’re installing Z2JH from the GitHub source code. This requires you to build all containers, and update some templated values, by following the development instructions:

If you just want to install the published Helm Chart it’s a lot easier, see:

1 Like

Thank you. I had it fixed after replacing the default “set-by-chartpress” to respective docker latest image versions in values file.
However I noticed terminal is not responding as expected, couldn’t reach internet as well.

This is caused by the notebook image not allowing non-root users to use ping:

$ sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 1   0

Only user IDs 0 to 1 are allowed.

You may use wget etc. for testing internet connectivity.

1 Like