Jupyterhub k8s failing | Error: ImagePullBackOff

NorthBloke · September 21, 2022, 2:24am

Hello Folks,

We were able to deploy Jupyterhub on our private shared cluster in GCP using default chart zero-to-jupyterhub-k8s.

I see installation was successful, however we could see jupyterhub pod is showing as “ImagePullBackoff” and logs shows as it is failed to pull image “jupyterhub/k8s-hub:set-by-chartpress”.

kubectl get all
NAME                         READY   STATUS             RESTARTS   AGE
pod/hub-5d9dc4b59b-mlnhh     1/2     ImagePullBackOff   0          10m
pod/proxy-77f45c8497-vrjhv   2/2     Running            0          10m

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/hub            ClusterIP      198.19.232.40   <none>          8081/TCP       10m
service/proxy-api      ClusterIP      198.19.236.82   <none>          8001/TCP       10m
service/proxy-public   LoadBalancer   198.19.230.51   35.203.76.242   80:32121/TCP   10m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hub     0/1     1            0           11m
deployment.apps/proxy   1/1     1            1           11m      

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hub-5d9dc4b59b     1         1         0       11m
replicaset.apps/proxy-77f45c8497   1         1         1       11m

NAME                                READY   AGE
statefulset.apps/user-placeholder   0/0     11m

 Normal   SuccessfulAttachVolume  2m24s                  attachdetach-controller                                  AttachVolume.Attach succeeded for volume "pvc-8e7644fe-de41-400d-838b-25183b3e61b4"
  Normal   Pulled                  2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m22s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-init
  Normal   Started                 2m21s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-init
  Normal   Pulled                  2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Container image "docker.io/istio/proxyv2:1.13.5" already present on machine
  Normal   Created                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Created container istio-proxy
  Normal   Started                 2m20s                  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Started container istio-proxy
  Normal   BackOff                 2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Back-off pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m19s (x2 over 2m20s)  kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ImagePullBackOff
  Normal   Pulling                 2m8s (x2 over 2m21s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Pulling image "jupyterhub/k8s-hub:set-by-chartpress"
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Failed to pull image "jupyterhub/k8s-hub:set-by-chartpress": rpc error: code = NotFound desc = failed to pull 
and unpack image "docker.io/jupyterhub/k8s-hub:set-by-chartpress": failed to resolve reference "docker.io/jupyterhub/k8s-hub:set-by-chartpress": docker.io/jupyterhub/k8s-hub:set-by-chartpress: not found
  Warning  Failed                  2m7s (x2 over 2m20s)   kubelet, gke-banting-gloucester-node-pool-85bca9bf-tjhg  Error: ErrImagePull

Is this related to this chart - jupyterhub/k8s-hub not availble? Any help is greatly appreciated.

manics · September 21, 2022, 1:09pm

This suggests you’re installing Z2JH from the GitHub source code. This requires you to build all containers, and update some templated values, by following the development instructions:

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/dfa9b1d91bb4b07474f10ac7fd976143917b4b53/CONTRIBUTING.md

# Contributing

Welcome dear open source colleague! As this is a [Jupyter](https://jupyter.org)
project please start by looking at the [Jupyter contributor
guide](https://jupyter.readthedocs.io/en/latest/contributing/content-contributor.html),
and follow [Project Jupyter's Code of
Conduct](https://github.com/jupyter/governance/blob/HEAD/conduct/code_of_conduct.md)
to help us sustain a warm and welcoming collaborative environment.

If you don't have [git](https://www.git-scm.com/) already, install it and clone
this repository.

```shell
git clone https://github.com/jupyterhub/zero-to-jupyterhub-k8s
```

# Setting up for documentation development

See [docs/README.md](docs/README.md).

This file has been truncated. show original

If you just want to install the published Helm Chart it’s a lot easier, see:

NorthBloke · September 21, 2022, 5:29pm

Thank you. I had it fixed after replacing the default “set-by-chartpress” to respective docker latest image versions in values file.
However I noticed terminal is not responding as expected, couldn’t reach internet as well.

BenGig · September 23, 2022, 9:43am

This is caused by the notebook image not allowing non-root users to use ping:

$ sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 1   0

Only user IDs 0 to 1 are allowed.

You may use wget etc. for testing internet connectivity.

Topic		Replies	Views
Helm Chart wont pull private image Zero to JupyterHub on Kubernetes jupyterhub	1	571	June 23, 2021
Error when starting Jupyterhub with image from my public docker registry Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	1	1570	December 11, 2021
Helm install on-prem K8 Zero to JupyterHub on Kubernetes	9	3306	January 21, 2021
Crash loopback on hub pod Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	2	676	November 11, 2021
Hook-image-puller not use the specific mirror Zero to JupyterHub on Kubernetes help-wanted	6	27	October 19, 2024

Jupyterhub k8s failing | Error: ImagePullBackOff

Related topics