# Hub pod stuck on pending - timed out binding volumes

**URL:** <https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176>\
**Category:** Zero to JupyterHub on Kubernetes\
**Tags:** help-wanted\
**Created:** [December 9, 2022, 9:59am UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176 "2022-12-09T09:59:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tsutch](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@tsutch](https://discourse.jupyter.org/u/tsutch)\
**Post date:** [December 9, 2022, 9:59am UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/1 "2022-12-09T09:59:37Z")

</div>

Hello, I’m trying to follow the Zero to JupyterHub guide, using AWS and EKS.  
As the docs aren’t fully up-to-date with eksctl, I also used [this post on the Github issue](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1652#issuecomment-825125283) as a reference.

I was able to launch the cluster fine using `eksctl`, but I’m getting stuck at the installation of JupyterHub - the hub pod does not move past pending (the others are running fine).

```bash
[cloudshell-user@ip-10-0-165-104 ~]$ cat cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: Z2JHKubernetesCluster
  region: eu-west-2

nodeGroups:
  - name: ng-1
    instanceType: t2.medium
    volumeSize: 10

```

```bash
eksctl create cluster -f cluster.yaml

```

For now I’m using the chart configuration file in the [instructions](https://z2jh.jupyter.org/en/latest/jupyterhub/installation.html) which consists just of comments, and I used the following command to install the chart:

```bash
helm upgrade --cleanup-on-fail \
  --install z2jh jupyterhub/jupyterhub \
  --namespace jhub \
  --create-namespace \
  --version=2.0.0 \
  --values config.yaml

```

I’ve been trying to see what’s going on using `kubectl get pod` and `kubectl describe pod`. All pods except the hub were in Running status after a few seconds.

```bash
$ kubectl --namespace=jhub get pod
NAME READY STATUS RESTARTS AGE
continuous-image-puller-fvxsp 1/1 Running 0 31m
continuous-image-puller-gcvtp 1/1 Running 0 32m
hub-786cbd7b46-b2kgk 0/1 Pending 0 32m
proxy-58fbcfc8b4-jt8gd 1/1 Running 0 32m
user-scheduler-699549567-b7d4v 1/1 Running 0 32m
user-scheduler-699549567-cwjkf 1/1 Running 0 32m

```

After about 10 minutes a FailedScheduling warning is visible in the `describe` output - as below.

```nohighlight
$ kubectl --namespace=jhub describe pod hub-786cbd7b46-b2kgk
Name: hub-786cbd7b46-b2kgk
Namespace: jhub
Priority: 0
Node: <none>
Labels: app=jupyterhub
                component=hub
                hub.jupyter.org/network-access-proxy-api=true
                hub.jupyter.org/network-access-proxy-http=true
                hub.jupyter.org/network-access-singleuser=true
                pod-template-hash=786cbd7b46
                release=z2jh
Annotations: checksum/config-map: 1f53da94c6f18f51205e774b58c880e71974b294d25e181a0fefe3efcf585c38
                checksum/secret: ee2c2a4aebbcee22a100918b40700dec56d1ed8bdd0b8a6d472406f871151c44
                kubernetes.io/psp: eks.privileged
Status: Pending
IP:             
IPs: <none>
Controlled By: ReplicaSet/hub-786cbd7b46
Containers:
  hub:
    Image: jupyterhub/k8s-hub:2.0.0
    Port: 8081/TCP
    Host Port: 0/TCP
    Args:
      jupyterhub
      --config
      /usr/local/etc/jupyterhub/jupyterhub_config.py
      --upgrade-db
    Liveness: http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30
    Readiness: http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000
    Environment:
      PYTHONUNBUFFERED: 1
      HELM_RELEASE_NAME: z2jh
      POD_NAMESPACE: jhub (v1:metadata.namespace)
      CONFIGPROXY_AUTH_TOKEN: <set to the key 'hub.config.ConfigurableHTTPProxy.auth_token' in secret 'hub'> Optional: false
    Mounts:
      /srv/jupyterhub from pvc (rw)
      /usr/local/etc/jupyterhub/config/ from config (rw)
      /usr/local/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
      /usr/local/etc/jupyterhub/secret/ from secret (rw)
      /usr/local/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzzlv (ro)
Conditions:
  Type Status
  PodScheduled False 
Volumes:
  config:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: hub
    Optional: false
  secret:
    Type: Secret (a volume populated by a Secret)
    SecretName: hub
    Optional: false
  pvc:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: hub-db-dir
    ReadOnly: false
  kube-api-access-bzzlv:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional: <nil>
    DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=core:NoSchedule
                             hub.jupyter.org_dedicated=core:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 3m37s (x3 over 23m) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

```

Being new to Kubernetes and all this, I’m not sure where to go next, and I couldn’t find any similar issues in previous discussions. Should the hub have launched successfully within 10 minutes, or does the timeout need extending somehow?

Thanks very much for any help you’re able to provide!

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 10, 2022, 3:35pm UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/2 "2022-12-10T15:35:28Z")

</div>

It sounds like there’s a problem with your [dynamic storage volumes](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/). JupyterHub requests a volume from k8s, and k8s should respond by automatically creating it, but it’s not.

Have you tried

> **[Use persistent storage in Amazon EKS](https://aws.amazon.com/premiumsupport/knowledge-center/eks-persistent-storage/)**

?  
I’ve successfully used [Amazon EBS CSI driver - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) but not with eksctl.

---

<div class="post-metadata">

**Author:** ![tsutch](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@tsutch](https://discourse.jupyter.org/u/tsutch)\
**Post date:** [December 13, 2022, 5:08pm UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/3 "2022-12-13T17:08:37Z")

</div>

Thank you for that. I followed the AWS persistent storage page and encountered a similar problem on the test project - so it looks like it’s nothing to do with JupyterHub, anyway!

I’ll keep investigating, and will post back if I get any further. I may try setting up the cluster without eksctl to see if there’s a crucial stage that is being missed out that way.

---

<div class="post-metadata">

**Author:** ![vcfvct](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/vcfvct/32/8861_2.png) [@vcfvct](https://discourse.jupyter.org/u/vcfvct)\
**Post date:** [January 29, 2023, 9:10pm UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/4 "2023-01-29T21:10:30Z")

</div>

Hi @tsutch  
Did you happen to find the solution to this?  
I am getting exactly the same error as you. the `hub-db-dir` PV stucks at Pending status results in the `hub` pod stay at `Pending` forever.

Thanks!

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [January 31, 2023, 2:23am UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/5 "2023-01-31T02:23:59Z")

</div>

I suspect you have run into the complexity of getting peristent storage allocated when upgrading/using EKS 1.23+.

What you need is to setup the EKS cluster addon ebs-csi-drivers-addon, and you need to provide permissions for pods that will start running in kube-system.

The permissions part i dont get fully. Maybe its enough to include the “addons” entry seen in my linked issue in your eksctl config - mentioning wellKnownPolicies. Maybe some policy is also needed also on nodes where the controller runs - I’m not 100%. I recommend adding the addons section like from that issue and only adding a policy to nodes if needed.

In [Update AWS EKS terraform jinja templates to so we can use k8s 1.24 · Issue #2054 · 2i2c-org/infrastructure · GitHub](https://github.com/2i2c-org/infrastructure/issues/2054) I’ve done work to resolve that while managing a k8s cluster, linking out to various stuff.

---

<div class="post-metadata">

**Author:** ![vcfvct](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/vcfvct/32/8861_2.png) [@vcfvct](https://discourse.jupyter.org/u/vcfvct)\
**Post date:** [February 3, 2023, 4:50am UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/6 "2023-02-03T04:50:46Z")

</div>

hi @consideRatio , thanks for the info. I just recall I asked this here.  
I was able to get it thru by adding the `aws-ebs-csi-driver` addon into the yaml file with a `attachPolicy` section for the iam permissions.

```auto
addons: 
  - name: aws-ebs-csi-driver
    attachPolicy: 
      Version: "2012-10-17"
      Statement:
      - Effect: Allow
        Action:
        - "ec2:AttachVolume"
        - "ec2:CreateSnapshot"
        - "ec2:CreateTags"
        - "ec2:CreateVolume"
        - "ec2:DeleteSnapshot"
        - "ec2:DeleteTags"
        - "ec2:DeleteVolume"
        - "ec2:DescribeInstances"
        - "ec2:DescribeSnapshots"
        - "ec2:DescribeTags"
        - "ec2:DescribeVolumes"
        - "ec2:DetachVolume"
        Resource: '*'

```

Hope it helps someone else for this issue.

---

<div class="post-metadata">

**Author:** ![bpfrd](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/bpfrd/32/10762_2.png) [@bpfrd](https://discourse.jupyter.org/u/bpfrd)\
**Post date:** [December 13, 2023, 9:28pm UTC](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176/7 "2023-12-13T21:28:51Z")

</div>

I’m getting the exact same error with k3s with storage class local-path (default). I was wondering how I can resolve it?

```auto
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-a24b35bc-6ee5-49d5-82af-dce32f7ec5be 1Gi RWO Delete Bound default/hub-db-dir local-path 60m

```

```auto
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nbgrader-exchange Pending local-path 60m
hub-db-dir Bound pvc-a24b35bc-6ee5-49d5-82af-dce32f7ec5be 1Gi RWO local-path 60m

```

```auto
kubectl get pod
NAME READY STATUS RESTARTS AGE
proxy-5dbf95cf76-5mjgl 1/1 Running 0 60m
hub-78db84f8fc-97xcd 0/1 Pending 0 60m

```

```auto
kubectl describe pod/hub-78db84f8fc-97xcd | tail
                             hub.jupyter.org_dedicated=core:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 51m default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
  Warning FailedScheduling 31m (x2 over 41m) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
  Warning FailedScheduling 21m default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
  Warning FailedScheduling 52s (x2 over 11m) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

```