# Recommendations on Dynamic PersistentVolumes on Newer AWS EKS Versions

**URL:** <https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829>\
**Category:** Zero to JupyterHub on Kubernetes\
**Tags:** how-to, help-wanted\
**Created:** [February 3, 2023, 3:55pm UTC](https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829 "2023-02-03T15:55:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![acorrado](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/acorrado/32/9033_2.png) [@acorrado](https://discourse.jupyter.org/u/acorrado)\
**Post date:** [February 3, 2023, 3:55pm UTC](https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829/1 "2023-02-03T15:55:22Z")

</div>

Hello,  
I’ve been walking through z2jh documentation which is thorough and great!

I set up an AWS EKS cluster with `eksctl`. I then `helm install` `jupyterhub-2.0.0`. The newer versions of EKS tend to use the EBS CSI driver for dynamic `PersistentVolume` provisioning using the annotation:

`volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs`

Your [older storage documentation](https://z2jh.jupyter.org/en/stable/kubernetes/amazon/step-zero-aws.html?highlight=ebs-csi#shared-storage) (guessing it’s older) seems to talk about EFS and I don’t see anything about the EBS CSI driver on there.

Should I not be using EBS CSI and using EFS? What is the recommended next steps after deploying jupyterhub to get it working with the EBS CSC driver? Currently my `hub` pod is in a `"PENDING"` state. I don’t want to make this a bug thread and can work with the EBS CSI community over there on potential resolutions.

What I’m looking for are pointers/recommendations from the experts in this community about how they went about installing v2.0.0 on newer EKS clusters. I could then contribute back to z2hk with some docs on the solution.

I’m about to follow this documentation to troubleshoot problems. Any other threads/docs I should be following besides this?

> <https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/b926e1ee4551e60b44c9b5a887409a8feaf91d86/docs/install.md>

**Version Info:**

```auto
$ eksctl info   
eksctl version: 0.127.0-dev+d97310bd4.2023-01-27T12:47:55Z
kubectl version: v1.25.0

```

```auto
jupyterhub-2.0.0.tgz

```

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [February 4, 2023, 1:57pm UTC](https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829/2 "2023-02-04T13:57:58Z")

</div>

There’s a related thread here:

> [@Hub pod stuck on pending - timed out binding volumes](https://discourse.jupyter.org/t/hub-pod-stuck-on-pending-timed-out-binding-volumes/17176):
>
> Hello, I’m trying to follow the Zero to JupyterHub guide, using AWS and EKS. As the docs aren’t fully up-to-date with eksctl, I also used [this post on the Github issue](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1652#issuecomment-825125283) as a reference. I was able to launch the cluster fine using eksctl, but I’m getting stuck at the installation of JupyterHub - the hub pod does not move past pending (the others are running fine). [cloudshell-user@ip-10-0-165-104 ~]$ cat cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: Z2JHKuber…

I think the best approach is to get EBS CSI working on your cluster ignoring JupyterHub (test it with a manually created Pod/PVC and check the PV is dynamically created), then work on the JupyterHub configuration.

---

<div class="post-metadata">

**Author:** ![acorrado](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/acorrado/32/9033_2.png) [@acorrado](https://discourse.jupyter.org/u/acorrado)\
**Post date:** [February 5, 2023, 11:09pm UTC](https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829/3 "2023-02-05T23:09:56Z")

</div>

For posterity’s sake here’s the correct order from `eksctl create cluster` through `helm install jupyterhub` on EKS 1.24:

**Create the cluster:**

```auto
eksctl create cluster \
--name blah --region us-west-1 \
--ssh-access=true --ssh-public-key=~/.ssh/id_ed25519_k8.pub \
--nodegroup-name=hub-node --node-type=t2.xlarge \
--nodes=1 --nodes-min=1 --nodes-max=5

```

**Check to see if you have an OIDC provider in EKS already**

If you just created you cluster you probably don’t, but hey bash is fun:

```auto
oidc_id=$(aws eks describe-cluster --name blah --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4
# no output means no existing provider

```

create the OIDC provider

```bash
eksctl utils associate-iam-oidc-provider \
--cluster blah--region us-west-1 --approve

```

**Create the IAM Role that the local K8 `ServiceAccount` Uses:**

# [Use Kubernetes volume storage with Amazon EBS - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html)

Ignore some of the document comments here. The k8 `ServiceAccount` and controller isn’t created yet even though the docs make you think it already is

```auto
eksctl create iamserviceaccount \
  --region us-west-1 \
  --name ebs-csi-controller-sa \ # name the ServiceAccount in k8 (cannot change)
  --namespace kube-system \
  --cluster blah \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name <blah-role> # name of the IAM Role

```

**Create the ebs-csi driver addon**

> **[Use Kubernetes volume storage with Amazon EBS - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html#managing-ebs-csi)**
>
> The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver manages the lifecycle of Amazon EBS volumes as storage for Kubernetes Volumes.

This step creates k8 `ServiceAccount` and ebs-sci pods as well:

```auto
eksctl create addon --name aws-ebs-csi-driver \
--region us-west-1 --cluster blah \
--service-account-role-arn arn:aws:iam::<your-aws-account-id>:role/<iam-role-name-from-last-step> --force

```

Check to make the `ServiceAccount` has an annotation of your IAM role:

```auto
$ kubectl get sa ebs-csi-controller-sa -n kube-system -o yaml | grep -a1 annotations
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/<iam-role-name>

```

Check to make sure we have controller pods up for ebs-csi

```auto
kubectl get pod -n kube-system | grep ebs-csi
ebs-csi-controller-5cbc775dc5-hr6mz 6/6 Running 0 4m51s
ebs-csi-controller-5cbc775dc5-knqnr 6/6 Running 0 4m51s

```

**Install JupyterHub:**

```auto
helm upgrade --cleanup-on-fail \
  --install blah-hub jupyterhub/jupyterhub \
  --namespace jupyterhub \
  --create-namespace \
  --version=2.0.0 \
  --values config.yaml

```

---

<div class="post-metadata">

**Author:** ![Derrik](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/derrik/32/9581_2.png) [@Derrik](https://discourse.jupyter.org/u/Derrik)\
**Post date:** [April 7, 2023, 12:47am UTC](https://discourse.jupyter.org/t/recommendations-on-dynamic-persistentvolumes-on-newer-aws-eks-versions/17829/4 "2023-04-07T00:47:28Z")

</div>

Does anyone know if the binderhub helm install supports eks 1.25 using the gp2 storageclass? Looks like the hub-db-dir pvc gets created but the singleuser.storage.capacity argument seems to get ignored. I have tried [0.2.0-n1011.hb49edf6] and the latest 1.0.0 dev release [1.0.0-0.dev.git.3035.h32213a4] with no luck.