User pod struck in pending state in EKS version 1.27

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Previously, we had EKS version 1.23. Now we updated the EKS version to 1.27 and deployed JupyterHub 3.0.0 (helm chart 2.0.0 - latest stable version) in it. All the components are up and running (pods, services). I am able to see the signin page. But once I login, the user pod is in pending state and not able to launch the jupyter server.

image
image711×25 1.33 KB

image
image1920×878 25.5 KB

PVC is also in pending state and these are the events when I describe the PVC.

image
image1472×103 5.34 KB

This is the error message in user-scheduler.

W0621 09:29:59.765092       1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
E0621 09:29:59.765118       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource

Does this issue relates to the EKS version update or because of any other reason?

1 Like
2

You’ll need a recent dev release of Z2JH to run on K8s 1.27

2 Likes
3

@manics Which release of Z2JH? I have the same problem and I am using Z2JH 3.2.1 with k8s 1.28
Thanks!

4

I haven’t checked exactly which version introduced the fixes, but the latest version 3.2.2-0.dev.git.6475.hec8e3e87 should work

5

hmm, 3.2.2-0.dev.git.6475.hec8e3e87 is not on docker hub yet. The latest one as of 2024-01-16 is 3.2.2-0.dev.git.6469.hb503bcac.

6

@manics Thank you for your help. I tried the latest Z2JH docker image I can find, 3.2.2-0.dev.git.6469.hb503bcac . I ran into the same problem. The user-placeholder-0 pod still stuck in pending state.

Do you have any info on the original problem? Is there a discussion thread? git hub PR? Thank you very much!

7

I had to disable user scheduler to get around this.

8

Can you test the latest Helm chart dev version? The version is not always the same as the Docker image tags, since the images are only rebuilt if they’re modified.

9

I expected that chart version 3.2.1 should have worked without issues in EKS k8s 1.28, doesnt it?

A recent dev release of the chart is available that may work due to some updates to user-scheduler last weeks.