Warning: Unable to mount volumes for pod

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,

I have deployed a JupyterHub with https://zero-to-jupyterhub.readthedocs.io

When the user logs into JupyterHub, here is the event log:

Event log
Server requested
2020-10-15 06:46:18+00:00 [Normal] Successfully assigned jhub8/jupyter-test to test-dnwwibnpja2o-node-0
2020-10-15 06:46:22+00:00 [Normal] AttachVolume.Attach succeeded for volume "pvc-c9132587-91a5-4cdc-9114-407c642bc482"
2020-10-15 06:48:21+00:00 [Warning] Unable to mount volumes for pod "jupyter-test_jhub8(34093689-e8da-47f3-8811-e2dbbaf35fc1)": timeout expired waiting for volumes to attach or mount for pod "jhub8"/"jupyter-test". list of unmounted volumes=[volume-test]. list of unattached volumes=[volume-test]

However, the user volume is readily available after login. In case it helps here is more information:

# kubectl get events -n jhub8
LAST SEEN   TYPE      REASON                   OBJECT                                 MESSAGE
2m55s       Normal    Scheduled                pod/jupyter-test                       Successfully assigned jhub8/jupyter-test to test-dnwwibnpja2o-node-0
2m51s       Normal    SuccessfulAttachVolume   pod/jupyter-test                       AttachVolume.Attach succeeded for volume "pvc-c9132587-91a5-4cdc-9114-407c642bc482"
52s         Warning   FailedMount              pod/jupyter-test                       Unable to mount volumes for pod "jupyter-test_jhub8(34093689-e8da-47f3-8811-e2dbbaf35fc1)": timeout expired waiting for volumes to attach or mount for pod "jhub8"/"jupyter-test". list of unmounted volumes=[volume-test]. list of unattached volumes=[volume-test]
25s         Normal    Pulled                   pod/jupyter-test                       Container image "jupyterhub/k8s-network-tools:0.9.1" already present on machine
25s         Normal    Created                  pod/jupyter-test                       Created container block-cloud-metadata
25s         Normal    Started                  pod/jupyter-test                       Started container block-cloud-metadata
24s         Normal    Pulled                   pod/jupyter-test                       Container image "amigahub/jupyter-desktop-server:20201014" already present on machine
24s         Normal    Created                  pod/jupyter-test                       Created container notebook
23s         Normal    Started                  pod/jupyter-test                       Started container notebook
3m43s       Normal    NoPods                   poddisruptionbudget/user-placeholder   No matching pods found

What else can I check to narrow down the root cause of the issue?

Many thanks,
Sebastian

2

I see that you are using Kubernetes for Jupyterhub. Did you install Storageclass for this namespace? You can get more information here. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/421

3

Thanks @nnbaokhang

Yes, I configured a default storage class for this kubernetes cluster
# kubectl get sc -n jhub8
NAME PROVISIONER AGE
cinder (default) kubernetes.io/cinder 16d

Here are the details:
# kubectl describe storageclass cinder
Name: cinder
IsDefaultClass: Yes
Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:“storage.k8s.io/v1”,“kind”:“StorageClass”,“metadata”:{“annotations”:{},“name”:“cinder”},“provisioner”:"kubernetes.io/cinder"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/cinder
Parameters:
AllowVolumeExpansion:
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:

Many thanks,
Sebastian

4

There seems to be a 120-second timeout triggered. I was wondering whether one could shorten this timeout? Even if it is hard-coded and the code needs to be patched for a specific deployment.