I am running a self-hosted MicroK8s cluster with JupyterHub installed via a Helm chart. The cluster uses an NFS server for persistent storage, but I am encountering an issue where one of my volumes fails to mount properly. The error from the kubelet logs is as follows:
MountVolume.SetUp failed for volume "pvc-xxxx" : applyFSGroup failed for vol xx.xx.xx.xx#srv/nfsv4#pvc-xxxx##: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/xxxx/volumes/kubernetes.io~csi/xxxx/mount: no such file or directory
The setup involves:
- MicroK8s as the Kubernetes distribution.
- Persistent volumes backed by NFS storage.
- JupyterHub installed using Helm.
- microk8s csi driver installed using helm
I’ve verified that the NFS server is accessible and the share is correctly configured. However, this error persists, and the pod cannot start due to the volume mount issue.
Could someone help me understand:
What might be causing this error?
How can I troubleshoot and resolve this problem?
Jupyterhub helm chart versions tested 3.1.0 and 4.0.0. Microk8s versions tested 1.28.15 and 1.31.3.
This setup was working for the past year but now is causing problems.
Anybody else facing this issue or have any suggestion. Please help.