We configured jupyterhub so that we mount NAS shares in our users’ notebooks, great.
Some are NFS mounts, other are kerberized ones, and we also have some case where we use SMB CSI and mount shares based on k8s secrets containing user/password
The problem is that our users regularly face failure when trying to start their notebook, because a password has been changed and one “mount” operation (one out of many) simply fails.
Is there a way to configure jupyter so that failed mounts are simply “skipped” instead of preventing completely the notebook startup ? The notebooks would run without some shares but would run anyway…
JupyterHub/Z2JH creates a specification for what to run and mount, which is passed on to Kubernetes to execute. JupyterHub doesn’t actually do the mounting itself. Kubernetes automatically retries things on failure, the idea being to handle temporary interruptions/outages, but it doesn’t support ignoring volume mounts if they fail.
In theory you could rewrite KubeSpawner to detect a volume mount failure, and after a while delete the failed pod specification and create a new one without the failed volume. This would get very complicated though, especially if you’ve got multiple volumes, and means users would have an inconsistent environment.
I think it would be a better use of time to work out the root cause of the incorrect passwords, e.g. prevent them from being changed without warning, or if they are changed ensure your system updates everything consistently.
Otherwise, you could just not mount the volumes in the first place since they’re optional?