Hey everyone! New user of discourse here, but at least mildly competent user of JupyterHub (or at least I’d like to think).
I’m essentially trying to use JupyterHub for a shared development environment for a team of about a dozen users (we may expand this to larger/other teams later, but for now I want to get this working for just our team), and one feature that would be extremely helpful, and looks doable, is having a shared directory for notebooks, files, and data. I think I’m pretty close to getting this set-up, but I’m running into a mounting issue that I can’t quite resolve. I’ll quickly explain my setup first and then the issue. I’d really appreciate any help/comments/hints that anyone has!
Currently, all of this setup is on a Kubernetes cluster in Azure or other Azure-hosted services. We have a resource group with a kubernetes cluster, App Service Domain, DNS Zone, virtual network, container registry (for our custom docker images), and storage account. Everything works fine, except that in the storage account, I have an Azure SMB (server message block, as opposed to NFS) file share that I’ve tried mounting via a PV (persistent volume) and PVC (persistent volume claim) to a JupyterHub server, but to no avail.
I’ve created the PV and PVC successfully using the following YAML files.
For the PV:
apiVersion: v1 kind: PersistentVolume metadata: name: azurefile spec: capacity: storage: 10Gi accessModes: - ReadWriteMany azureFile: secretName: azure-secret shareName: aksshare readOnly: false mountOptions: - dir_mode=0777 - file_mode=0777 - uid=1000 - gid=1000 - mfsymlinks - nobrl
For the PVC:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: azurefile spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 10Gi
kubectl apply -f azurefile-mount-options-pv.yaml kubectl apply -f azurefile-mount-options-pvc.yaml
$ kubectl get pvc azurefile NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE azurefile Bound azurefile 10Gi RWX 19h
I also created the corresponding kubernetes secret from the storage account key via
# Get storage account key STORAGE_KEY=$(az storage account keys list --resource-group $resourceGroupName --account-name $storageAccountName --query ".value" -o tsv) kubectl create secret generic azure-secret \ --from-literal=azurestorageaccountname=$storageAccountName \ --from-literal=azurestorageaccountkey=$STORAGE_KEY
I’m pretty confident that I can verify that this secret was properly created and transmitted (mainly since I previously had a bug where it wasn’t being read, and I was able to fix that).
Lastly, I added the following to my Helm
singleuser: storage: extraVolumes: - name: azure persistentVolumeClaim: claimName: azurefile extraVolumeMounts: - name: azure mountPath: /home/shared
I can also post my entire Helm config if needed, but I think this is the only relevant portion, and I’m not currently testing on my custom Docker image (although I can if needed), so that shouldn’t be a complicating issue.
With this setup, I repeatedly get the following error when trying to spawn a server:
2021-07-14T00:56:00Z [Warning] MountVolume.SetUp failed for volume "azurefile" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/dbfc3ed8-9b4a-44bd-94a8-f6f1990c5285/volumes/kubernetes.io~azure-file/azurefile --scope -- mount -t cifs -o dir_mode=0777,file_mode=0777,gid=1000,mfsymlinks,nobrl,uid=1000,vers=3.0,actimeo=30,<masked> //wintermutehdd.file.core.windows.net/aksshare /var/lib/kubelet/pods/dbfc3ed8-9b4a-44bd-94a8-f6f1990c5285/volumes/kubernetes.io~azure-file/azurefile Output: Running scope as unit: run-r952aaf8cbead4524b23895a7836cf070.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
I get about a dozen of these errors, with
Output: Running scope as unit: run-<random number>.scope being the part that changes in every error. Eventually, the entire spawn fails after via a 300 second timeout.
Generally, I’m following a suggestion that I got on the JupyterHub Gitter
and the only real difference I can see is that I’m using an SMB file share instead of an NFS one. I could switch over to NFS (although this requires a “premium” file share), but I don’t see why this couldn’t work with SMB.
I’ve also looked through this post Scaleable JupyetrHub Deployments in Education (Teaching) - Special Topics / Education - Jupyter Community Forum and its linked posts, but I don’t think they exactly solve my problem.
I tried to include all the relevant info that I thought would be needed, but let me know if I missed anything. Again, thank you in advance!
I caved and tried to use the premium file share using NFS, but I seem to have just moved the issue around. When trying to create a PV using the following yaml
apiVersion: v1 kind: PersistentVolume metadata: name: shared-nfs-pv spec: capacity: storage: 100Gi accessModes: - ReadWriteMany azureFile: secretName: azure-secret shareName: aksshare readOnly: false nfs: server: wintermutessd.file.core.windows.net:/wintermutessd/wintermutessdshare path: /home/shared readOnly: false storageClassName: premium-nfs mountOptions: - dir_mode=0777 - file_mode=0777 - uid=1000 - gid=1000 - mfsymlinks - nobrl
I get the error:
Failed to create the persistentvolume 'shared-nfs-pv'. Error: Invalid (422) : PersistentVolume "shared-nfs-pv" is invalid: spec.azureFile: Forbidden: may not specify more than 1 volume type. Removing the
azureFile options solves this error, but I feel like it would be necessary to specify the kubernetes secret that I created. Going along with the PV created by removing
azureFile, I can create a PVC for it and then try to mount it on jupyterhub using
extraVolumeMounts as before. This seems to be slightly better, as I only get one mounting error instead of many:
Just in case this is relevant, the NFS azure file share is only accessible via a private endpoint, but this should be fine since my kubernetes cluster is running in the same virtual network. In fact, Azure tells me that I could just mount this NFS share on linux with
sudo apt-get -y update sudo apt-get install nfs-common sudo mkdir -p /mount/wintermutessd/wintermutessdshare sudo mount -t nfs wintermutessd.file.core.windows.net:/wintermutessd/wintermutessdshare /mount/wintermutessd/wintermutessdshare -o vers=4,minorversion=1,sec=sys
As always, any ideas welcome