Help getting the right blob mount permissions

JupyterHub Zero to JupyterHub on Kubernetes
1

Hello,
I have a cluster running on AKS after following the guide
I am trying to mount a static Azure Blob Storage (bring your own storage) shared by all of the singleuser pods (I have a custom spawner that mounts subdirectories of the blob). I have setup an SC, PV, and PVC and successfully mounted the blob using the Azure CSI Driver with NFS and fsGroup enabled; however, I am having issues with permissions.

hub:
...
singleuser:
  ...
  uid: 1000 # This sets the User ID for the pod to 1000, which corresponds to the jovyan user. It ensures that processes inside the container run as jovyan.
  fsGid: 100 # The filesystem group ID to use for the POD (users/100). This determines the group ownership of mounted volumes, ensuring that files and directories within these volumes are accessible by members of the users group.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-blob-nfs-sc
provisioner: blob.csi.azure.com
parameters:
  protocol: nfs
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  # I am honestly not even certain these are being respected.
  - nconnect=4
  - dir_mode=2770
  - file_mode=0770
  - uid=1000
  - gid=100

apiVersion: v1
kind: PersistentVolume
metadata:
  name: azure-blob-nfs-pv
spec:
  capacity:
    storage: 1Pi # While the Kubernetes API capacity attribute is mandatory, this value isn't used by the Azure Blob storage CSI driver because you can flexibly write data until you reach your storage account's capacity limit. The value of the capacity attribute is used only for size matching between PersistentVolumes and PersistenVolumeClaims. We recommend using a fictitious high value. The pod sees a mounted volume with a fictitious size of 5 Petabytes.
  accessModes:
    - ReadWriteMany # Allow our many singleuser nodes to read and write to the volume.
  persistentVolumeReclaimPolicy: Retain # Keep the volume after the PVC is deleted.
  storageClassName: azure-blob-nfs-sc
  mountOptions:
    - nconnect=4
  csi:
    driver: blob.csi.azure.com
    readOnly: false
    volumeHandle: ... # Change this to a combination of the name of the Azure Storage Account and Container.
    volumeAttributes:
      resourceGroup: ... # Change this to the name of the Azure Resource Group.
      storageAccount: ... # Change this to the name of the Azure Storage Account.
      containerName: ...
      protocol: nfs
      mountPermissions: '2770' # Enforces that new files in the mounted folder inherit the group ID, and grants full read, write, and execute permissions to the owner and group, while denying all access to others.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azure-blob-nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 3Gi # How much storage to request from the volume per user.
  volumeName: azure-blob-nfs-pv
  storageClassName: azure-blob-nfs-sc

I see that the group is correctly set for new files added to the mount (somehow) and I have read and write, but no execute access.

jovyan@jupyter-brooke:~$ ls -al /mnt/data/
total 150
drwxrwsrwx 2 root   users      0 Dec 28 21:24 .
drwxr-xr-x 1 root   root    4096 Dec 28 22:00 ..
-rw-rw---- 1 root   users 148574 Dec 28 21:21 alice_in_wonderland.txt
-rw-rw-r-- 1 jovyan users     15 Dec 28 21:24 hello_blob.txt

jovyan@jupyter-brooke:~$ echo "Now in Azure Blob Storage!" >> /mnt/data/hello_blob.txt 
jovyan@jupyter-brooke:~$ tail -n 1 /mnt/data/hello_blob.txt 
Now in Azure Blob Storage!

jovyan@jupyter-brooke:~$ echo "Now in Azure Blob Storage!" >> /mnt/data/alice_in_wonderland.txt
jovyan@jupyter-brooke:~$ tail -n 3 /mnt/data/alice_in_wonderland.txt 
remembering her own child-life, and the happy summer days.

                             THE END

hello_blob.txt is owned by jovyan because it was created from inside the container. alice_in_wonderland.txt does not, because it was created in the azure portal or through the blob API.

I am not sure why the execute access is not coming through on any of these files nor why the owner is set to root instead of jovyan (1000) given that my pod’s yaml correctly lists aspects of the securityContext such as runAsUser: 1000…
Additionally even though it says I have write group permissions on the files I can’t actually seem to be able to modify the files with a script.

The end goal would be to have all files in the blob (created either internally or externally through the portal/api) owned by jovyan with rwx, however I would settle for owned by group with rwx.

I have been banging my head at this for a few days thus any help would be very appreciated!

2

Z2JH relies on Kubernetes to setup it’s resources correctly, so it’s going to require someone with access to AKS to figure out the permissions (assuming it’s even possible, e.g. blob storage may have some limitations on what you can do).

It’s probably worth trying to manually create a minimal PVC and pod manifest outside of Z2JH, and play around with the settings. If you can find something that works we can try and figure out how to translate that back into the Z2JH configuration. If you can’t then it could be worth sharing your minimal pod/PVC example on an Azure support forum if there is one?