- Some notes:
I was directed here via this issue on github. I assume this is not a bug and is instead a support request as tagged. The issue is 3175 in jupyterhub/zero-to-jupyterhub-k8s, but unfortunately I’m limited to 2 links per post as as new user.
- Some environment setup:
I am running z2jh via EKS on AWS. With the requirement to use a [ReadWriteMany] drive, I switched from EBS to EFS following this guide.
To provide more control I am using custom Docker images. My hub is pulling from jupyterhub/k8s-hub:3.0.0-beta.3 while my user is pulling from jupyter/minimal-notebook:2023-07-17. Currently - these just change the environment variables from jovyan to a custom user name, and runs usermod to reflect the changes.
USER root
ENV NB_USER ${NB_USER}
ENV HOME ${HOME}
ENV PWD ${PWD}
ENV JUPYTER_SERVER_ROOT ${JUPYTER_SERVER_ROOT}
ENV NB_UID ${NB_UID}
ENV NB_GID ${NB_GID}
RUN usermod -l ${NB_USER} jovyan
RUN usermod -d ${HOME} -m ${NB_USER}
WORKDIR ${HOME}
USER ${NB_USER}
CMD ["/bin/bash", "-c", "fix-permissions /home;start-singleuser.sh"]
With using jupyter/minimal-notebook I following this suggestion to mount the EFS.
- The overall goal/requirement:
Have one volume location for user storage, and a separate volume for shared storage between users.
The EFS should reflect:
├── home
│ ├── shared
│ ├── user1
│ ├── user2
...
While the user pod should reflect:
├── home
│ ├── custom
│ │ ├── user1
│ │ ├── shared
...
- Baseline 1:
I can connect to the EBS using static storage without issue, the below works as expected.
storage:
type: static
homeMountPath: /home/custom/{username}
static:
pvcName: static-pvc
subPath: "{username}"
- Baseline 2:
I can connect to the EBS using an initContainer without issue, the below works as expected.
storage:
type: none
extraVolumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: shared-pvc
extraVolumeMounts:
- name: persistent-storage
mountPath: /home/custom/shared
subPath: shared
capacity: 5Gi
initContainers:
- name: nfs-fixer
image: alpine
securityContext:
runAsUser: 0
volumeMounts:
- name: persistent-storage
mountPath: /nfs
subPath: shared
command:
- sh
- -c
- (chmod 0775 /nfs; chown -R 1000:100 /nfs)
- Attempt 1:
Mark type: static while also using an extraVolume / extraVolumeMounts.
Issue: The pod hangs on spawning.
storage:
type: static
homeMountPath: /home/custom/{username}
static:
pvcName: static-pvc
subPath: "{username}"
extraVolumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: shared-pvc
extraVolumeMounts:
- name: persistent-storage
mountPath: /home/custom/shared
subPath: shared
capacity: 5Gi
initContainers:
- name: nfs-fixer
image: alpine
securityContext:
runAsUser: 0
volumeMounts:
- name: persistent-storage
mountPath: /nfs
subPath: shared
command:
- sh
- -c
- (chmod 0775 /nfs; chown -R 1000:100 /nfs)
Neither the hub logs nor the user pod have any relevant information, and eventually the pod times out with
2023-07-27T22:25:54Z [Warning] Unable to attach or mount volumes: unmounted volumes=[home], unattached volumes=, failed to process volumes=[home]: timed out waiting for the condition
- Attempt 2:
Use two separate initContainers.
storage:
type: none
extraVolumes:
- name: persistent-shared
persistentVolumeClaim:
claimName: shared-pvc
- name: persistent-user
persistentVolumeClaim:
claimName: user-pvc
extraVolumeMounts:
- name: persistent-shared
mountPath: /home/custom/shared
subPath: home/custom/shared
- name: persistent-user
mountPath: /home/custom/{username}
subPath: home/custom/{username}
initContainers:
- name: nfs-shared
image: alpine
securityContext:
runAsUser: 0
volumeMounts:
- name: persistent-shared
mountPath: /nfs/shared
subPath: home/custom/shared
command:
- sh
- -c
- (chmod 0775 /nfs; chown -R 1000:100 /nfs)
initContainers:
- name: nfs-user
image: alpine
securityContext:
runAsUser: 0
volumeMounts:
- name: persistent-user
mountPath: /nfs/user
subPath: home/custom/personal/{username}
command:
- sh
- -c
- (chmod 0775 /nfs; chown -R 1000:100 /nfs)
Again, the pod times out with a similar message.
2023-07-28T18:28:56Z [Warning] Unable to attach or mount volumes: unmounted volumes=[persistent-shared], unattached volumes=, failed to process volumes=[persistent-shared]: timed out waiting for the condition
While the hub/user logs doesn’t seem to be helpful, the describe pod output for the user pod may have something relevant, though I’m unsure:
Volumes:
persistent-shared:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: shared-pvc
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: user-pvc
ReadOnly: false
Both PVC appear to be on the same level, unstead the persistent-shared volume. However, in the config.yaml they are set as different volumes.
Though other relevant sections appear as expected:
Containers:
Mounts:
/home/custom/username from persistent-storage (rw,path="home/custom/personal/username")
/home/custom/shared from persistent-shared (rw,path="home/custom/shared")
Init Containers:
nfs-shared-fix:
Container ID:
Image: alpine
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
(chmod 0775 /nfs; chown -R 1000:100 /nfs)
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/nfs/user from persistent-storage (rw,path="home/custom/personal/username")
nfs-user-fix:
Container ID:
Image: alpine
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
(chmod 0775 /nfs; chown -R 1000:100 /nfs)
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/nfs/shared from persistent-shared (rw,path="home/custom/shared")
Is there any other way I’m missing to mount more than one volume?