PVCs as home directories?

JupyterHub
1

Hi - I was wondering what the right method of using PVCs for home directories (with kubespawner) are. Currently using the singleuser 0.9.4 docker image, but I’m comfortable building my own image if that becomes necessary.

I’m provisioning the PVCs myself (so pvc_ensure) is off, and then passing the PVC via volumes, and volume_mounts into kube spawner.

I’ve tried setting the path of the volume mount to be “/”, “/home”, and “/home/jovyan”.

  • Kubernetes won’t let me set the PVC to be “/”.
  • If I set the PVC to be “/home”, then the single user server won’t startup becuase “/jhome/jovyan” does not exist".
  • If I set the PVC mount point to be “/home/jovyan”, the permissions are owned by root, so we have permission issues (writing to “~/.local”).

From what I can gather, all the dockerfile code is excuted BEFORE the pvc is mounted into the container, so wherever I mount the PVC, It has to be separate from what docker touches?

Thanks!

2

(I’m experimenting with this outside of kubespawner now)

IF I do this:

apiVersion: v1
kind: Pod
metadata:
name: kuard
namespace: community-namespace
spec:
volumes:
- name: vol
persistentVolumeClaim:
claimName: “some-claim”
containers:

  • image: “jupyterhub/singleuser:0.9.4”
    name: kuard
    volumeMounts:
    • name: vol
      mountPath: /home

I get:

Error: failed to start container “kuard”: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused “chdir to cwd ( “/home/jovyan”) set in config.json failed: no such file or directory”

3

I think I’ve been able to confirm my hypothesis. If I setup a shim Dockerfile like this:

FROM ubuntu:18.04
RUN touch “/home/foo.txt”
RUN touch “/tmp/foo.txt”
RUN touch “/var/foo.txt”
CMD ["/bin/sleep", “100”]

And I mount a pvc onto /home, then /home/foo.txt does not exist, but the other txt files do exist (so PVCs just replace mount points of the docker image.)

That says to me that I have to do a few things

  • mount my PVC inside the home directory (maybe in work?)
  • use an initContainer or something else to set the proper perms on the PVC

I don’t see any way around this - but definitely open to suggestions!

4

Well, I don’t know if there is a better way, but I confirmed that mounting a PVC in an innit container and chgrp to 100 and chown to 1000 works, and leaves you with proper permissions in the resulting notebook.

I do think if I do this I can replace the entire /home/jovyan

5

The initial permissions of a PV seem to be an issue with the volume provisioners. Creating root-private volumes is ~never the right thing to do, but it’s a common, longstanding, well-known issue. I don’t really understand what it is that makes some volume provisioners not have this issue (GKE volumes start with fine permissions) while others don’t (EmptyDir often starts root-only). I believe the fix should mostly be in the volume provisioner itself, fixing the permissions on volume creation, if you have access to that.

However, if you can’t fix the provisioner, then fixing permissions (and missing directories) in an init container seems like a very sensible and practical solution.

6

Does setting a securityContext fsGroup help?
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

7

Hello, I’am reviving this topic because I think I have a related issue: I’m using Rook.io to orchestrate my storage and it uses Ceph as storage provisioner. When I launch a server the directory /home/jovyan is empty while it should have all the notebooks that I included in the image. I was wondering if there have been made improvements on this issue and what is now the best way is to go about this.