Mount local directory to jupyterhub pods

JupyterHub Zero to JupyterHub on Kubernetes
,
1

I am trying to mount local directory to spawned jupyterlab with kubernetes, but spawner freezes on last

  1. I created cluster with with kind:
kind create cluster --config ./multinode-conf.yaml --name jupyter
  1. I have default storageclass output:
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  87s
  1. created namespace:
kubectl create namespace jupyter
  1. installed helm:
helm install jhub-datascience jupyterhub/jupyterhub -f scipy-jhub-values-kind.yaml -n jupyter --timeout 180s
  1. output of kubectl get pods
NAME                              READY   STATUS              RESTARTS   AGE
continuous-image-puller-mflnt     1/1     Running             0          11s
continuous-image-puller-pzm56     1/1     Running             0          11s
hub-54cc8f7c5c-v4xch              0/1     ContainerCreating   0          11s
proxy-7699bccdcb-t6wfc            1/1     Running             0          11s
user-scheduler-5498fc849d-ggv8h   0/1     ContainerCreating   0          11s
user-scheduler-5498fc849d-ql9bm   1/1     Running             0          11s
user-scheduler-5498fc849d-ggv8h   0/1     Running             0          12s
user-scheduler-5498fc849d-ggv8h   1/1     Running             0          14s
hub-54cc8f7c5c-v4xch              0/1     Running             0          21s
  1. Event log freezed on
Event log
Server requested
2022-09-21T21:16:00.492361Z [Normal] Successfully assigned jupyter/jupyter-myname to jupyter-worker
2022-09-21T21:16:01Z [Normal] Container image "jupyterhub/k8s-network-tools:2.0.0" already present on machine
2022-09-21T21:16:01Z [Normal] Created container block-cloud-metadata
2022-09-21T21:16:01Z [Normal] Started container block-cloud-metadata
2022-09-21T21:16:02Z [Normal] Container image "miykael/nipype_tutorial:latest" already present on machine
2022-09-21T21:16:02Z [Normal] Created container notebook
2022-09-21T21:16:03Z [Normal] Started container notebook
2022-09-21T21:16:04Z [Warning] Back-off restarting failed container

How should be error on mount error?

Screenshot 2022-09-22 at 00.21.39
Screenshot 2022-09-22 at 00.21.391920×1108 88.5 KB 

 scipy-jhub-values-kind.yaml:

# imagePullSecret is configuration to create a k8s Secret that Helm chart's pods
# can get credentials from to pull their images.
imagePullSecret:
  create: true
  automaticReferenceInjection: true
  registry: docker-registry 
  username: ## <<Change to your User Name>>
  email: ## <<Change to your Email>>
  password: ## <<Change to your Pass>>
hub:
  service:
    type: ClusterIP
  deploymentStrategy:
    type: Recreate
  db:
    type: sqlite-pvc
    upgrade:
    pvc:
      accessModes:
        - ReadWriteOnce
      storage: 3Gi
      storageClassName: standard  ## <Change to your Storage Class>
singleuser:
  image:
    name: miykael/nipype_tutorial
    tag: latest
  memory:
    limit: 3G
    guarantee: 2.5G
  defaultUrl: "/lab"
  storage:
    type: dynamic
    extraVolumes:
      - name: local-mount
        hostPath:
          path: /home/myname/neuroml_course_data
    extraVolumeMounts:
      - name: local-mount
        mountPath: /data  # where each user can reach the shared data
        readOnly: true

proxy:
  service:
    type: ClusterIP
  secretToken: d292ed1a5f212fb2e31fca0a4a34257932d67c5384150f7d94d1538bfc42b1c8


kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "127.0.0.1"
  apiServerPort: 6443
nodes:
- role: control-plane
- role: worker
  extraMounts:
    - hostPath: /home/myname/neuroml_course_data
      containerPath: /data
- role: worker
  extraMounts:
    - hostPath: /home/myname/neuroml_course_data
      containerPath: /data
2
  • Does this work if you disable the extra volumes?
  • Can you turn on debug logging and show us the hub logs?
3

Hi @manics,

  1. No, I found out it didn’t work without mount, I used miykael/nipype_tutorial docker images and there was an error with entrypoint.
  2. However, I changed image to jupyter/scipy-notebook:latest, now jupyterhub is working creating spawning pod, but mounted /data directory is empty inside container.
  3. I added
    'debug:
    enabled: true
    ’
    but can’t see where the should I see extended log?

output of kubectl describe pod :

Volumes:
  volume-myuser:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  claim-myuser
    ReadOnly:   false
  local-mount:
    Type:          HostPath (bare host directory volume)
    Path:          /home/myuser/neuroml_course_data
    HostPathType:
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       hub.jupyter.org/dedicated=user:NoSchedule
                   hub.jupyter.org_dedicated=user:NoSchedule
                   node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
4

You should be able to fetch the hub logs using kubectl logs deploy/hub

How many nodes do you have in your k8s cluster? Host volumes aren’t shared across nodes, so if you’ve copied files to the directory in one node they won’t appear on other nodes.

5

I have created 2 nodes in cluster with kind, according this tutorial Data Science with Python & JupyterHub on Kubernetes - Part 1 | VMware Tanzu Developer Center. But added extraMounts there:

nodes:
- role: control-plane
- role: worker
  extraMounts:
    - hostPath: /home/yarkin/neuroml_course_data
      containerPath: /data
- role: worker
  extraMounts:
    - hostPath: /home/yarkin/neuroml_course_data
      containerPath: /data

Hub-logs with kubectl logs of hub:
https://drive.google.com/file/d/1cIcA0V00_975DG9oizmZRY-bbPH35JI5/view?usp=sharing

Should I create additional volumeclaim to assign access to every spawned pod?

Thanks

6

I’m not sure what you mean by this- could you explain exactly what you’re thinking?

In general I think you’re better off looking for a shared storage solution that works across your nodes, e.g. NFS.

7

Im thinking of adding new persistantVolumeClaim to deployment file, from the documentation:

Should it do the work?