1 node(s) did not find available persistent volumes to bind

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Hello Community,

For days I have been getting the following “error message” when a user wants to start a kernel/image.

[Warning] 0/1 nodes are available: 1 node(s) did not find available persistent volumes to bind.

The Pod for the user remains in “PENDING” status.
The PVC for the user shows two warnings:

  • waiting for pod jupyter-admin to be scheduled.
  • waiting for first consumer to be created before binding.

I have read countless posts about this, but none have helped or really fit the situation.

I have installed jupyterhub and kubernetes on a single virtual machine. This means that the master node is also the woker node. The storage that can be used is therefore only the storage of the VM. For a short time everything ran smoothly until suddenly the warning appeared and it no longer worked. Maybe I have a huge misunderstanding somewhere, but at the moment I am completely at a loss. I hope that you can help me somehow.

Here are the configurations that are used:
Upgrade-Command:

helm upgrade --cleanup-on-fail --install jupyterhub jupyterhub/jupyterhub --namespace jhub --create-namespace --version=1.1.3-n254.h9b546a54 --timeout=10m --values config.yaml

config.yaml

debug:
  enabled: true

proxy:
  secretToken: *****

hub:
  cookieSecret: *****
  config:
    Authenticator:
      admin_users:
        - admin
        - user
    DummyAuthenticator:
      password: *****
    JupyterHub:
      admin_access: false
      authenticator_class: dummy

singleuser:
  cpu:
    limit: 1
    guarantee: 0.05
  memory:
    limit: 1G
    guarantee: 512M
  storage:
    capacity: 1Gi
    homeMountPath: /home/jhub
    dynamic:
      storageClass: local-storage
  defaultUrl: "/lab"
  extraEnv:
    JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp"
  image:
    name: *****
    tag: *****
  lifecycleHooks:
    postStart:
      exec:
        command:
          [
            "gitpuller",
            "*****",
            "main",
            "Examples",
          ]
  profileList:
    - display_name: "Minimal environment"
      description: "To avoid too much bells and whistles: Python."
      default: true
    *****

cull:
  enabled: true
  timeout: 1800
  every: 300

prePuller:
  continuous:
    enabled: true

scheduling:
  userScheduler:
    enabled: true
  podPriority:
    enabled: true

storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jhub-pv
  labels:
    type: local
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/jhub
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - jupyterhub-test

I can make the following statements about the system.

  • The name of the node is jupyterhub-test.
  • The PVC hub-db-dir has the status Bound.
  • The PV jhub-pv has the status Bound.
2

According to Storage Classes | Kubernetes

Local volumes do not currently support dynamic provisioning

Can you show us the output of kubectl get pv,pvc?

3

pv_pvc
pv_pvc776×97 28.4 KB

How can I make it possible to use the local storage if dynamic provisioning is not supported? I had not found any other examples than the configs I use.

4

Storage provisioning is managed by your K8s cluster, not Z2JH. I think you’ll have to manually create your persistent volumes for each user.

Alternatively you could look at a dynamic volume provisioner that works with local volumes, for example K3s includes one by default: Rancher Docs: Volumes and Storage

5

Thanks, I had to install the “local path provisioner” from rancher. I also changed homeMountPath to the default /home/jovyan and storageClass to local-path.
Now I just need to get it to create a separate PVC/PV for each profile, but I think and hope I can manage that.
Thanks a lot for your help @manics.