Cannot spawn on the master kubernetes node

JupyterHub Zero to JupyterHub on Kubernetes
1

Dear all,

I would like to thank you for the software you have produced and maintained.

I have a “bare-metal” cluster I set up with the jupyterhub guide using microk8s. It consists of two nodes: thinkpad-node (master node, the jhub is running on the node) and gpu-server (a computer with GPU).

I created the following profileList:

  profileList:
    - display_name: "thinkpad-node"
      description: "thinkpad-node running on the laptop"
      default: true
      kubespawner_override:
        image: ghcr.io/mbi-div-b/mbi-div-b-notebook-base
        node_selector: {'node-role.kubernetes.io/cpu': 'cpu'}
    - display_name: "gpu-server"
      description: "gpu-node running on the real computer in the rack"
      kubespawner_override:
        image: ghcr.io/mbi-div-b/mbi-div-b-notebook-cuda
        #node_selector: {'name': 'gpu-server'}
        node_selector: {'node-role.kubernetes.io/gpu': 'gpu'}

where both nodes have the following labels. The gpu-server profile works as expected but the thinkpad-node profile cannot be spawned with the error:

2023-06-28T12:59:24.169862Z [Warning] 0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..

I suspected that the problem is the restriction of NoSchedule on the master node but I have removed it. I thought that maybe this is a general issue of my k8s setup but I was able to create and assign a pause (kinda dummy/hello-world) pod on the “thinkpad-node” with the following yaml config:

apiVersion: v1
kind: Pod
metadata:
  name: pause-thinkpad
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause:2.0
  nodeSelector:
    node-role.kubernetes.io/cpu : cpu

Here are the descriptions of the both nodes:

lunin@thinkpad-node:~/k8s-jhub-configs$ microk8s kubectl describe node thinkpad-node
Name:               thinkpad-node
Roles:              cpu
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    feature.node.kubernetes.io/cpu-cpuid.ADX=true
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                    feature.node.kubernetes.io/cpu-cpuid.IBPB=true
                    feature.node.kubernetes.io/cpu-cpuid.MPX=true
                    feature.node.kubernetes.io/cpu-cpuid.RTM_ALWAYS_ABORT=true
                    feature.node.kubernetes.io/cpu-cpuid.STIBP=true
                    feature.node.kubernetes.io/cpu-cpuid.VMX=true
                    feature.node.kubernetes.io/cpu-cstate.enabled=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=true
                    feature.node.kubernetes.io/cpu-pstate.scaling_governor=powersave
                    feature.node.kubernetes.io/cpu-pstate.status=active
                    feature.node.kubernetes.io/cpu-pstate.turbo=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-75-generic
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/pci-8086.present=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=ubuntu
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=22
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
                    feature.node.kubernetes.io/usb-ef_04f2_b604.present=true
                    feature.node.kubernetes.io/usb-ff_06cb_009a.present=true
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=thinkpad-node
                    kubernetes.io/os=linux
                    microk8s.io/cluster=true
                    name=thinkpad-node
                    node-role.kubernetes.io/cpu=cpu
                    node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
                    topology.cstor.openebs.io/nodeName=thinkpad-node
                    topology.jiva.openebs.io/nodeName=thinkpad-node
Annotations:        csi.volume.kubernetes.io/nodeid: {"cstor.csi.openebs.io":"thinkpad-node","jiva.csi.openebs.io":"thinkpad-node"}
                    nfd.node.kubernetes.io/extended-resources: 
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.FMA3,cpu-cpuid.IBPB,cpu-cpuid.MPX,cpu-cpuid.RTM_ALWAYS_ABORT,cpu-cpui...
                    nfd.node.kubernetes.io/master.version: v0.10.1
                    nfd.node.kubernetes.io/worker.version: v0.10.1
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.6.17.62/22
                    projectcalico.org/IPv4VXLANTunnelAddr: 10.1.102.192
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 20 Jun 2023 12:18:43 +0000
Taints:             <none>
Unschedulable:      false

lunin@thinkpad-node:~/k8s-jhub-configs$ microk8s kubectl describe node gpu-server
Name:               gpu-server
Roles:              gpu
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVXSLOW=true
                    feature.node.kubernetes.io/cpu-cpuid.CPBOOST=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA4=true
                    feature.node.kubernetes.io/cpu-cpuid.IBS=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSBRNTRGT=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSFETCHSAM=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSFFV=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSOPCNT=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSOPCNTEXT=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSOPSAM=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSRDWROPCNT=true
                    feature.node.kubernetes.io/cpu-cpuid.IBSRIPINVALIDCHK=true
                    feature.node.kubernetes.io/cpu-cpuid.SSE4A=true
                    feature.node.kubernetes.io/cpu-cpuid.XOP=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-75-generic
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/pci-10de.present=true
                    feature.node.kubernetes.io/pci-10ec.present=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=ubuntu
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=22
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=gpu-server
                    kubernetes.io/os=linux
                    microk8s.io/cluster=true
                    name=gpu-server
                    node-role.kubernetes.io/gpu=gpu
                    node.kubernetes.io/microk8s-worker=microk8s-worker
                    nvidia.com/cuda.driver.major=515
                    nvidia.com/cuda.driver.minor=43
                    nvidia.com/cuda.driver.rev=04
                    nvidia.com/cuda.runtime.major=11
                    nvidia.com/cuda.runtime.minor=7
                    nvidia.com/gfd.timestamp=1687266579
                    nvidia.com/gpu.compute.major=7
                    nvidia.com/gpu.compute.minor=5
                    nvidia.com/gpu.count=1
                    nvidia.com/gpu.deploy.container-toolkit=true
                    nvidia.com/gpu.deploy.dcgm=true
                    nvidia.com/gpu.deploy.dcgm-exporter=true
                    nvidia.com/gpu.deploy.device-plugin=true
                    nvidia.com/gpu.deploy.driver=pre-installed
                    nvidia.com/gpu.deploy.gpu-feature-discovery=true
                    nvidia.com/gpu.deploy.node-status-exporter=true
                    nvidia.com/gpu.deploy.operator-validator=true
                    nvidia.com/gpu.family=turing
                    nvidia.com/gpu.machine=System-Product-Name
                    nvidia.com/gpu.memory=8192
                    nvidia.com/gpu.present=true
                    nvidia.com/gpu.product=NVIDIA-GeForce-RTX-2070
                    nvidia.com/gpu.replicas=1
                    nvidia.com/mig.capable=false
                    nvidia.com/mig.strategy=single
                    topology.cstor.openebs.io/nodeName=gpu-server
                    topology.jiva.openebs.io/nodeName=gpu-server
Annotations:        csi.volume.kubernetes.io/nodeid: {"cstor.csi.openebs.io":"gpu-server","jiva.csi.openebs.io":"gpu-server"}
                    nfd.node.kubernetes.io/extended-resources: 
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVXSLOW,cpu-cpuid.CPBOOST,cpu-cpuid.FMA4,cpu-cpuid.IBS,cpu-cpuid.IBSBRNTRGT,cpu-cpuid.IBSFETCHSAM,...
                    nfd.node.kubernetes.io/worker.version: v0.10.1
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.6.5.53/24
                    projectcalico.org/IPv4VXLANTunnelAddr: 10.1.246.192
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 20 Jun 2023 12:22:54 +0000
Taints:             <none>
Unschedulable:      false
2

Here is a description of the failed jupyter pod. I guess there is no additional information.

lunin@thinkpad-node:~/k8s-jhub-configs$ microk8s kubectl describe pod jupyter-lunin -n k8s-jhub-test
Name:             jupyter-lunin
Namespace:        k8s-jhub-test
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=jupyterhub
                  chart=jupyterhub-2.0.0
                  component=singleuser-server
                  heritage=jupyterhub
                  hub.jupyter.org/network-access-hub=true
                  hub.jupyter.org/servername=
                  hub.jupyter.org/username=lunin
                  release=helm-release-test
Annotations:      hub.jupyter.org/username: lunin
Status:           Pending
IP:               
IPs:              <none>
Init Containers:
  block-cloud-metadata:
    Image:      jupyterhub/k8s-network-tools:2.0.0
    Port:       <none>
    Host Port:  <none>
    Command:
      iptables
      -A
      OUTPUT
      -d
      169.254.169.254
      -j
      DROP
    Environment:  <none>
    Mounts:       <none>
Containers:
  notebook:
    Image:      ghcr.io/mbi-div-b/mbi-div-b-notebook-base
    Port:       8888/TCP
    Host Port:  0/TCP
    Requests:
      memory:  1073741824
    Environment:
      GRANT_SUDO:                              yes
      JPY_API_TOKEN:                           557e357ab27e46bab82bb451c73de2d5
      JUPYTERHUB_ACTIVITY_URL:                 http://hub:8081/hub/api/users/lunin/activity
      JUPYTERHUB_ADMIN_ACCESS:                 1
      JUPYTERHUB_API_TOKEN:                    557e357ab27e46bab82bb451c73de2d5
      JUPYTERHUB_API_URL:                      http://hub:8081/hub/api
      JUPYTERHUB_BASE_URL:                     /
      JUPYTERHUB_CLIENT_ID:                    jupyterhub-user-lunin
      JUPYTERHUB_HOST:                         
      JUPYTERHUB_OAUTH_ACCESS_SCOPES:          ["access:servers!server=lunin/", "access:servers!user=lunin"]
      JUPYTERHUB_OAUTH_CALLBACK_URL:           /user/lunin/oauth_callback
      JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES:  []
      JUPYTERHUB_OAUTH_SCOPES:                 ["access:servers!server=lunin/", "access:servers!user=lunin"]
      JUPYTERHUB_SERVER_NAME:                  
      JUPYTERHUB_SERVICE_PREFIX:               /user/lunin/
      JUPYTERHUB_SERVICE_URL:                  http://0.0.0.0:8888/user/lunin/
      JUPYTERHUB_USER:                         lunin
      JUPYTER_IMAGE:                           ghcr.io/mbi-div-b/mbi-div-b-notebook-base
      JUPYTER_IMAGE_SPEC:                      ghcr.io/mbi-div-b/mbi-div-b-notebook-base
      MEM_GUARANTEE:                           1073741824
      NB_USER:                                 lunin
    Mounts:
      /home/lunin from volume-lunin (rw)
      /home/shared from jupyterhub-shared (rw)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  volume-lunin:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  claim-lunin
    ReadOnly:   false
  jupyterhub-shared:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   pvc-shared
    ReadOnly:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/cpu=cpu
Tolerations:     hub.jupyter.org/dedicated=user:NoSchedule
                 hub.jupyter.org_dedicated=user:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  1s    default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..
3

I am also able to create the pod manually using the same image with the test_pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: lunin-jupyterhub-cpu
spec:
  containers:
  - name: lunin-jupyterhub
    image: ghcr.io/mbi-div-b/mbi-div-b-notebook-base
  nodeSelector:
    node-role.kubernetes.io/cpu : cpu

If the network would be an issue then the pod would fail later, right? Not with the FailedScheduling, isn’t it?

4

Is it possible you’re running out of resources on your node?

5

Thank you for quick response. As far as I can understand it should not be an issue. There are only 25 pods running on the thinkpad-node. I attach the full output of kubectl describe node here:

Name:               thinkpad-node
Roles:              cpu
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    feature.node.kubernetes.io/cpu-cpuid.ADX=true
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                    feature.node.kubernetes.io/cpu-cpuid.IBPB=true
                    feature.node.kubernetes.io/cpu-cpuid.MPX=true
                    feature.node.kubernetes.io/cpu-cpuid.RTM_ALWAYS_ABORT=true
                    feature.node.kubernetes.io/cpu-cpuid.STIBP=true
                    feature.node.kubernetes.io/cpu-cpuid.VMX=true
                    feature.node.kubernetes.io/cpu-cstate.enabled=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=true
                    feature.node.kubernetes.io/cpu-pstate.scaling_governor=powersave
                    feature.node.kubernetes.io/cpu-pstate.status=active
                    feature.node.kubernetes.io/cpu-pstate.turbo=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ=true
                    feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-75-generic
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/pci-8086.present=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=ubuntu
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=22
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
                    feature.node.kubernetes.io/usb-ef_04f2_b604.present=true
                    feature.node.kubernetes.io/usb-ff_06cb_009a.present=true
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=thinkpad-node
                    kubernetes.io/os=linux
                    microk8s.io/cluster=true
                    name=thinkpad-node
                    node-role.kubernetes.io/cpu=cpu
                    node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
                    topology.cstor.openebs.io/nodeName=thinkpad-node
                    topology.jiva.openebs.io/nodeName=thinkpad-node
Annotations:        csi.volume.kubernetes.io/nodeid: {"cstor.csi.openebs.io":"thinkpad-node","jiva.csi.openebs.io":"thinkpad-node"}
                    nfd.node.kubernetes.io/extended-resources: 
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.FMA3,cpu-cpuid.IBPB,cpu-cpuid.MPX,cpu-cpuid.RTM_ALWAYS_ABORT,cpu-cpui...
                    nfd.node.kubernetes.io/master.version: v0.10.1
                    nfd.node.kubernetes.io/worker.version: v0.10.1
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.6.17.62/22
                    projectcalico.org/IPv4VXLANTunnelAddr: 10.1.102.192
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 20 Jun 2023 12:18:43 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  thinkpad-node
  AcquireTime:     <unset>
  RenewTime:       Thu, 29 Jun 2023 08:09:42 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Wed, 28 Jun 2023 08:32:23 +0000   Wed, 28 Jun 2023 08:32:23 +0000   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Thu, 29 Jun 2023 08:09:36 +0000   Tue, 20 Jun 2023 12:18:43 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 29 Jun 2023 08:09:36 +0000   Tue, 20 Jun 2023 12:18:43 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 29 Jun 2023 08:09:36 +0000   Tue, 20 Jun 2023 12:18:43 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 29 Jun 2023 08:09:36 +0000   Wed, 28 Jun 2023 06:29:03 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  10.6.17.62
  Hostname:    thinkpad-node
Capacity:
  cpu:                8
  ephemeral-storage:  102626232Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16260504Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  101577656Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16158104Ki
  pods:               110
System Info:
  Machine ID:                 623b24d6e5134241862ba660ff3cd43e
  System UUID:                20013d4c-30da-11b2-a85c-fbce986fcb21
  Boot ID:                    4daf8d45-870a-44c5-b746-7dc8ac7a91fc
  Kernel Version:             5.15.0-75-generic
  OS Image:                   Ubuntu 22.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.15
  Kubelet Version:            v1.27.2
  Kube-Proxy Version:         v1.27.2
Non-terminated Pods:          (19 in total)
  Namespace                   Name                                                           CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                           ------------  ----------  ---------------  -------------  ---
  openebs                     openebs-ndm-wzpv8                                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-cstor-cvc-operator-56fdf4cf9-xp96l                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-cstor-cspc-operator-6b44df76f4-6bpnx                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-cstor-csi-controller-0                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-jiva-operator-747cf78485-vlm65                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  kube-system                 coredns-7745f9f87f-cf8b9                                       100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     8d
  k8s-jhub-test               proxy-5cc8bd7b5f-qp7gv                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  kube-system                 calico-kube-controllers-6c99c8747f-ggg2b                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  kube-system                 calico-node-g9bgv                                              250m (3%)     0 (0%)      0 (0%)           0 (0%)         8d
  gpu-operator-resources      gpu-operator-node-feature-discovery-master-6fbc745786-kgt2x    0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  gpu-operator-resources      gpu-operator-76998cd846-nhbvw                                  200m (2%)     500m (6%)   100Mi (0%)       350Mi (2%)     8d
  metallb-system              controller-8467d88d69-tmfj5                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  metallb-system              speaker-fztmq                                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-ndm-operator-5c4d8c948c-qp7pt                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-cstor-csi-node-hqhkg                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  openebs                     openebs-jiva-csi-node-mflck                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  gpu-operator-resources      gpu-operator-node-feature-discovery-worker-46qt5               0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  k8s-jhub-test               hub-b9fb8b7df-srpq5                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         19h
  k8s-jhub-test               continuous-image-puller-tbbzl                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         19h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                550m (6%)   500m (6%)
  memory             170Mi (1%)  520Mi (3%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

I see the limits below but they have nothing to do with the problem, right?

Can I somehow find exact the command which kubespawner uses to create a pod to try to debug it?

6

I tried to apply the minimal requirements for the jupyterlab pod with

singleuser:
  memory:
    guarantee: 1G
  cpu:
    guarantee: 0.5

and still cannot run it on the desired node.

7

Maybe it’s something to do with your microk8s configuration? I’ve used K3S successfully, from what I remember microk8s has been more difficult to setup in the past.

Can I somehow find exact the command which kubespawner uses to create a pod to try to debug it?

KubeSpawner calls the K8s API with a pod definition. You can obtain this using kubectl get pod <pod-name> -o yaml, then edit it to remove the autogenerated K8s fields.