Hook-image-puller fails + user-placeholder stuck in pending

JupyterHub
,
1

Hello everyone,

I’m using Jupyterhub in k8s cluster,
We would like to optimize the time that the user are waiting for new pod,
I’m struggling a bit with prePuller hook and with userPlaceholder:

  1. prePuller configuration:
 prePuller:
    continuous:
      enabled: true
    hook:
      enabled: true
      serviceAccount:
        create: true

It seems the pod hook-image-awaiter has been created successfully however the logs indicates that it can’t perform a get request to the daemomst hook-image-puller:

2023/11/13 21:11:43 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/dev03/daemonsets/hook-image-puller: retrying in 10s (4 left)

I’ve checked that daemon set is configured:

 kubectl get daemonsets                                                                         (base)  ⎈ 
NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
continuous-image-puller   0         0         0       0            0           spotinst.io/node-lifecycle=od   34h
hook-image-puller         0         0         0       0            0           spotinst.io/node-lifecycle=od   11h

and also that the service account hook-image-awaiter was created.
what could be the cause?

  1. userPlaceholder configuration:
    userPlaceholder:
      # Specify three dummy user pods will be used as placeholders
      enabled: true
      replicas: 3
    userPods:
      nodeAffinity:
        matchNodePurpose: require

the pods are stuck in pending:
with this error:

  Warning  FailedScheduling  46s (x21 over 110m)  default-scheduler  (combined from similar events): 0/16 nodes are available: 16 node(s) didn't match Pod's node affinity/selector. preemption: 0/16 nodes are available: 16 Preemption is not helpful for scheduling.

also attaching the pod spec:

Priority:             500
Priority Class Name:  medium-priority
Node:                 <none>
Labels:               app=jupyterhub
                      component=user-placeholder
                      controller-revision-hash=user-placeholder-79d5467d7c
                      release=jupyterhub
                      statefulset.kubernetes.io/pod-name=user-placeholder-0
Annotations:          kubernetes.io/psp: eks.privileged
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/user-placeholder
Containers:
  pause:
    Image:      registry.k8s.io/pause:3.9
    Port:       <none>
    Host Port:  <none>
    Requests:
      memory:     1G
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type           Status
  PodScheduled   False
Volumes:         <none>
QoS Class:       Burstable
Node-Selectors:  spotinst.io/node-lifecycle=od
Tolerations:     hub.jupyter.org/dedicated=user:NoSchedule
                 hub.jupyter.org_dedicated=user:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Thanks!!