I’m using Jupyterhub in k8s cluster,
We would like to optimize the time that the user are waiting for new pod,
I’m struggling a bit with prePuller hook and with userPlaceholder:
- prePuller configuration:
prePuller: continuous: enabled: true hook: enabled: true serviceAccount: create: true
It seems the pod hook-image-awaiter has been created successfully however the logs indicates that it can’t perform a get request to the daemomst hook-image-puller:
2023/11/13 21:11:43 [DEBUG] GET https://kubernetes.default.svc:443/apis/apps/v1/namespaces/dev03/daemonsets/hook-image-puller: retrying in 10s (4 left)
I’ve checked that daemon set is configured:
kubectl get daemonsets (base) ⎈ NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE continuous-image-puller 0 0 0 0 0 spotinst.io/node-lifecycle=od 34h hook-image-puller 0 0 0 0 0 spotinst.io/node-lifecycle=od 11h
and also that the service account hook-image-awaiter was created.
what could be the cause?
- userPlaceholder configuration:
userPlaceholder: # Specify three dummy user pods will be used as placeholders enabled: true replicas: 3 userPods: nodeAffinity: matchNodePurpose: require
the pods are stuck in pending:
with this error:
Warning FailedScheduling 46s (x21 over 110m) default-scheduler (combined from similar events): 0/16 nodes are available: 16 node(s) didn't match Pod's node affinity/selector. preemption: 0/16 nodes are available: 16 Preemption is not helpful for scheduling.
also attaching the pod spec:
Priority: 500 Priority Class Name: medium-priority Node: <none> Labels: app=jupyterhub component=user-placeholder controller-revision-hash=user-placeholder-79d5467d7c release=jupyterhub statefulset.kubernetes.io/pod-name=user-placeholder-0 Annotations: kubernetes.io/psp: eks.privileged Status: Pending IP: IPs: <none> Controlled By: StatefulSet/user-placeholder Containers: pause: Image: registry.k8s.io/pause:3.9 Port: <none> Host Port: <none> Requests: memory: 1G Environment: <none> Mounts: <none> Conditions: Type Status PodScheduled False Volumes: <none> QoS Class: Burstable Node-Selectors: spotinst.io/node-lifecycle=od Tolerations: hub.jupyter.org/dedicated=user:NoSchedule hub.jupyter.org_dedicated=user:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s