Placeholder user Preemption fails status.hostIPs[0].ip

gshand · March 18, 2024, 9:21pm

I’m running a JH deployment on AWS EKS (k8s v1.29, JH helm chart v3.2.1, in a private VPC).

I’ve implemented user placeholders for efficient autoscaling as described in Optimizations — Zero to JupyterHub with Kubernetes documentation.

However, when spawning a new user server that should trigger preemption and take one of the slots allocated to a placeholder user, the scheduler instead fails to schedule the pod causing the cluster to scale up.

I see the following error in the scheduler logs

E0318 20:52:53.050975       1 preemption.go:365] "Preparing pod preemption" err="Pod \"user-placeholder-2\" is invalid: status.hostIPs[0].ip: Invalid value: \"<NODE_IP>\": must be equal to `hostIP`" pod="jupyterhub/user-placeholder-2" preemptor="jupyterhub/jupyter-gshand"

Anyone know where else I could look to troubleshoot this further?

gshand · March 19, 2024, 12:57pm

Seems to be hitting this error

github.com

kubernetes-csi/csi-driver-smb/blob/16ac78df8dfa2a5bada28ed34f1121e880aa75c4/vendor/k8s.io/kubernetes/pkg/apis/core/validation/validation.go#L3897


      
          	allErrs := field.ErrorList{}
          
          	if len(pod.Status.HostIPs) == 0 {
          		return allErrs
          	}
          
          	hostIPsField := field.NewPath("status", "hostIPs")
          
          	// hostIP must be equal to hostIPs[0].IP
          	if pod.Status.HostIP != pod.Status.HostIPs[0].IP {
          		allErrs = append(allErrs, field.Invalid(hostIPsField.Index(0).Child("ip"), pod.Status.HostIPs[0].IP, "must be equal to `hostIP`"))
          	}
          
          	// all HostPs must be valid IPs
          	for i, hostIP := range pod.Status.HostIPs {
          		for _, msg := range validation.IsValidIP(hostIP.IP) {
          			allErrs = append(allErrs, field.Invalid(hostIPsField.Index(i), hostIP.IP, msg))
          		}
          	}
          
          	// if we have more than one Pod.HostIP then

Arthur_Rio · September 9, 2024, 5:25pm

We are hitting the same error, any luck finding a workaround?

Edit: This will be fixed on 4+ as the kube-scheduler version will be 1.28.13. That version includes the following fix: Pod IP temporarily removed from status when pod transitions to a terminal state · Issue #125370 · kubernetes/kubernetes · GitHub

Our current workaround is to pull the helm chart manually, and make the following changes:

jupyterhub/templates/scheduling/user-scheduler/configmap.yaml: change endpointsleases to leases
jupyterhub/values.yaml: Change userScheduler image tag from 1.26.11 to 1.28.13

Topic		Replies	Views
Trouble understanding behavior of autoscaling support Zero to JupyterHub on Kubernetes	3	762	June 5, 2019
userPlaceholder pods stuck Pending in zero-to-jupyterhub v0.8.2 JupyterHub	4	568	April 3, 2020
Hook-image-puller fails + user-placeholder stuck in pending JupyterHub jupyterhub , help-wanted	0	318	November 14, 2023
Common userPlaceholder / podPriority configuration for multiple Z2JH instances Zero to JupyterHub on Kubernetes help-wanted	1	11	November 12, 2024
GKE autoscale test failure, cluster RECONCILING Zero to JupyterHub on Kubernetes	7	1976	January 30, 2021

Placeholder user Preemption fails status.hostIPs[0].ip

Related topics