Placeholder user Preemption fails status.hostIPs[0].ip

gshand · March 18, 2024, 9:21pm

I’m running a JH deployment on AWS EKS (k8s v1.29, JH helm chart v3.2.1, in a private VPC).

I’ve implemented user placeholders for efficient autoscaling as described in Optimizations — Zero to JupyterHub with Kubernetes documentation.

However, when spawning a new user server that should trigger preemption and take one of the slots allocated to a placeholder user, the scheduler instead fails to schedule the pod causing the cluster to scale up.

I see the following error in the scheduler logs

E0318 20:52:53.050975       1 preemption.go:365] "Preparing pod preemption" err="Pod \"user-placeholder-2\" is invalid: status.hostIPs[0].ip: Invalid value: \"<NODE_IP>\": must be equal to `hostIP`" pod="jupyterhub/user-placeholder-2" preemptor="jupyterhub/jupyter-gshand"

Anyone know where else I could look to troubleshoot this further?

gshand · March 19, 2024, 12:57pm

Seems to be hitting this error

github.com

kubernetes-csi/csi-driver-smb/blob/16ac78df8dfa2a5bada28ed34f1121e880aa75c4/vendor/k8s.io/kubernetes/pkg/apis/core/validation/validation.go#L3897


      
          	allErrs := field.ErrorList{}
          
          	if len(pod.Status.HostIPs) == 0 {
          		return allErrs
          	}
          
          	hostIPsField := field.NewPath("status", "hostIPs")
          
          	// hostIP must be equal to hostIPs[0].IP
          	if pod.Status.HostIP != pod.Status.HostIPs[0].IP {
          		allErrs = append(allErrs, field.Invalid(hostIPsField.Index(0).Child("ip"), pod.Status.HostIPs[0].IP, "must be equal to `hostIP`"))
          	}
          
          	// all HostPs must be valid IPs
          	for i, hostIP := range pod.Status.HostIPs {
          		for _, msg := range validation.IsValidIP(hostIP.IP) {
          			allErrs = append(allErrs, field.Invalid(hostIPsField.Index(i), hostIP.IP, msg))
          		}
          	}
          
          	// if we have more than one Pod.HostIP then

Topic		Replies	Views
Trouble understanding behavior of autoscaling support Zero to JupyterHub on Kubernetes	3	713	June 5, 2019
Hook-image-puller fails + user-placeholder stuck in pending JupyterHub jupyterhub , help-wanted	0	187	November 14, 2023
userPlaceholder pods stuck Pending in zero-to-jupyterhub v0.8.2 JupyterHub	4	504	April 3, 2020
User's pod with pending state in JupyterHub JupyterHub jupyterhub , help-wanted	2	93	April 10, 2024
User placeholder deployment fails with not enough resources Zero to JupyterHub on Kubernetes how-to , help-wanted	0	783	November 12, 2020

Placeholder user Preemption fails status.hostIPs[0].ip

Related Topics