From zero-to-jupyterhub-k8s/blob/main/jupyterhub/templates/scheduling/user-scheduler/configmap.yaml
…
config.yaml: |
{{- if ge (atoi .Capabilities.KubeVersion.Minor) 21 }}
…
{{- else }}
apiVersion: kubescheduler.config.k8sio/v1beta1
kind: KubeSchedulerConfiguration
This is my first zero-to-jupyterhub-k8s deployment: k8s v1.23.4, helm v3.8.1, rhel 7 on three physical servers, CSI: spectrum scale 5.1.2.2
For some reason, the configmap user-scheduler has
apiVersion: kubescheduler.config.k8s.io/v1beta1
and the user-scheduler pods failed with msg:
"no kind “KubeSchedulerConfiguration” is registered for version “kubescheduler.config.k8sio/v1beta1” in scheme “k8sio/kubernetes/pkg/scheduler/apis/config/scheme/scheme.go:30”
When I change apiVersion from v1beta1 to v1beta3 via
kubectl --namespace=k8s-jhub edit deployment.apps/user-scheduler
the two user-scheduler pods are running but not ready, giving CSI error msg:
E0330 22:17:28.042199 1 reflector.go:138] k8sio/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: csistoragecapacities.storage.k8sio is forbidden: User “system:serviceaccount:k8s-jhub:user-scheduler” cannot list resource “csistoragecapacities” in API group “storage.k8sio” at the cluster scope
…
E0330 22:17:28.042520 1 reflector.go:138] k8sio/client-go/informers/factory.go:134: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User “system:serviceaccount:k8s-jhub:user-scheduler” cannot list resource “namespaces” in API group “” at the cluster scope
…
E0330 22:17:28.042804 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8sio is forbidden: User “system:serviceaccount:k8s-jhub:user-scheduler” cannot list resource “csidrivers” in API group “storage.k8sio” at the cluster scope
…
E0330 22:17:28.048257 1 plugin.go:138] “getting namespace, assuming empty set of namespace labels” err=“namespace “kube-system” not found” namespace=“kube-system”
…
E0330 22:17:28.048338 1 plugin.go:138] “getting namespace, assuming empty set of namespace labels” err=“namespace “ibm-spectrum-scale-csi-driver” not found” namespace=“ibm-spectrum-scale-csi-driver”
…
E0330 22:17:28.048391 1 plugin.go:138] “getting namespace, assuming empty set of namespace labels” err=“namespace “k8s-jhub” not found” namespace=“k8s-jhub”
…
Even though rbac.yaml has all those csi*, and k8s has all the namespaces, the user-scheduler pods fail.
$ kubectl get namespace
NAME STATUS AGE
default Active 21d
ibm-spectrum-scale-csi-driver Active 20d
k8s-jhub Active 5d7h
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
kubernetes-dashboard Active 21d