Hello, I’ve been trying to upgrade helmchart from 2.0.0 to the latest 3.1.0 (speaking as of date 8 Nov. 2023) on k8s (v1.25) AWS cluster. I got this error message when I check the logs of helmrelease
Last Helm logs:
Patch Deployment "hub" in namespace jupyterhub
Patch Deployment "proxy" in namespace jupyterhub
Patch Deployment "user-scheduler" in namespace jupyterhub
Patch Ingress "jupyterhub" in namespace jupyterhub
warning: Upgrade "jupyterhub" failed: cannot patch "jupyterhub-user-scheduler" with kind ClusterRole: clusterroles.rbac.authorization.k8s.io "jupyterhub-user-scheduler" is forbidden: user "system:serviceaccount:jupyterhub:flux-jupyterhub" (groups=["system:serviceaccounts" "system:serviceaccounts:jupyterhub" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["bindings"], Verbs:["create"]}
{APIGroups:[""], Resources:["endpoints"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:[""], Resources:["endpoints"], Verbs:["create"]}
{APIGroups:[""], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:[""], Resources:["namespaces"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["nodes"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["pods"], Verbs:["delete" "get" "list" "watch"]}
{APIGroups:[""], Resources:["pods/binding"], Verbs:["create"]}
{APIGroups:[""], Resources:["pods/status"], Verbs:["patch" "update"]}
{APIGroups:[""], Resources:["replicationcontrollers"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["statefulsets"], Verbs:["get" "list" "watch"]}
{APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]}
{APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["create"]}
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:["extensions"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["policy"], Resources:["poddisruptionbudgets"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csidrivers"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csinodes"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csistoragecapacities"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["storageclasses"], Verbs:["get" "list" "watch"]} && cannot patch "jupyterhub-user-scheduler" with kind ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io "jupyterhub-user-scheduler" is forbidden: user "system:serviceaccount:jupyterhub:flux-jupyterhub" (groups=["system:serviceaccounts" "system:serviceaccounts:jupyterhub" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["bindings"], Verbs:["create"]}
{APIGroups:[""], Resources:["endpoints"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:[""], Resources:["endpoints"], Verbs:["create"]}
{APIGroups:[""], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:[""], Resources:["namespaces"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["nodes"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["pods"], Verbs:["delete" "get" "list" "watch"]}
{APIGroups:[""], Resources:["pods/binding"], Verbs:["create"]}
{APIGroups:[""], Resources:["pods/status"], Verbs:["patch" "update"]}
{APIGroups:[""], Resources:["replicationcontrollers"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["statefulsets"], Verbs:["get" "list" "watch"]}
{APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]}
{APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["create"]}
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:["extensions"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["policy"], Resources:["poddisruptionbudgets"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csidrivers"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csinodes"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csistoragecapacities"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["storageclasses"], Verbs:["get" "list" "watch"]}
Reason: UpgradeFailed
Status: False
Type: Released
Failures: 6
Helm Chart: jupyterhub/jupyterhub-jupyterhub
Last Applied Revision: 2.0.0
Last Attempted Revision: 3.1.0
Last Attempted Values Checksum: 4b29d2e274a752ba61102614c7f324e2ef814c65
Last Release Revision: 19
Observed Generation: 26
Upgrade Failures: 1
% k describe clusterrole jupyterhub-user-scheduler
Name: jupyterhub-user-scheduler
Labels: app=jupyterhub
app.kubernetes.io/managed-by=Helm
chart=jupyterhub-2.0.0
component=user-scheduler
helm.toolkit.fluxcd.io/name=jupyterhub
helm.toolkit.fluxcd.io/namespace=jupyterhub
heritage=Helm
release=jupyterhub
Annotations: meta.helm.sh/release-name: jupyterhub
meta.helm.sh/release-namespace: jupyterhub
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch update]
events.events.k8s.io [] [] [create patch update]
bindings [] [] [create]
endpoints [] [] [create]
pods/binding [] [] [create]
tokenreviews.authentication.k8s.io [] [] [create]
subjectaccessreviews.authorization.k8s.io [] [] [create]
leases.coordination.k8s.io [] [] [create]
pods [] [] [delete get list watch]
persistentvolumeclaims [] [] [get list watch patch update]
persistentvolumes [] [] [get list watch patch update]
namespaces [] [] [get list watch]
nodes [] [] [get list watch]
replicationcontrollers [] [] [get list watch]
services [] [] [get list watch]
replicasets.apps [] [] [get list watch]
statefulsets.apps [] [] [get list watch]
replicasets.extensions [] [] [get list watch]
poddisruptionbudgets.policy [] [] [get list watch]
csidrivers.storage.k8s.io [] [] [get list watch]
csinodes.storage.k8s.io [] [] [get list watch]
csistoragecapacities.storage.k8s.io [] [] [get list watch]
storageclasses.storage.k8s.io [] [] [get list watch]
endpoints [] [user-scheduler-lock] [get update]
leases.coordination.k8s.io [] [user-scheduler-lock] [get update]
pods/status [] [] [patch update]
Failing pod was only “hub” one. Pod messages:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 57s default-scheduler Successfully assigned jupyterhub/hub-6b48cc8878-vllr4 to ip-xxxxxxx.ec2.internal
Normal SuccessfulAttachVolume 47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-69b7208b-702c-45f2-a937-adcd565a71b2"
Normal Pulling 46s kubelet Pulling image "jupyterhub/k8s-hub:3.0.0"
Normal Pulled 38s kubelet Successfully pulled image "jupyterhub/k8s-hub:3.0.0" in 7.490308961s (7.490318101s including waiting)
Normal Created 26s (x2 over 38s) kubelet Created container hub
Normal Started 26s (x2 over 38s) kubelet Started container hub
Normal Pulled 26s kubelet Container image "jupyterhub/k8s-hub:3.0.0" already present on machine
Warning Unhealthy 17s (x16 over 38s) kubelet Readiness probe failed: Get "http://10.100.10.20:8081/hub/health": dial tcp 10.100.10.20:8081: connect: connection refused
Warning BackOff 7s (x2 over 15s) kubelet Back-off restarting failed container
I tried to update helmchart “only” to the major release version “3.0.0”, but the same behaviour. It looks some more priviledges are needed inside of the kubernetes, but I haven’t found anything mentioned in the release notes. Did I miss something?
Rolled back the helmchart to previous working version (2.0.0) and everything is running fine.
When I investigated the warning message that pointed to clusterrole, it looks a little bit messy for me, and I cannot find out what RBAC permissions it needs.
Thanks for any kind of help.