Creating Binderhub without ClusterRoles

henricbjork2liu · September 15, 2021, 11:54am

Hello all!

Is it possible to set up Binderhub without ClusterRoles?

I’ve set up binderhub using Azure Kubernetes, which worked fine. However, we decided to migrate binderhub to the Kubernetes cluster we have on prem to save money. I am not the admin of this cluster, and I’m missing permissions to list, edit, and add cluster roles. A workaround I’m currently testing is hosting binderhub via VCluster.

As of now my config file is very bare bones:

config:
  BinderHub:
    use_registry: true
    image_prefix: "myacr.azurecr.io/binder/binder-"
    hub_url: "http://my-hub-url-which-currently-is-just-an-ip-address"
  DockerRegistry:
    token_url: "https://myacr.azurecr.io/oauth2/token?service=myacr.azurecr.io"

dind:
  enabled: true
  hostLibDir: "/var/lib/dind/stuff"
  hostSocketDir: "/var/run/dind/stuff"
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"

manics · September 15, 2021, 8:11pm

It looks like the ClusterRole is used for the image-cleaner:

github.com

jupyterhub/binderhub/blob/1f81316f7d8983ecbf99dd1c2c80c57263c83fb3/helm-chart/binderhub/templates/rbac.yaml#L47-L61

    
      
          # image-cleaner role
          # needs to cordon nodes during image cleaning
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            labels:
              app: binderhub
              chart: {{ .Chart.Name }}-{{ .Chart.Version }}
              heritage: {{ .Release.Service }}
              release: {{ .Release.Name }}
            name: {{ .Release.Name }}-image-cleaner
          rules:
          - apiGroups: [""] # "" indicates the core API group
            resources: ["nodes"]
            verbs: ["get", "patch"]

github.com

jupyterhub/binderhub/blob/1f81316f7d8983ecbf99dd1c2c80c57263c83fb3/helm-chart/images/image-cleaner/image-cleaner.py#L2-L8

    
      
          """
          Clean docker images
          
          
This serves as a substitute for Kubernetes ImageGC
          which has thresholds that are not sufficiently configurable on GKE
          at this time.
          """

I don’t know what would happen if you disabled it, you could fork the chart and try and let us know?

henricbjork2liu · September 17, 2021, 12:26pm

To be honest, I’m quite new to Kubernetes, Helm, Docker, and all of this.
I forked the Binderhub repository, and tried creating a new helm chart but didn’t get it to work.

I did however edit the config file so it looks like this:

config:
  BinderHub:
    use_registry: true
    image_prefix: "myacr.azurecr.io/binder/binder-"
    hub_url: "http://my-hub-url-which-currently-is-just-an-ip-address"
  DockerRegistry:
    token_url: "https://myacr.azurecr.io/oauth2/token?service=myacr.azurecr.io"

dind:
  enabled: true
  hostLibDir: "/var/lib/dind/stuff"
  hostSocketDir: "/var/run/dind/stuff"
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"
      
imageCleaner:
  host:
    enabled: false
    
rbac:
  enabled: false

Now Image-Cleaner no longer requires a ClusterRole. I tested running
helm install "binderhub" jupyterhub/binderhub --version="0.2.0-n645.h06139f0" -f myconfig.yaml -f mysecret.yaml
I got error message: clusterroles.rbac.authorization.k8s.io "binderhub-user-scheduler" is forbidden

I then tested:
helm template"binderhub" jupyterhub/binderhub --version="0.2.0-n645.h06139f0" -f myconfig.yaml -f mysecret.yaml
And as the error message said, it seems that User-Scheduler also requires a ClusterRole

# Source: binderhub/charts/jupyterhub/templates/scheduling/user-scheduler/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: binderhub-user-scheduler
  labels:
    component: user-scheduler
    app: jupyterhub
    release: binderhub
    chart: jupyterhub-1.1.1
    heritage: Helm
rules:
  # Copied from the system:kube-scheduler ClusterRole of the k8s version
  # matching the kube-scheduler binary we use. A modification of two resource
  # name references from kube-scheduler to user-scheduler-lock was made.
  #
  # NOTE: These rules have been unchanged between 1.12 and 1.15, then changed in
  #       1.16 and in 1.17, but unchanged in 1.18 and 1.19.
  #
  # ref: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml#L696-L829
  - apiGroups:
    - ""
    - events.k8s.io
    resources:
    - events
    verbs:
    - create
    - patch
    - update
  - apiGroups:
    - coordination.k8s.io
    resources:
    - leases
    verbs:
    - create
  - apiGroups:
    - coordination.k8s.io
    resourceNames:
    - user-scheduler-lock
    resources:
    - leases
    verbs:
    - get
    - update
  - apiGroups:
    - ""
    resources:
    - endpoints
    verbs:
    - create
  - apiGroups:
    - ""
    resourceNames:
    - user-scheduler-lock
    resources:
    - endpoints
    verbs:
    - get
    - update
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - delete
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - bindings
    - pods/binding
    verbs:
    - create
  - apiGroups:
    - ""
    resources:
    - pods/status
    verbs:
    - patch
    - update
  - apiGroups:
    - ""
    resources:
    - replicationcontrollers
    - services
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    - extensions
    resources:
    - replicasets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    resources:
    - statefulsets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - policy
    resources:
    - poddisruptionbudgets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - persistentvolumeclaims
    - persistentvolumes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - authentication.k8s.io
    resources:
    - tokenreviews
    verbs:
    - create
  - apiGroups:
    - authorization.k8s.io
    resources:
    - subjectaccessreviews
    verbs:
    - create
  - apiGroups:
    - storage.k8s.io
    resources:
    - csinodes
    verbs:
    - get
    - list
    - watch

  # Copied from the system:volume-scheduler ClusterRole of the k8s version
  # matching the kube-scheduler binary we use.
  #
  # NOTE: These rules have not changed between 1.12 and 1.19.
  #
  # ref: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml#L1213-L1240
  - apiGroups:
    - ""
    resources:
    - persistentvolumes
    verbs:
    - get
    - list
    - patch
    - update
    - watch
  - apiGroups:
    - storage.k8s.io
    resources:
    - storageclasses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - persistentvolumeclaims
    verbs:
    - get
    - list
    - patch
    - update
    - watch

henricbjork2liu · September 17, 2021, 12:42pm

Out of curiosity, I tested changing the ClusterRoles to Roles, and the ClusterRoleBindings to RoleBindings, in the template file I got from “helm template”. I then ran
kubectl apply -f template.yaml -n "mynamespace"

I was able to access binderhub and build repositories just fine, but I couldn’t launch them.
The logs for the User-Sheduler pods repeated thins like:

E0917 12:34:34.500785 1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:mynamespace:user-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope

I won’t list all the errors here, but it seems it complains about everything wanting to be run on cluster scope. Replicasets, PersistentVolume, csinodes, pod binding, and more.

Is it possible to make User Scheduler to use regular roles instead of clusterroles?

minrk · September 17, 2021, 2:50pm

No, schedulers aren’t per-namespace, so they can only use ClusterRoles. You have to opt-out of the user scheduler to avoid this:

scheduling:
  userScheduler:
    enabled: false

BinderHub functions just fine without the custom scheduler. the custom scheduler mainly improves autoscaling behavior, especially scale-down. WIth this and the image cleaner disabled, there should be no ClusterRoles, I think.

henricbjork2liu · September 20, 2021, 1:37pm

Disabling userScheduler seemed to work!

Now I have issues with pvc instead, but I think it’s an issue I have to check with the admin of the cluster.
Thanks for the help!

Topic		Replies	Views
Do I have to use a repository with Binderhub? If not, how do I locally run them? Binder	4	1277	August 6, 2021
403 Forbidden when trying to install binder on AWS and dockerhub BinderHub	2	782	September 18, 2019
Building a "The Littlest BinderHub" BinderHub	9	1858	September 15, 2023
AWS integration work Binder	3	1804	August 7, 2019
Debugging BinderHub: Immediately fails to build a binder BinderHub	6	1550	March 20, 2019

Creating Binderhub without ClusterRoles

Related Topics