# Creating Binderhub without ClusterRoles

**URL:** <https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771>\
**Category:** Binder\
**Tags:** how-to, help-wanted\
**Created:** [September 15, 2021, 11:54am UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771 "2021-09-15T11:54:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![henricbjork2liu](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/henricbjork2liu/32/5209_2.png) [@henricbjork2liu](https://discourse.jupyter.org/u/henricbjork2liu)\
**Post date:** [September 15, 2021, 11:54am UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/1 "2021-09-15T11:54:36Z")

</div>

Hello all!

Is it possible to set up Binderhub without ClusterRoles?

I’ve set up binderhub using Azure Kubernetes, which worked fine. However, we decided to migrate binderhub to the Kubernetes cluster we have on prem to save money. I am not the admin of this cluster, and I’m missing permissions to list, edit, and add cluster roles. A workaround I’m currently testing is hosting binderhub via VCluster.

As of now my config file is very bare bones:

```auto
config:
  BinderHub:
    use_registry: true
    image_prefix: "myacr.azurecr.io/binder/binder-"
    hub_url: "http://my-hub-url-which-currently-is-just-an-ip-address"
  DockerRegistry:
    token_url: "https://myacr.azurecr.io/oauth2/token?service=myacr.azurecr.io"

dind:
  enabled: true
  hostLibDir: "/var/lib/dind/stuff"
  hostSocketDir: "/var/run/dind/stuff"
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"

```

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [September 15, 2021, 8:11pm UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/2 "2021-09-15T20:11:25Z")

</div>

It looks like the `ClusterRole` is used for the image-cleaner:

> <https://github.com/jupyterhub/binderhub/blob/1f81316f7d8983ecbf99dd1c2c80c57263c83fb3/helm-chart/binderhub/templates/rbac.yaml#L47-L61>

> <https://github.com/jupyterhub/binderhub/blob/1f81316f7d8983ecbf99dd1c2c80c57263c83fb3/helm-chart/images/image-cleaner/image-cleaner.py#L2-L8>

I don’t know what would happen if you disabled it, you could fork the chart and try and let us know?

---

<div class="post-metadata">

**Author:** ![henricbjork2liu](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/henricbjork2liu/32/5209_2.png) [@henricbjork2liu](https://discourse.jupyter.org/u/henricbjork2liu)\
**Post date:** [September 17, 2021, 12:26pm UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/3 "2021-09-17T12:26:32Z")

</div>

To be honest, I’m quite new to Kubernetes, Helm, Docker, and all of this.  
I forked the Binderhub repository, and tried creating a new helm chart but didn’t get it to work.

I did however edit the config file so it looks like this:

```auto
config:
  BinderHub:
    use_registry: true
    image_prefix: "myacr.azurecr.io/binder/binder-"
    hub_url: "http://my-hub-url-which-currently-is-just-an-ip-address"
  DockerRegistry:
    token_url: "https://myacr.azurecr.io/oauth2/token?service=myacr.azurecr.io"

dind:
  enabled: true
  hostLibDir: "/var/lib/dind/stuff"
  hostSocketDir: "/var/run/dind/stuff"
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"
      
imageCleaner:
  host:
    enabled: false
    
rbac:
  enabled: false

```

Now Image-Cleaner no longer requires a ClusterRole. I tested running  
`helm install "binderhub" jupyterhub/binderhub --version="0.2.0-n645.h06139f0" -f myconfig.yaml -f mysecret.yaml`  
I got error message: `clusterroles.rbac.authorization.k8s.io "binderhub-user-scheduler" is forbidden`

I then tested:  
`helm template"binderhub" jupyterhub/binderhub --version="0.2.0-n645.h06139f0" -f myconfig.yaml -f mysecret.yaml`  
And as the error message said, it seems that User-Scheduler also requires a ClusterRole

```auto
# Source: binderhub/charts/jupyterhub/templates/scheduling/user-scheduler/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: binderhub-user-scheduler
  labels:
    component: user-scheduler
    app: jupyterhub
    release: binderhub
    chart: jupyterhub-1.1.1
    heritage: Helm
rules:
  # Copied from the system:kube-scheduler ClusterRole of the k8s version
  # matching the kube-scheduler binary we use. A modification of two resource
  # name references from kube-scheduler to user-scheduler-lock was made.
  #
  # NOTE: These rules have been unchanged between 1.12 and 1.15, then changed in
  # 1.16 and in 1.17, but unchanged in 1.18 and 1.19.
  #
  # ref: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml#L696-L829
  - apiGroups:
    - ""
    - events.k8s.io
    resources:
    - events
    verbs:
    - create
    - patch
    - update
  - apiGroups:
    - coordination.k8s.io
    resources:
    - leases
    verbs:
    - create
  - apiGroups:
    - coordination.k8s.io
    resourceNames:
    - user-scheduler-lock
    resources:
    - leases
    verbs:
    - get
    - update
  - apiGroups:
    - ""
    resources:
    - endpoints
    verbs:
    - create
  - apiGroups:
    - ""
    resourceNames:
    - user-scheduler-lock
    resources:
    - endpoints
    verbs:
    - get
    - update
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - delete
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - bindings
    - pods/binding
    verbs:
    - create
  - apiGroups:
    - ""
    resources:
    - pods/status
    verbs:
    - patch
    - update
  - apiGroups:
    - ""
    resources:
    - replicationcontrollers
    - services
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    - extensions
    resources:
    - replicasets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    resources:
    - statefulsets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - policy
    resources:
    - poddisruptionbudgets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - persistentvolumeclaims
    - persistentvolumes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - authentication.k8s.io
    resources:
    - tokenreviews
    verbs:
    - create
  - apiGroups:
    - authorization.k8s.io
    resources:
    - subjectaccessreviews
    verbs:
    - create
  - apiGroups:
    - storage.k8s.io
    resources:
    - csinodes
    verbs:
    - get
    - list
    - watch

  # Copied from the system:volume-scheduler ClusterRole of the k8s version
  # matching the kube-scheduler binary we use.
  #
  # NOTE: These rules have not changed between 1.12 and 1.19.
  #
  # ref: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml#L1213-L1240
  - apiGroups:
    - ""
    resources:
    - persistentvolumes
    verbs:
    - get
    - list
    - patch
    - update
    - watch
  - apiGroups:
    - storage.k8s.io
    resources:
    - storageclasses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - persistentvolumeclaims
    verbs:
    - get
    - list
    - patch
    - update
    - watch

```

---

<div class="post-metadata">

**Author:** ![henricbjork2liu](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/henricbjork2liu/32/5209_2.png) [@henricbjork2liu](https://discourse.jupyter.org/u/henricbjork2liu)\
**Post date:** [September 17, 2021, 12:42pm UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/4 "2021-09-17T12:42:07Z")

</div>

Out of curiosity, I tested changing the ClusterRoles to Roles, and the ClusterRoleBindings to RoleBindings, in the template file I got from “helm template”. I then ran  
`kubectl apply -f template.yaml -n "mynamespace"`

I was able to access binderhub and build repositories just fine, but I couldn’t launch them.  
The logs for the User-Sheduler pods repeated thins like:

```auto
E0917 12:34:34.500785 1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:mynamespace:user-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope

```

I won’t list all the errors here, but it seems it complains about everything wanting to be run on cluster scope. Replicasets, PersistentVolume, csinodes, pod binding, and more.

Is it possible to make User Scheduler to use regular roles instead of clusterroles?

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [September 17, 2021, 2:50pm UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/5 "2021-09-17T14:50:59Z")

</div>

> [@henricbjork2liu](#):
>
> Is it possible to make User Scheduler to use regular roles instead of clusterroles?

No, schedulers aren’t per-namespace, so they can only use ClusterRoles. You have to opt-out of the user scheduler to avoid this:

```yaml
scheduling:
  userScheduler:
    enabled: false

```

BinderHub functions just fine without the custom scheduler. the custom scheduler mainly improves autoscaling behavior, especially scale-down. WIth this and the image cleaner disabled, there should be no ClusterRoles, I think.

---

<div class="post-metadata">

**Author:** ![henricbjork2liu](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/henricbjork2liu/32/5209_2.png) [@henricbjork2liu](https://discourse.jupyter.org/u/henricbjork2liu)\
**Post date:** [September 20, 2021, 1:37pm UTC](https://discourse.jupyter.org/t/creating-binderhub-without-clusterroles/10771/6 "2021-09-20T13:37:22Z")

</div>

Disabling userScheduler seemed to work!

Now I have issues with pvc instead, but I think it’s an issue I have to check with the admin of the cluster.  
Thanks for the help!