Hey!
I’ve set up Binderhub for our university and it works if I use version “0.2.0-n1011.hb49edf6” but not the newest ones (“1.0.0-0.dev.git.3018.he00ec49” is the latest one I’ve tried).
When trying to access ‘hub.binder.mydomain.com’, I receive error “Service Unavailable”.
When checking the logs for the hub pod, it says:
[I 2023-02-13 13:14:10.447 JupyterHub oauth2:102] OAuth redirect: 'https://hub.binder.edu.liu.se/hub/oauth_callback'
[I 2023-02-13 13:14:10.448 JupyterHub log:186] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://login.microsoftonline.com/913f18ec-7f26-4c5f-a816-784fe9a58edd/oauth2/authorize?response_type=code&redirect_uri=https%3A%2F%2Fhub.binder.edu.liu.se%2Fhub%2Foauth_callback&client_id=5d07725f-e78b-4cf8-bf9c-df7df65b34bb&state=[secret] (@130.236.18.90) 1.01ms
[I 2023-02-13 13:14:10.965 JupyterHub roles:238] Adding role user for User: henbj71@liu.se
[I 2023-02-13 13:14:11.440 JupyterHub base:810] User logged in: henbj71@liu.se
[I 2023-02-13 13:14:11.441 JupyterHub log:186] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /hub/ (@130.236.18.90) 720.65ms
[I 2023-02-13 13:14:11.625 JupyterHub log:186] 302 GET /hub/ -> /hub/home (henbj71@liu.se@130.236.18.90) 159.45ms
[I 2023-02-13 13:14:11.673 JupyterHub log:186] 200 GET /hub/home (henbj71@liu.se@130.236.18.90) 32.06ms
[E 2023-02-13 13:14:11.685 JupyterHub reflector:385] Initial list of events failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
await self._list_and_update()
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
for p in initial_resources["items"]
KeyError: 'items'
[E 2023-02-13 13:14:11.685 JupyterHub spawner:2402] Reflector for events failed to start.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2400, in catch_reflector_start
await f
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
await self._list_and_update()
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
for p in initial_resources["items"]
KeyError: 'items'
Task was destroyed but it is pending!
task: <Task pending name='Task-394' coro=<shared_client.<locals>.close_client_task() running at /usr/local/lib/python3.9/site-packages/kubespawner/clients.py:58> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f4e50aa4e80>()]>>
Task exception was never retrieved
future: <Task finished name='Task-398' coro=<KubeSpawner._start_reflector.<locals>.catch_reflector_start() done, defined at /usr/local/lib/python3.9/site-packages/kubespawner/spawner.py:2398> exception=SystemExit(1)>
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2400, in catch_reflector_start
await f
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
await self._list_and_update()
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
for p in initial_resources["items"]
KeyError: 'items'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3313, in launch_instance
loop.start()
File "/usr/local/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 215, in start
self.asyncio_loop.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
handle._run()
File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2403, in catch_reflector_start
sys.exit(1)
SystemExit: 1
Exception ignored in:
<coroutine object shared_client.<locals>.close_client_task at 0x7f4e50a58e40>
RuntimeError: coroutine ignored GeneratorExit
My config file looks like this (removed passwords and other secrets):
config:
BinderHub:
debug: true
cors_allow_origin: '*'
use_registry: true
image_prefix: "gitlab.it.liu.se:5000/drs/binderhub/prod-"
hub_url: "https://hub.binder.edu.liu.se" #If no cert, use http. If cert, use https
auth_enabled: true
use_named_servers: true
# To display the university logo
template_path: /etc/binderhub/custom/templates
extra_static_path: /etc/binderhub/custom/static
extra_static_url_prefix: /extra_static/
template_variables:
EXTRA_STATIC_URL_PREFIX: "/extra_static/"
registry:
url: "https://gitlab.it.liu.se:5000/drs/binderhub"
username: "SECRET"
jupyterhub:
custom:
binderauth_enabled: true
singleuser:
cmd: jupyter-labhub
memory:
guarantee: 128M
limit: 4G
cpu:
guarantee: .1
limit: 1
prePuller:
continuous:
enabled: true
scheduling:
userScheduler:
enabled: false # Requires Cluster Roles if set to true
cull:
enabled: true
users: false
removeNamedServers: true
every: 60
timeout: 900
hub:
db:
pvc:
storageClassName: ceph
config:
BinderSpawner:
cors_allow_origin: '*'
allowNamedServers: true
namedServerLimitPerUser: 50
redirectToServer: false
loadRoles:
user:
scopes:
- self
- "access:services"
services:
binder:
oauth_no_confirm: true
oauth_redirect_uri: "https://binder.edu.liu.se/oauth_callback"
oauth_client_id: "service-SECRET" #cookie
config:
Authenticator:
admin_users:
- "henbj71@ad.liu.se"
auto_login: true
AzureAdOAuthenticator:
client_id: "SECRET"
oauth_callback_url: "https://hub.binder.edu.liu.se/hub/oauth_callback"
tenant_id: "SECRET"
username_claim: upn
JupyterHub:
authenticator_class: azuread
BinderSpawner:
auth_enabled: false
rbac:
create: false
proxy:
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- hub.binder.edu.liu.se
annotations:
kubernetes.io/ingress.class: nginx-public
cert-manager.io/issuer: binder-letsencrypt-issuer
tls:
- secretName: hub-binder-edu-liu-se-tls
hosts:
- hub.binder.edu.liu.se
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- binder.edu.liu.se
annotations:
kubernetes.io/ingress.class: nginx-public
cert-manager.io/issuer: binder-letsencrypt-issuer
tls:
- secretName: binder-edu-liu-se-tls
hosts:
- binder.edu.liu.se
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/login/"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/_oauth"
imageBuilderType: dind
dind:
hostLibDir: "/var/lib/dind/henbj71"
hostSocketDir: "/var/run/dind/henbj71"
daemonset:
image:
name: docker
tag: 20.10.12-dind
# These require clusterroles. So we disable them
imageCleaner:
enabled: false
pdb:
enabled: false
# Only for displaying the LiU logo
initContainers:
- name: git-clone-templates
image: alpine/git
args:
- clone
- --single-branch
- --branch=master
- --depth=1
- https://github.com/henricbjork2liu/binderhub-custom-files
- /etc/binderhub/custom
securityContext:
runAsUser: 0
volumeMounts:
- name: custom-templates
mountPath: /etc/binderhub/custom
extraVolumes:
- name: custom-templates
emptyDir: {}
extraVolumeMounts:
- name: custom-templates
mountPath: /etc/binderhub/custom
The newest one I’ve tried is “1.0.0-0.dev.git.3018.he00ec49”. Now there were a few things I had to change in the config files so I might have missed something.
I’ve not updated my default roles, but from what I can see they shouldn’t be the issue.
# Source: binderhub/templates/rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: default-roles
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list", "create", "delete"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list"]
---
# Source: binderhub/templates/rbac.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: default-roles
subjects:
- kind: ServiceAccount
namespace: henbj71-binderhub
name: default
roleRef:
kind: Role
name: default-roles
apiGroup: rbac.authorization.k8s.io