Jupyerhub spawner failing with an exception: "Initial list of events failed"

JupyterHub Zero to JupyterHub on Kubernetes
1

I am installing jupyterhub on k8s cluster using the following instructions

Kubectl Client Version: v1.27.1
Kubectl Server Version: v1.20.12
helm version: v3.11.3
Jupyterhub version: 2.0.0

I used the following configuration file

rbac:
  create: false
prePuller:
  hook:
    enabled: false
  continuous:
    # NOTE: if used with a Cluster Autoscaler, also add user-placeholders
    enabled: false
ingress:
  enabled: true
  pathSuffix: hub
  hosts:
    - <redacted>
proxy:
  service:
    type: NodePort
hub:
  service:
    type: NodePort
debug:
  enabled: true
scheduling:
  userScheduler:
    enabled: false

I see all the expected pods, services are running. But, when I try to login or start the server, I see following error in hub pod.

[D 2023-05-01 13:35:50.328 JupyterHub base:320] Refreshing auth for test
[D 2023-05-01 13:35:50.342 JupyterHub user:430] Creating <class 'kubespawner.spawner.KubeSpawner'> for test:
[I 2023-05-01 13:35:50.385 JupyterHub log:186] 302 GET /hub/ -> /hub/spawn (test@<redacted>) 77.48ms
[E 2023-05-01 13:35:50.393 JupyterHub reflector:385] Initial list of events failed
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
        await self._list_and_update()
      File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
        for p in initial_resources["items"]
    KeyError: 'items'
    
[E 2023-05-01 13:35:50.394 JupyterHub spawner:2402] Reflector for events failed to start.
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2400, in catch_reflector_start
        await f
      File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
        await self._list_and_update()
      File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
        for p in initial_resources["items"]
    KeyError: 'items'
    
Task exception was never retrieved
future: <Task finished name='Task-126013' coro=<KubeSpawner._start_reflector.<locals>.catch_reflector_start() done, defined at /usr/local/lib/python3.9/site-packages/kubespawner/spawner.py:2398> exception=SystemExit(1)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2400, in catch_reflector_start
    await f
  File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 383, in start
    await self._list_and_update()
  File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 233, in _list_and_update
    for p in initial_resources["items"]
KeyError: 'items'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3313, in launch_instance
    loop.start()
  File "/usr/local/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 215, in start
    self.asyncio_loop.run_forever()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
    handle._run()
  File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.9/site-packages/kubespawner/spawner.py", line 2403, in catch_reflector_start
    sys.exit(1)
SystemExit: 1
Task was destroyed but it is pending!
task: <Task pending name='Task-126009' coro=<shared_client.<locals>.close_client_task() running at /usr/local/lib/python3.9/site-packages/kubespawner/clients.py:58> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7ffff05fa8e0>()]>>
Exception ignored in: <coroutine object shared_client.<locals>.close_client_task at 0x7ffff05b4d40>
RuntimeError: coroutine ignored GeneratorExit

Any suggestions or help on what would be an issue is greatly appreciated

2

One cause of this error can be insufficient permissions for the Kubernets service account used by the pod.

Since you’ve disabled RBAC creation you’ll need to set up the role and service account yourself.

3

Thank you @manics. I will get access to a less restrictive cluster next week. I will try jupyterhub installation on it and I will let you know if it works for me.
I will enable rbac, create the role and see how things goes

4

I got access to less restrictive cluster where I have permissions to manage roles. I was able to enable RBAC and install jupyterhub.
I have to set following config item for everything to work
singleuser.cloudMetadata.blockWithIptables: False

Thanks for the help