I have deployed jupyterhub in my local k3d cluster and I have change the config to spawn the user pods in a separate namespace. I have set the interNamespaceAccessLabels: accept and after many failed attempts to give the correct rbac, I gave to hub the cluster-admin role and set blockWithIptables: false .
I have the following config for Profile:
{
'display_name': 'Datascience environment',
'description': 'If you want the additional bells and whistles: Python, R, and Julia.',
'server_name': 'datasciencel_notebook',
'kubespawner_override': {
'image': 'hello-world',
'namespace': 'jupyterhub-client-pods'
}
}
The pod remain in pending status until the timeout:
[D 2024-04-29 10:02:46.349 JupyterHub scopes:884] Checking access to /hub/spawn via scope servers
[D 2024-04-29 10:02:46.350 JupyterHub scopes:697] Argument-based access to /hub/spawn via servers
[D 2024-04-29 10:02:46.351 JupyterHub pages:258] Triggering spawn with supplied form options for blina
[D 2024-04-29 10:02:46.355 JupyterHub base:1061] Initiating spawn for blina
[D 2024-04-29 10:02:46.356 JupyterHub base:1065] 0/64 concurrent spawns
[D 2024-04-29 10:02:46.356 JupyterHub base:1070] 0 active servers
[I 2024-04-29 10:02:46.412 JupyterHub provider:660] Creating oauth client jupyterhub-user-blina
[D 2024-04-29 10:02:46.463 JupyterHub user:797] Calling Spawner.start for blina
[D 2024-04-29 10:02:46.465 JupyterHub spawner:3220] Applying KubeSpawner override for profile 'Datascience environment'
[D 2024-04-29 10:02:46.465 JupyterHub spawner:3198] .. overriding KubeSpawner value image=hello-world
[D 2024-04-29 10:02:46.465 JupyterHub spawner:3198] .. overriding KubeSpawner value namespace=jupyterhub-client-pods
[I 2024-04-29 10:02:46.475 JupyterHub log:192] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/blina?_xsrf=[secret] (blina@::ffff:127.0.0.1) 152.29ms
[D 2024-04-29 10:02:46.500 JupyterHub scopes:884] Checking access to /hub/spawn-pending/blina via scope servers
[D 2024-04-29 10:02:46.501 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/blina via servers
[I 2024-04-29 10:02:46.501 JupyterHub pages:399] blina is pending spawn
[I 2024-04-29 10:02:46.523 JupyterHub log:192] 200 GET /hub/spawn-pending/blina?_xsrf=[secret] (blina@::ffff:127.0.0.1) 29.50ms
[I 2024-04-29 10:02:46.532 JupyterHub reflector:282] watching for events with field selector='involvedObject.kind=Pod' in namespace jupyterhub-client-pods
[D 2024-04-29 10:02:46.532 JupyterHub reflector:289] Connecting events watcher
[I 2024-04-29 10:02:46.552 JupyterHub spawner:2551] Attempting to create pvc claim-blina, with timeout 3
[D 2024-04-29 10:02:46.597 JupyterHub scopes:884] Checking access to /hub/api/users/blina/server/progress via scope read:servers
[D 2024-04-29 10:02:46.598 JupyterHub scopes:697] Argument-based access to /hub/api/users/blina/server/progress via read:servers
[D 2024-04-29 10:02:46.604 JupyterHub spawner:2303] progress generator: jupyter-blina
[I 2024-04-29 10:02:46.605 JupyterHub spawner:2567] PVC claim-blina already exists, so did not create new pvc.
[I 2024-04-29 10:02:46.632 JupyterHub spawner:2511] Attempting to create pod jupyter-blina, with timeout 3
[D 2024-04-29 10:02:46.981 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 6.94ms
[D 2024-04-29 10:02:48.986 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 5.93ms
[D 2024-04-29 10:02:50.974 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 4.38ms
[D 2024-04-29 10:02:52.974 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 4.02ms
[D 2024-04-29 10:02:54.974 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 4.98ms
[D 2024-04-29 10:02:55.048 JupyterHub reflector:374] pods watcher timeout
[D 2024-04-29 10:02:55.049 JupyterHub reflector:289] Connecting pods watcher
...
[D 2024-04-29 10:07:46.222 JupyterHub reflector:374] pods watcher timeout
[D 2024-04-29 10:07:46.224 JupyterHub reflector:289] Connecting pods watcher
[W 2024-04-29 10:07:46.474 JupyterHub user:879] blina's server failed to start in 300 seconds, giving up.
Common causes of this timeout, and debugging tips:
1. Everything is working, but it took too long.
To fix: increase `Spawner.start_timeout` configuration
to a number of seconds that is enough for spawners to finish starting.
2. The server didn't finish starting,
or it crashed due to a configuration issue.
Check the single-user server's logs for hints at what needs fixing.
[D 2024-04-29 10:07:46.479 JupyterHub user:985] Stopping blina
[I 2024-04-29 10:07:46.485 JupyterHub spawner:2849] Deleting pod jupyterhub-client-pods/jupyter-blina
[D 2024-04-29 10:07:46.625 JupyterHub user:1005] Deleting oauth client jupyterhub-user-blina
[D 2024-04-29 10:07:46.640 JupyterHub user:1008] Finished stopping blina
[E 2024-04-29 10:07:46.653 JupyterHub gen:630] Exception in Future <Task finished name='Task-219' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:1081> exception=TimeoutError('Timeout')> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
future.result()
File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 1088, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 905, in spawn
raise e
File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 801, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: Timeout
[I 2024-04-29 10:07:46.674 JupyterHub log:192] 200 GET /hub/api/users/blina/server/progress?_xsrf=[secret] (blina@::ffff:127.0.0.1) 300085.04ms
[D 2024-04-29 10:07:46.984 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 6.38ms
[D 2024-04-29 10:07:46.986 JupyterHub log:192] 200 GET /hub/health (@10.42.0.1) 5.91ms
[D 2024-04-29 10:07:47.723 JupyterHub reflector:374] events watcher timeout
[D 2024-04-29 10:07:47.723 JupyterHub reflector:289] Connecting events watcher
[E 2024-04-29 10:07:48.555 JupyterHub spawner:2810] Pod jupyterhub-client-pods/jupyter-blina never showed up in reflector, restarting pod reflector
[E 2024-04-29 10:07:48.556 JupyterHub spawner:2814] Pods: []
[E 2024-04-29 10:07:48.567 JupyterHub gen:630] Exception in Future <Task finished name='Task-220' coro=<KubeSpawner._start() done, defined at /usr/local/lib/python3.11/site-packages/kubespawner/spawner.py:2663> exception=TimeoutError('pod jupyterhub-client-pods/jupyter-blina did not start in 300 seconds!')> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
future.result()
File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2801, in _start
await exponential_backoff(
File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 257, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
TimeoutError: pod jupyterhub-client-pods/jupyter-blina did not start in 300 seconds!
[I 2024-04-29 10:07:48.582 JupyterHub reflector:282] watching for pods with label selector='component=singleuser-server' in namespace jupyterhub-client-pods
[D 2024-04-29 10:07:48.583 JupyterHub reflector:289] Connecting pods watcher
[D 2024-04-29 10:07:48.596 JupyterHub reflector:355] Cancelled watching pods
[I 2024-04-29 10:07:48.596 JupyterHub reflector:378] pods watcher stopped: outer
[W 2024-04-29 10:07:48.596 JupyterHub reflector:380] pods watcher finished
Can you help me figure out what I am missing here?