Trying to spawn pods in other namespace

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Hello community,

I’m facing some challenges while trying to spawn pods in another namespace. I want to spawn all the pods of a specific profile to another namespace. Right now I am working within Minikube with Calico installed. Below are the configurations I changed in the values.yaml:

hub.KubeSpawner:
    hub_connect_url: "http://hub.jupyterhub-system.svc.cluster.local:8081"
    environment: { JUPYTERHUB_API_URL: "http://hub.jupyterhub-system.svc.cluster.local:8081/hub/api" }
hub.networkPolicy:
    interNamespaceAccessLabels: accept

singleuser.profileList
      kubespawner_override:
        namespace: other-namespace
        extra_labels:
          hub.jupyter.org/network-access-hub: "true"

I also created a RoleBinding to allow the hub ServiceAccount to CRUD Pods and PVCs in the other-namespace.

Problem:
When trying to start a Server in the UI, it leads to the loading screen, but nothing happens. The user pod is running in the other-namespace but I’m not getting redirected. I also get no error messages, except for the timeout after 600 seconds.

Logs: (before timeout)
Hub:

[I 2023-11-13 15:31:59.416 JupyterHub log:191] 200 GET /hub/login (@::ffff:10.244.120.64) 1.90ms
[I 2023-11-13 15:32:02.008 JupyterHub base:837] User logged in: a
[I 2023-11-13 15:32:02.014 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (a@::ffff:10.244.120.64) 16.85ms
[I 2023-11-13 15:32:02.040 JupyterHub log:191] 200 GET /hub/spawn (a@::ffff:10.244.120.64) 13.67ms
[I 2023-11-13 15:32:03.120 JupyterHub provider:659] Creating oauth client jupyterhub-user-a
[I 2023-11-13 15:32:03.130 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/a?_xsrf=[secret] (a@::ffff:10.244.120.64) 37.28ms
[I 2023-11-13 15:32:03.133 JupyterHub spawner:2497] Attempting to create pod jupyter-a, with timeout 3
[I 2023-11-13 15:32:03.139 JupyterHub pages:398] a is pending spawn
[I 2023-11-13 15:32:03.140 JupyterHub log:191] 200 GET /hub/spawn-pending/a?_xsrf=[secret] (a@::ffff:10.244.120.64) 1.99ms
[I 2023-11-13 15:32:05.187 JupyterHub log:191] 200 GET /hub/api (@10.244.120.76) 0.32ms
[I 2023-11-13 15:32:05.210 JupyterHub log:191] 200 POST /hub/api/users/a/activity (a@10.244.120.76) 7.49ms

User-Pod:

[I 2023-11-13 15:32:05.188 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2023-11-13 15:32:05.188 ServerApp] jupyterhub | extension was successfully loaded.
[I 2023-11-13 15:32:05.191 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-11-13 15:32:05.191 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-11-13 15:32:05.191 LabApp] Extension Manager is 'pypi'.
[I 2023-11-13 15:32:05.192 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-11-13 15:32:05.195 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2023-11-13 15:32:05.197 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-11-13 15:32:05.197 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2023-11-13 15:32:05.198 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-11-13 15:32:05.198 ServerApp] Jupyter Server 2.7.3 is running at:
[I 2023-11-13 15:32:05.198 ServerApp] http://jupyter-a:8888/user/a/lab?token=...
[I 2023-11-13 15:32:05.198 ServerApp]     http://127.0.0.1:8888/user/a/lab?token=...
[I 2023-11-13 15:32:05.198 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Any insights or guidance on what might be causing the issue would be greatly appreciated. Thank you!

2

Can you turn on debug logging and share your logs?
Does Z2JH work in a default configuration (everything in the same namespace)?
Does it work if you disable network policies for all components?

3

Hello manics,

thank you for your quick response and the tips on further debugging!
In the default setting everything works fine, I can even use a profile as long as I dont use the kubespawner_override.namespace option.

Here are the logs from the profile+namespace setting with the debug logging:
Hub:

[I 2023-11-16 19:58:45.827 JupyterHub log:191] 200 GET /hub/login (@::ffff:10.244.120.64) 2.22ms
[D 2023-11-16 19:58:47.699 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 1.16ms
[D 2023-11-16 19:58:49.245 JupyterHub roles:281] Assigning default role to User b
[I 2023-11-16 19:58:49.248 JupyterHub roles:238] Adding role user for User: b
[D 2023-11-16 19:58:49.257 JupyterHub roles:281] Assigning default role to User b
[D 2023-11-16 19:58:49.260 JupyterHub base:587] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2023-11-16 19:58:49.260 JupyterHub base:591] Setting cookie for b: jupyterhub-hub-login
[D 2023-11-16 19:58:49.260 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2023-11-16 19:58:49.260 JupyterHub base:837] User logged in: b
[D 2023-11-16 19:58:49.260 JupyterHub user:431] Creating <class 'kubespawner.spawner.KubeSpawner'> for b:
[I 2023-11-16 19:58:49.263 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (b@::ffff:10.244.120.64) 23.69ms
[D 2023-11-16 19:58:49.271 JupyterHub base:299] Recording first activity for <User(b 0/1 running)>
[D 2023-11-16 19:58:49.275 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 19:58:49.275 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 19:58:49.284 JupyterHub pages:209] Serving options form for b
[I 2023-11-16 19:58:49.285 JupyterHub log:191] 200 GET /hub/spawn (b@::ffff:10.244.120.64) 16.17ms
[D 2023-11-16 19:58:49.690 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.80ms
[D 2023-11-16 19:58:50.321 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 19:58:50.322 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 19:58:50.322 JupyterHub pages:257] Triggering spawn with supplied form options for b
[D 2023-11-16 19:58:50.322 JupyterHub base:961] Initiating spawn for b
[D 2023-11-16 19:58:50.322 JupyterHub base:965] 0/64 concurrent spawns
[D 2023-11-16 19:58:50.322 JupyterHub base:970] 1 active servers
[I 2023-11-16 19:58:50.335 JupyterHub provider:659] Creating oauth client jupyterhub-user-b
[D 2023-11-16 19:58:50.346 JupyterHub user:794] Calling Spawner.start for b
[D 2023-11-16 19:58:50.346 JupyterHub spawner:3199] Applying KubeSpawner override for profile '<big>Tensorflow 2.10 ohne GPU Support</big>'
[D 2023-11-16 19:58:50.346 JupyterHub spawner:3177] .. overriding KubeSpawner value extra_labels={'hub.jupyter.org/network-access-hub': 'true'}
[D 2023-11-16 19:58:50.346 JupyterHub spawner:3177] .. overriding KubeSpawner value namespace=other-namespace
[I 2023-11-16 19:58:50.347 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/b?_xsrf=[secret] (b@::ffff:10.244.120.64) 28.88ms
[I 2023-11-16 19:58:50.350 JupyterHub spawner:2497] Attempting to create pod jupyter-b, with timeout 3
[D 2023-11-16 19:58:50.354 JupyterHub scopes:877] Checking access to /hub/spawn-pending/b via scope servers
[D 2023-11-16 19:58:50.354 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/b via servers
[I 2023-11-16 19:58:50.354 JupyterHub pages:398] b is pending spawn
[I 2023-11-16 19:58:50.354 JupyterHub log:191] 200 GET /hub/spawn-pending/b?_xsrf=[secret] (b@::ffff:10.244.120.64) 2.02ms
[D 2023-11-16 19:58:50.387 JupyterHub scopes:877] Checking access to /hub/api/users/b/server/progress via scope read:servers
[D 2023-11-16 19:58:50.388 JupyterHub scopes:690] Argument-based access to /hub/api/users/b/server/progress via read:servers
[D 2023-11-16 19:58:50.388 JupyterHub spawner:2296] progress generator: jupyter-b
[D 2023-11-16 19:58:50.682 JupyterHub reflector:374] pods watcher timeout
[D 2023-11-16 19:58:50.682 JupyterHub reflector:289] Connecting pods watcher
[I 2023-11-16 19:58:51.538 JupyterHub log:191] 200 GET /hub/api (@10.244.120.78) 0.34ms
[D 2023-11-16 19:58:51.551 JupyterHub base:299] Recording first activity for <APIToken('6195...', user='b', client_id='jupyterhub')>
[D 2023-11-16 19:58:51.555 JupyterHub scopes:877] Checking access to /hub/api/users/b/activity via scope users:activity
[D 2023-11-16 19:58:51.555 JupyterHub scopes:690] Argument-based access to /hub/api/users/b/activity via users:activity
[D 2023-11-16 19:58:51.555 JupyterHub users:874] Activity for user b: 2023-11-16T19:58:51.528437Z
[D 2023-11-16 19:58:51.555 JupyterHub users:892] Activity on server b/: 2023-11-16T19:58:51.528437Z
[I 2023-11-16 19:58:51.557 JupyterHub log:191] 200 POST /hub/api/users/b/activity (b@10.244.120.78) 7.05ms
[D 2023-11-16 19:58:51.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.45ms
[D 2023-11-16 19:58:53.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.42ms
[D 2023-11-16 19:58:55.697 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 3.30ms
[D 2023-11-16 19:58:57.699 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.07ms
[D 2023-11-16 19:58:59.691 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.61ms
[D 2023-11-16 19:59:00.689 JupyterHub reflector:374] pods watcher timeout
[D 2023-11-16 19:59:00.690 JupyterHub reflector:289] Connecting pods watcher
[D 2023-11-16 19:59:01.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.41ms
[D 2023-11-16 19:59:03.688 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.36ms
[D 2023-11-16 19:59:05.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.76ms
[D 2023-11-16 19:59:07.695 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.20ms
[D 2023-11-16 19:59:09.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.72ms
[D 2023-11-16 19:59:10.711 JupyterHub reflector:374] pods watcher timeout
[D 2023-11-16 19:59:10.711 JupyterHub reflector:289] Connecting pods watcher
[D 2023-11-16 19:59:11.693 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.50ms
[D 2023-11-16 19:59:13.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.45ms
[D 2023-11-16 19:59:15.701 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 3.26ms
[D 2023-11-16 19:59:17.698 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.02ms
[D 2023-11-16 19:59:19.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.50ms
[D 2023-11-16 19:59:20.733 JupyterHub reflector:374] pods watcher timeout
[D 2023-11-16 19:59:20.734 JupyterHub reflector:289] Connecting pods watcher
[D 2023-11-16 19:59:21.696 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.65ms
[D 2023-11-16 19:59:23.690 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.40ms
[D 2023-11-16 19:59:25.699 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 1.86ms
[D 2023-11-16 19:59:27.701 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 6.81ms
[D 2023-11-16 19:59:29.690 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.50ms
[D 2023-11-16 19:59:30.668 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2023-11-16 19:59:30.684 JupyterHub proxy:392] Checking routes
[D 2023-11-16 19:59:30.746 JupyterHub reflector:374] pods watcher timeout
[D 2023-11-16 19:59:30.746 JupyterHub reflector:289] Connecting pods watcher
[D 2023-11-16 19:59:31.698 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.50ms
[D 2023-11-16 19:59:33.689 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 0.39ms
[D 2023-11-16 19:59:35.698 JupyterHub log:191] 200 GET /hub/health (@10.244.120.64) 2.29ms

User-Pod:

Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[I 2023-11-16 19:58:51.295 ServerApp] Package jupyterhub took 0.0000s to import
[I 2023-11-16 19:58:51.301 ServerApp] Package jupyter_lsp took 0.0056s to import
[W 2023-11-16 19:58:51.301 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-16 19:58:51.304 ServerApp] Package jupyter_server_terminals took 0.0032s to import
[I 2023-11-16 19:58:51.345 ServerApp] Package jupyterlab took 0.0407s to import
[I 2023-11-16 19:58:51.367 ServerApp] Package nbclassic took 0.0008s to import
[W 2023-11-16 19:58:51.368 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-16 19:58:51.368 ServerApp] Package nbgitpuller took 0.0005s to import
[I 2023-11-16 19:58:51.368 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-11-16 19:58:51.368 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-16 19:58:51.369 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-11-16 19:58:51.371 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-11-16 19:58:51.371 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2023-11-16 19:58:51.371 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2023-11-16 19:58:51.372 ServerApp] jupyterhub | extension was successfully linked.
[W 2023-11-16 19:58:51.373 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-11-16 19:58:51.375 ServerApp] jupyterlab | extension was successfully linked.
[W 2023-11-16 19:58:51.375 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-11-16 19:58:51.376 ServerApp] nbclassic | extension was successfully linked.
[I 2023-11-16 19:58:51.376 ServerApp] nbgitpuller | extension was successfully linked.
[I 2023-11-16 19:58:51.377 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-11-16 19:58:51.523 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-11-16 19:58:51.532 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-11-16 19:58:51.533 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-11-16 19:58:51.534 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-11-16 19:58:51.538 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2023-11-16 19:58:51.539 ServerApp] jupyterhub | extension was successfully loaded.
[I 2023-11-16 19:58:51.540 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-11-16 19:58:51.540 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-11-16 19:58:51.540 LabApp] Extension Manager is 'pypi'.
[I 2023-11-16 19:58:51.541 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-11-16 19:58:51.545 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-11-16 19:58:51.546 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2023-11-16 19:58:51.546 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-11-16 19:58:51.546 ServerApp] Jupyter Server 2.7.3 is running at:
[I 2023-11-16 19:58:51.546 ServerApp] http://jupyter-b:8888/user/b/lab?token=...
[I 2023-11-16 19:58:51.546 ServerApp]     http://127.0.0.1:8888/user/b/lab?token=...
[I 2023-11-16 19:58:51.546 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2023-11-16 19:58:51.549 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

Also if I disable all network policies by setting all networkPolicy.enabled settings in values.yaml to false, I get the same results (or at least I didn’t see a difference):

Hub:

[I 2023-11-16 20:07:40.998 JupyterHub log:191] 200 GET /hub/login (@::ffff:10.244.0.1) 0.96ms
[D 2023-11-16 20:07:42.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.36ms
[D 2023-11-16 20:07:43.312 JupyterHub roles:281] Assigning default role to User c
[I 2023-11-16 20:07:43.321 JupyterHub roles:238] Adding role user for User: c
[D 2023-11-16 20:07:43.329 JupyterHub roles:281] Assigning default role to User c
[D 2023-11-16 20:07:43.330 JupyterHub base:587] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2023-11-16 20:07:43.330 JupyterHub base:591] Setting cookie for c: jupyterhub-hub-login
[D 2023-11-16 20:07:43.330 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2023-11-16 20:07:43.330 JupyterHub base:837] User logged in: c
[D 2023-11-16 20:07:43.330 JupyterHub user:431] Creating <class 'kubespawner.spawner.KubeSpawner'> for c:
[I 2023-11-16 20:07:43.331 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (c@::ffff:10.244.0.1) 21.55ms
[D 2023-11-16 20:07:43.338 JupyterHub base:299] Recording first activity for <User(c 0/1 running)>
[D 2023-11-16 20:07:43.341 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 20:07:43.341 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 20:07:43.346 JupyterHub pages:209] Serving options form for c
[I 2023-11-16 20:07:43.347 JupyterHub log:191] 200 GET /hub/spawn (c@::ffff:10.244.0.1) 9.53ms
[D 2023-11-16 20:07:44.037 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.76ms
[D 2023-11-16 20:07:44.242 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 20:07:44.242 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 20:07:44.242 JupyterHub pages:257] Triggering spawn with supplied form options for c
[D 2023-11-16 20:07:44.242 JupyterHub base:961] Initiating spawn for c
[D 2023-11-16 20:07:44.242 JupyterHub base:965] 1/64 concurrent spawns
[D 2023-11-16 20:07:44.242 JupyterHub base:970] 1 active servers
[I 2023-11-16 20:07:44.252 JupyterHub provider:659] Creating oauth client jupyterhub-user-c
[D 2023-11-16 20:07:44.259 JupyterHub user:794] Calling Spawner.start for c
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3199] Applying KubeSpawner override for profile '<big>Tensorflow 2.10 ohne GPU Support</big>'
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3177] .. overriding KubeSpawner value extra_labels={'hub.jupyter.org/network-access-hub': 'true'}
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3177] .. overriding KubeSpawner value namespace=other-namespace
[I 2023-11-16 20:07:44.259 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/c?_xsrf=[secret] (c@::ffff:10.244.0.1) 18.95ms
[I 2023-11-16 20:07:44.261 JupyterHub spawner:2497] Attempting to create pod jupyter-c, with timeout 3
[D 2023-11-16 20:07:44.264 JupyterHub scopes:877] Checking access to /hub/spawn-pending/c via scope servers
[D 2023-11-16 20:07:44.264 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/c via servers
[I 2023-11-16 20:07:44.264 JupyterHub pages:398] c is pending spawn
[I 2023-11-16 20:07:44.265 JupyterHub log:191] 200 GET /hub/spawn-pending/c?_xsrf=[secret] (c@::ffff:10.244.0.1) 1.59ms
[D 2023-11-16 20:07:44.303 JupyterHub scopes:877] Checking access to /hub/api/users/c/server/progress via scope read:servers
[D 2023-11-16 20:07:44.304 JupyterHub scopes:690] Argument-based access to /hub/api/users/c/server/progress via read:servers
[D 2023-11-16 20:07:44.304 JupyterHub spawner:2296] progress generator: jupyter-c
[D 2023-11-16 20:07:46.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.34ms
[I 2023-11-16 20:07:46.428 JupyterHub log:191] 200 GET /hub/api (@10.244.0.13) 0.32ms
[D 2023-11-16 20:07:46.443 JupyterHub base:299] Recording first activity for <APIToken('ad22...', user='c', client_id='jupyterhub')>
[D 2023-11-16 20:07:46.447 JupyterHub scopes:877] Checking access to /hub/api/users/c/activity via scope users:activity
[D 2023-11-16 20:07:46.447 JupyterHub scopes:690] Argument-based access to /hub/api/users/c/activity via users:activity
[D 2023-11-16 20:07:46.448 JupyterHub users:874] Activity for user c: 2023-11-16T20:07:46.416655Z
[D 2023-11-16 20:07:46.448 JupyterHub users:892] Activity on server c/: 2023-11-16T20:07:46.416655Z
[I 2023-11-16 20:07:46.450 JupyterHub log:191] 200 POST /hub/api/users/c/activity (c@10.244.0.13) 7.33ms
[D 2023-11-16 20:07:48.037 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.91ms
[D 2023-11-16 20:07:50.039 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.87ms
[D 2023-11-16 20:07:52.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.38ms
[D 2023-11-16 20:07:54.040 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 1.18ms
[D 2023-11-16 20:07:56.035 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.38ms
[D 2023-11-16 20:07:58.038 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 1.05ms
[D 2023-11-16 20:07:58.903 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2023-11-16 20:07:58.909 JupyterHub proxy:392] Checking routes
[D 2023-11-16 20:08:00.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.75ms
[D 2023-11-16 20:08:02.037 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 1.11ms
[D 2023-11-16 20:08:04.039 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.72ms

User-Pod:

[I 2023-11-16 20:07:40.998 JupyterHub log:191] 200 GET /hub/login (@::ffff:10.244.0.1) 0.96ms
[D 2023-11-16 20:07:42.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.36ms
[D 2023-11-16 20:07:43.312 JupyterHub roles:281] Assigning default role to User c
[I 2023-11-16 20:07:43.321 JupyterHub roles:238] Adding role user for User: c
[D 2023-11-16 20:07:43.329 JupyterHub roles:281] Assigning default role to User c
[D 2023-11-16 20:07:43.330 JupyterHub base:587] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2023-11-16 20:07:43.330 JupyterHub base:591] Setting cookie for c: jupyterhub-hub-login
[D 2023-11-16 20:07:43.330 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2023-11-16 20:07:43.330 JupyterHub base:837] User logged in: c
[D 2023-11-16 20:07:43.330 JupyterHub user:431] Creating <class 'kubespawner.spawner.KubeSpawner'> for c:
[I 2023-11-16 20:07:43.331 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (c@::ffff:10.244.0.1) 21.55ms
[D 2023-11-16 20:07:43.338 JupyterHub base:299] Recording first activity for <User(c 0/1 running)>
[D 2023-11-16 20:07:43.341 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 20:07:43.341 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 20:07:43.346 JupyterHub pages:209] Serving options form for c
[I 2023-11-16 20:07:43.347 JupyterHub log:191] 200 GET /hub/spawn (c@::ffff:10.244.0.1) 9.53ms
[D 2023-11-16 20:07:44.037 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.76ms
[D 2023-11-16 20:07:44.242 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2023-11-16 20:07:44.242 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-11-16 20:07:44.242 JupyterHub pages:257] Triggering spawn with supplied form options for c
[D 2023-11-16 20:07:44.242 JupyterHub base:961] Initiating spawn for c
[D 2023-11-16 20:07:44.242 JupyterHub base:965] 1/64 concurrent spawns
[D 2023-11-16 20:07:44.242 JupyterHub base:970] 1 active servers
[I 2023-11-16 20:07:44.252 JupyterHub provider:659] Creating oauth client jupyterhub-user-c
[D 2023-11-16 20:07:44.259 JupyterHub user:794] Calling Spawner.start for c
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3199] Applying KubeSpawner override for profile '<big>Tensorflow 2.10 ohne GPU Support</big>'
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3177] .. overriding KubeSpawner value extra_labels={'hub.jupyter.org/network-access-hub': 'true'}
[D 2023-11-16 20:07:44.259 JupyterHub spawner:3177] .. overriding KubeSpawner value namespace=other-namespace
[I 2023-11-16 20:07:44.259 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/c?_xsrf=[secret] (c@::ffff:10.244.0.1) 18.95ms
[I 2023-11-16 20:07:44.261 JupyterHub spawner:2497] Attempting to create pod jupyter-c, with timeout 3
[D 2023-11-16 20:07:44.264 JupyterHub scopes:877] Checking access to /hub/spawn-pending/c via scope servers
[D 2023-11-16 20:07:44.264 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/c via servers
[I 2023-11-16 20:07:44.264 JupyterHub pages:398] c is pending spawn
[I 2023-11-16 20:07:44.265 JupyterHub log:191] 200 GET /hub/spawn-pending/c?_xsrf=[secret] (c@::ffff:10.244.0.1) 1.59ms
[D 2023-11-16 20:07:44.303 JupyterHub scopes:877] Checking access to /hub/api/users/c/server/progress via scope read:servers
[D 2023-11-16 20:07:44.304 JupyterHub scopes:690] Argument-based access to /hub/api/users/c/server/progress via read:servers
[D 2023-11-16 20:07:44.304 JupyterHub spawner:2296] progress generator: jupyter-c
[D 2023-11-16 20:07:46.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.34ms
[I 2023-11-16 20:07:46.428 JupyterHub log:191] 200 GET /hub/api (@10.244.0.13) 0.32ms
[D 2023-11-16 20:07:46.443 JupyterHub base:299] Recording first activity for <APIToken('ad22...', user='c', client_id='jupyterhub')>
[D 2023-11-16 20:07:46.447 JupyterHub scopes:877] Checking access to /hub/api/users/c/activity via scope users:activity
[D 2023-11-16 20:07:46.447 JupyterHub scopes:690] Argument-based access to /hub/api/users/c/activity via users:activity
[D 2023-11-16 20:07:46.448 JupyterHub users:874] Activity for user c: 2023-11-16T20:07:46.416655Z
[D 2023-11-16 20:07:46.448 JupyterHub users:892] Activity on server c/: 2023-11-16T20:07:46.416655Z
[I 2023-11-16 20:07:46.450 JupyterHub log:191] 200 POST /hub/api/users/c/activity (c@10.244.0.13) 7.33ms
[D 2023-11-16 20:07:48.037 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.91ms
[D 2023-11-16 20:07:50.039 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.87ms
[D 2023-11-16 20:07:52.036 JupyterHub log:191] 200 GET /hub/health (@10.244.0.1) 0.38ms

Do you have an idea what else I can try?

4

The multi-namespace configuration isn’t tested as well as the standard configuration, so it’s possible some bugs have crept in somewhere.

Once thing worth trying is giving the JupyterHub service account full admin privileges- if it still doesn’t work then we know there’s some other problem.

Another thing worth trying is to get rid of Calico and use the default network provider, or to try a standard public cloud Kubernetes deployment, to rule out any K8s configuration issues.

1 Like
5

After I gave the hub ServiceAccount full admin privileges it works!
kubectl create clusterrolebinding cluster_role_binding_jupyterhub_hub --clusterrole=cluster-admin --serviceaccount=jupyterhub-system:hub
Even with all networkPolicies enabled and Calico.

Can you give me some insight, why it works with the ClusterRole and why my RoleBinding for the other-namespace wasn’t enough?
Role+RoleBinding other-namespace

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: other-namespace
  name: manage-pvc-and-pods
rules:
  - apiGroups: [""]
    resources:
      - persistentvolumeclaims
      - pods
    verbs:
      - create
      - read
      - update
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manage-pvc-and-pods
  namespace: other-namespace
subjects:
  - kind: ServiceAccount
    name: hub 
    namespace: jupyterhub-system
roleRef:
  kind: Role 
  name: manage-pvc-and-pods
  apiGroup: rbac.authorization.k8s.io

Are there any downsides to just giving the hub ServiceAccount full cluster-admin privileges?

Update: I was able to resolve this by adding all verbs to the Role.
When I started this and had no Roles or RoleBindings I got an error, when the hub ServiceAccount tried to create pods or pvcs in the UI. After I created the first Role+RoleBinding above with only CRUD-verbs for the specified other-namespace I got no errors anymore, but apparently it wasn’t sufficient for the hub. With the wildcard for the verbs it’s now working even without the ClusterRole.

New Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: other-namespace
  name: manage-pods-and-pvcs
rules:
  - apiGroups: [""]
    resources: ["pods", "persistentvolumeclaims"]
    verbs: ["*"] # <-- adding all verbs

Thank you so much for your help! I was stuck with this for way to long and ready to give up :sweat_smile:

6

Why are these two lines necessary to be able to spawn pods in another ns?

hub.KubeSpawner:
    hub_connect_url: "http://hub.jupyterhub-system.svc.cluster.local:8081"
    environment: { JUPYTERHUB_API_URL: "http://hub.jupyterhub-system.svc.cluster.local:8081/hub/api" }