Singleuser pod unable to register to hub on k8s

Hi,
I am trying to deploy the zero to jupyterhub on k8s charts on my k8s cluster (microk8s with calico networking).
I am stuck where I can access the hub UI (I am using domain name so I bind the hub to domain.com/jupyterhub) and I am able to login (with the dummy authenticator) and to start singleuser pod.
But the pod start and after 30s are deleted by the hub as they did no showed up.
(I am using the latest versions of the images)
Here are the logs:
SingleUser pod:

Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[I 2023-08-23 11:36:21.440 ServerApp] Package jupyterlab took 0.0000s to import
[I 2023-08-23 11:36:21.457 ServerApp] Package jupyter_lsp took 0.0164s to import
[W 2023-08-23 11:36:21.457 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.466 ServerApp] Package jupyter_server_terminals took 0.0082s to import
[I 2023-08-23 11:36:21.469 ServerApp] Package nbclassic took 0.0022s to import
[W 2023-08-23 11:36:21.471 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.473 ServerApp] Package nbgitpuller took 0.0014s to import
[I 2023-08-23 11:36:21.473 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-08-23 11:36:21.473 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.474 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-08-23 11:36:21.480 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-08-23 11:36:21.487 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-08-23 11:36:21.492 ServerApp] nbclassic | extension was successfully linked.
[I 2023-08-23 11:36:21.492 ServerApp] nbgitpuller | extension was successfully linked.
[I 2023-08-23 11:36:21.906 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-08-23 11:36:21.930 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-08-23 11:36:21.934 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-08-23 11:36:21.935 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-08-23 11:36:21.937 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-08-23 11:36:21.937 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-08-23 11:36:21.937 LabApp] Extension Manager is 'pypi'.
[I 2023-08-23 11:36:21.940 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-08-23 11:36:21.944 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-08-23 11:36:21.945 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2023-08-23 11:36:21.946 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-08-23 11:36:21.946 ServerApp] Jupyter Server 2.7.2 is running at:
[I 2023-08-23 11:36:21.946 ServerApp] http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp]     http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-08-23 11:36:21.967 ServerApp] No web browser found: Error('could not locate runnable browser').
[C 2023-08-23 11:36:21.967 ServerApp] 
    To access the server, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
        http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.990 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[C 2023-08-23 11:36:54.538 ServerApp] received signal 15, stopping
[I 2023-08-23 11:36:54.551 ServerApp] Shutting down 6 extensions

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|
                                                                           
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.
https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html
Please note that updating to Notebook 7 might break some of your extensions.

The Hub logs:

[I 2023-08-23 11:36:18.572 JupyterHub spawner:2479] Attempting to create pod jupyter-qweqweqweqw, with timeout 3
[W 2023-08-23 11:36:54.503 JupyterHub user:933] qweqweqweqw's server never showed up at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ after 30 seconds. Giving up.
    
    Common causes of this timeout, and debugging tips:
    
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.
    
[I 2023-08-23 11:36:54.504 JupyterHub spawner:2810] Deleting pod testxnat/jupyter-qweqweqweqw
[W 2023-08-23 11:36:55.558 JupyterHub base:1057] 4 consecutive spawns failed.  Hub will exit if failure count reaches 5 before succeeding
[E 2023-08-23 11:36:55.559 JupyterHub gen:630] Exception in Future <Task finished name='Task-3702' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 914, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 958, in _wait_up
        raise e
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 928, in _wait_up
        resp = await server.wait_up(
               ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
        re = await exponential_backoff(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    TimeoutError: Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds

I have a few suspicions on the IP/hostname used:

[I 2023-08-23 11:36:21.946 ServerApp] http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp]     http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9

This is only using local adresse that are not accessible from outside the container.

TimeoutError: Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds

Is using the IP of the singleuser pod which seems strange as it is very very rare to use pod IP in k8s.

Concerning the configuration I haven’t changed anything from the default one expect for the UID/GID of the singleuser user (needed for permissions on files), the ingress configuration, and the proxy service type (I use a ClusterIP as I use an Ingress).

If you have any clue of what is happening.

Thank you.
PS: Don’t mind the username, I have been on the issue for a few hours :slight_smile:

So you are running JupyterHub at a prefix /jupyterhub on your server. How did you configure your JupyterHub to run it at that prefix? The problem here is that your single user server is being run at http://localhost:8888/lab?... instead of http://localhost:8888/jupyterhub/lab?.... Your single user servers should also run at the /jupyterhub prefix for hub to proxy them properly.

If you use c.JupyterHub.base_url = '/jupyterhub' configuration, it should work out-of-the-box.

1 Like

I am actually using the chart value for this:

hub:
    baseUrl: /jupyterhub

I’ll try to add this line to the jupyterhub_config.py.
But it may modify the hub url like jupyterhub/juptyterhub

I dont have experience on Kubernetes deployment of JupyterHub. Maybe you should do a similar config on single user server side as well? If not, just do it in jupyterhub_config.py and remove config on the helm chart and hopefully that will be propagated to single user servers as well.

How did you setup your k8s cluster? Are you using a standard public cloud deployment, or your own customised setup? Have you verified that your k8s network controller is correctly setup?

I am using microk8s with node deployed on VMs, we have our own datacenter.
I have other applications deployed with services and ingresses which are working fine.
We use calico as pod networking.

Have you tried following the debugging suggestions in

?

Can you show us your outputs?

I do not see any additional output with the debut option, but here are the full logs of the pods:
Hub:

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2023-08-24 12:14:42.421 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-08-24 12:14:42.421 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.DummyAuthenticator-4.0.2
[I 2023-08-24 12:14:42.421 JupyterHub app:2889] Using Spawner: kubespawner.spawner.KubeSpawner-6.0.0
[I 2023-08-24 12:14:42.422 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-08-24 12:14:42.454 alembic.runtime.migration migration:213] Context impl SQLiteImpl.
[I 2023-08-24 12:14:42.454 alembic.runtime.migration migration:216] Will assume non-transactional DDL.
[I 2023-08-24 12:14:42.460 alembic.runtime.migration migration:619] Running stamp_revision  -> 0eee8c825d24
[I 2023-08-24 12:14:42.551 JupyterHub roles:172] Role jupyterhub-idle-culler added to database
[I 2023-08-24 12:14:42.552 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-08-24 12:14:42.594 JupyterHub app:2928] Initialized 0 spawners in 0.002 seconds
[I 2023-08-24 12:14:42.600 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-08-24 12:14:42.601 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2023-08-24 12:14:42.602 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2023-08-24 12:14:42.602 JupyterHub app:3142] Not starting proxy
[I 2023-08-24 12:14:42.611 JupyterHub app:3178] Hub API listening on http://:8081/jupyterhub/hub/
[I 2023-08-24 12:14:42.611 JupyterHub app:3180] Private Hub API connect url http://hub:8081/jupyterhub/hub/
[I 2023-08-24 12:14:42.611 JupyterHub app:3189] Starting managed service jupyterhub-idle-culler
[I 2023-08-24 12:14:42.611 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/jupyterhub/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2023-08-24 12:14:42.613 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/jupyterhub/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2023-08-24 12:14:42.616 JupyterHub proxy:477] Adding route for Hub: /jupyterhub/ => http://hub:8081
[I 2023-08-24 12:14:42.620 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://hub:8081/jupyterhub/hub/
[I 2023-08-24 12:14:43.520 JupyterHub log:191] 200 GET /jupyterhub/hub/api/ (jupyterhub-idle-culler@::1) 6.19ms
[I 2023-08-24 12:14:43.531 JupyterHub log:191] 200 GET /jupyterhub/hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 8.29ms
[W 2023-08-24 12:16:23.139 JupyterHub base:415] Invalid or expired cookie token
[I 2023-08-24 12:16:23.141 JupyterHub log:191] 302 GET /jupyterhub/hub/home -> /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2Fhome (@10.254.254.198) 2.78ms
[I 2023-08-24 12:16:23.386 JupyterHub log:191] 200 GET /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2Fhome (@10.254.254.198) 38.51ms
[I 2023-08-24 12:16:27.531 JupyterHub roles:238] Adding role user for User: qwe
[I 2023-08-24 12:16:27.548 JupyterHub base:837] User logged in: qwe
[I 2023-08-24 12:16:27.549 JupyterHub log:191] 302 POST /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2Fhome -> /jupyterhub/hub/home (qwe@10.254.254.198) 32.37ms
[I 2023-08-24 12:16:27.760 JupyterHub log:191] 200 GET /jupyterhub/hub/home (qwe@10.254.254.198) 23.13ms
[I 2023-08-24 12:16:33.872 JupyterHub provider:659] Creating oauth client jupyterhub-user-qwe
[I 2023-08-24 12:16:33.889 JupyterHub log:191] 302 GET /jupyterhub/hub/spawn/qwe -> /jupyterhub/hub/spawn-pending/qwe (qwe@10.254.254.198) 40.04ms
[I 2023-08-24 12:16:33.912 JupyterHub reflector:274] watching for pods with label selector='component=singleuser-server' in namespace testxnat
[I 2023-08-24 12:16:33.918 JupyterHub reflector:274] watching for events with field selector='involvedObject.kind=Pod' in namespace testxnat
[I 2023-08-24 12:16:33.921 JupyterHub spawner:2519] Attempting to create pvc claim-qwe, with timeout 3
[I 2023-08-24 12:16:33.953 JupyterHub spawner:2479] Attempting to create pod jupyter-qwe, with timeout 3
[I 2023-08-24 12:16:34.044 JupyterHub pages:398] qwe is pending spawn
[I 2023-08-24 12:16:34.048 JupyterHub log:191] 200 GET /jupyterhub/hub/spawn-pending/qwe (qwe@10.254.254.198) 6.50ms
[W 2023-08-24 12:17:09.451 JupyterHub user:933] qwe's server never showed up at http://10.1.153.18:8888/jupyterhub/user/qwe/ after 30 seconds. Giving up.

    Common causes of this timeout, and debugging tips:

    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.

[I 2023-08-24 12:17:09.452 JupyterHub spawner:2810] Deleting pod testxnat/jupyter-qwe
[E 2023-08-24 12:17:10.631 JupyterHub gen:630] Exception in Future <Task finished name='Task-261' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://10.1.153.18:8888/jupyterhub/user/qwe/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 914, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 958, in _wait_up
        raise e
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 928, in _wait_up
        resp = await server.wait_up(
               ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
        re = await exponential_backoff(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    TimeoutError: Server at http://10.1.153.18:8888/jupyterhub/user/qwe/ didn't respond in 30 seconds

[I 2023-08-24 12:17:10.636 JupyterHub log:191] 200 GET /jupyterhub/hub/api/users/qwe/server/progress?_xsrf=[secret] (qwe@10.254.254.198) 36369.77ms

SingleUser:

[I 2023-08-24 12:16:37.222 ServerApp] Package jupyterlab took 0.0000s to import
[I 2023-08-24 12:16:37.238 ServerApp] Package jupyter_lsp took 0.0156s to import
[W 2023-08-24 12:16:37.239 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-24 12:16:37.247 ServerApp] Package jupyter_server_terminals took 0.0078s to import
[I 2023-08-24 12:16:37.249 ServerApp] Package nbclassic took 0.0022s to import
[W 2023-08-24 12:16:37.252 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-24 12:16:37.254 ServerApp] Package nbgitpuller took 0.0016s to import
[I 2023-08-24 12:16:37.254 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-08-24 12:16:37.254 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-24 12:16:37.255 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-08-24 12:16:37.259 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-08-24 12:16:37.265 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-08-24 12:16:37.269 ServerApp] nbclassic | extension was successfully linked.
[I 2023-08-24 12:16:37.269 ServerApp] nbgitpuller | extension was successfully linked.
[I 2023-08-24 12:16:37.286 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-08-24 12:16:37.719 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-08-24 12:16:37.744 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-08-24 12:16:37.747 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-08-24 12:16:37.748 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-08-24 12:16:37.750 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-08-24 12:16:37.750 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-08-24 12:16:37.751 LabApp] Extension Manager is 'pypi'.
[I 2023-08-24 12:16:37.754 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-08-24 12:16:37.758 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-08-24 12:16:37.759 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2023-08-24 12:16:37.760 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-08-24 12:16:37.760 ServerApp] Jupyter Server 2.7.2 is running at:
[I 2023-08-24 12:16:37.760 ServerApp] http://localhost:8888/lab?token=d14c245ebe6825a6643655f12bc965da4a6cf807975e9822
[I 2023-08-24 12:16:37.760 ServerApp]     http://127.0.0.1:8888/lab?token=d14c245ebe6825a6643655f12bc965da4a6cf807975e9822
[I 2023-08-24 12:16:37.760 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-08-24 12:16:37.780 ServerApp] No web browser found: Error('could not locate runnable browser').
[C 2023-08-24 12:16:37.781 ServerApp]

    To access the server, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=d14c245ebe6825a6643655f12bc965da4a6cf807975e9822
        http://127.0.0.1:8888/lab?token=d14c245ebe6825a6643655f12bc965da4a6cf807975e9822
[I 2023-08-24 12:16:37.801 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[C 2023-08-24 12:17:09.493 ServerApp] received signal 15, stopping

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2023-08-24 12:17:09.504 ServerApp] Shutting down 6 extensions

So I tried to remove the prefix jupyterhub, as it may be the problem, but it doesn’t changed anything.
Still unable to access a notebook, the singleuser pod is killed after 30s because it “doesn’t showed up”

Single user pods should be able to communicate with hub. When Hub spawn a single user server, after successful spawn, the single user server will have to notify hub that it is running. If that communication never arrives from single user pod, hub assumes that single user server never popped up and thus, kill the pod.

Are you sure that your single user pod can communicate with hub pod? Maybe checking your network config can be a good place to start.

I finally get it working, I was using the chart from the github repo (which seems broken or so), and now using the one from the helm repo with the same values.yaml is working. It may be chartpress which as caused me issues idk.
Thank you for your help anyway.

1 Like