Hi,
I am trying to deploy the zero to jupyterhub on k8s charts on my k8s cluster (microk8s with calico networking).
I am stuck where I can access the hub UI (I am using domain name so I bind the hub to domain.com/jupyterhub) and I am able to login (with the dummy authenticator) and to start singleuser pod.
But the pod start and after 30s are deleted by the hub as they did no showed up.
(I am using the latest versions of the images)
Here are the logs:
SingleUser pod:
Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[I 2023-08-23 11:36:21.440 ServerApp] Package jupyterlab took 0.0000s to import
[I 2023-08-23 11:36:21.457 ServerApp] Package jupyter_lsp took 0.0164s to import
[W 2023-08-23 11:36:21.457 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.466 ServerApp] Package jupyter_server_terminals took 0.0082s to import
[I 2023-08-23 11:36:21.469 ServerApp] Package nbclassic took 0.0022s to import
[W 2023-08-23 11:36:21.471 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.473 ServerApp] Package nbgitpuller took 0.0014s to import
[I 2023-08-23 11:36:21.473 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-08-23 11:36:21.473 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-08-23 11:36:21.474 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-08-23 11:36:21.480 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-08-23 11:36:21.487 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-08-23 11:36:21.492 ServerApp] nbclassic | extension was successfully linked.
[I 2023-08-23 11:36:21.492 ServerApp] nbgitpuller | extension was successfully linked.
[I 2023-08-23 11:36:21.906 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-08-23 11:36:21.930 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-08-23 11:36:21.934 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-08-23 11:36:21.935 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-08-23 11:36:21.937 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-08-23 11:36:21.937 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-08-23 11:36:21.937 LabApp] Extension Manager is 'pypi'.
[I 2023-08-23 11:36:21.940 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-08-23 11:36:21.944 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-08-23 11:36:21.945 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2023-08-23 11:36:21.946 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-08-23 11:36:21.946 ServerApp] Jupyter Server 2.7.2 is running at:
[I 2023-08-23 11:36:21.946 ServerApp] http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp] http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-08-23 11:36:21.967 ServerApp] No web browser found: Error('could not locate runnable browser').
[C 2023-08-23 11:36:21.967 ServerApp]
To access the server, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
Or copy and paste one of these URLs:
http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.990 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[C 2023-08-23 11:36:54.538 ServerApp] received signal 15, stopping
[I 2023-08-23 11:36:54.551 ServerApp] Shutting down 6 extensions
_ _ _ _
| | | |_ __ __| |__ _| |_ ___
| |_| | '_ \/ _` / _` | _/ -_)
\___/| .__/\__,_\__,_|\__\___|
|_|
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.
https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html
Please note that updating to Notebook 7 might break some of your extensions.
The Hub logs:
[I 2023-08-23 11:36:18.572 JupyterHub spawner:2479] Attempting to create pod jupyter-qweqweqweqw, with timeout 3
[W 2023-08-23 11:36:54.503 JupyterHub user:933] qweqweqweqw's server never showed up at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ after 30 seconds. Giving up.
Common causes of this timeout, and debugging tips:
1. The server didn't finish starting,
or it crashed due to a configuration issue.
Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
This may be a configuration issue specific to your chosen Spawner.
Check the single-user server logs and resource to make sure the URL
is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
To fix: increase `Spawner.http_timeout` configuration
to a number of seconds that is enough for servers to become responsive.
[I 2023-08-23 11:36:54.504 JupyterHub spawner:2810] Deleting pod testxnat/jupyter-qweqweqweqw
[W 2023-08-23 11:36:55.558 JupyterHub base:1057] 4 consecutive spawns failed. Hub will exit if failure count reaches 5 before succeeding
[E 2023-08-23 11:36:55.559 JupyterHub gen:630] Exception in Future <Task finished name='Task-3702' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds")> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
future.result()
File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 914, in spawn
await self._wait_up(spawner)
File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 958, in _wait_up
raise e
File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 928, in _wait_up
resp = await server.wait_up(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
re = await exponential_backoff(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
TimeoutError: Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds
I have a few suspicions on the IP/hostname used:
[I 2023-08-23 11:36:21.946 ServerApp] http://localhost:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
[I 2023-08-23 11:36:21.946 ServerApp] http://127.0.0.1:8888/lab?token=1c4880911e586c7bad9a1b994c21c43a4a7559fe169832c9
This is only using local adresse that are not accessible from outside the container.
TimeoutError: Server at http://10.1.153.22:8888/jupyterhub/user/qweqweqweqw/ didn't respond in 30 seconds
Is using the IP of the singleuser pod which seems strange as it is very very rare to use pod IP in k8s.
Concerning the configuration I haven’t changed anything from the default one expect for the UID/GID of the singleuser user (needed for permissions on files), the ingress configuration, and the proxy service type (I use a ClusterIP as I use an Ingress).
If you have any clue of what is happening.
Thank you.
PS: Don’t mind the username, I have been on the issue for a few hours 