TimeoutError: Server at http://hub-77b44f7655-ndwqr:36959/user/admin/ didn't respond in 30 seconds

[D 2023-11-15 08:04:14.873 JupyterHub reflector:357] pods watcher timeout
[D 2023-11-15 08:04:14.874 JupyterHub reflector:282] Connecting pods watcher
[D 2023-11-15 08:04:15.878 JupyterHub reflector:357] events watcher timeout
[D 2023-11-15 08:04:15.879 JupyterHub reflector:282] Connecting events watcher
[D 2023-11-15 08:04:16.786 JupyterHub log:189] 200 GET /hub/health (@ 0.72ms
[D 2023-11-15 08:04:18.786 JupyterHub log:189] 200 GET /hub/health (@ 0.69ms
[D 2023-11-15 08:04:20.786 JupyterHub log:189] 200 GET /hub/health (@ 0.89ms
[D 2023-11-15 08:04:22.493 JupyterHub user:845] Deleting oauth client jupyterhub-user-admin
[D 2023-11-15 08:04:22.499 JupyterHub user:848] Finished stopping admin
[W 2023-11-15 08:04:22.504 JupyterHub base:976] 2 consecutive spawns failed. Hub will exit if failure count reaches 10 before succeeding
[E 2023-11-15 08:04:22.504 JupyterHub gen:623] Exception in Future <Task finished name=‘Task-5487’ coro=<BaseHandler.spawn_single_user…finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError(“Server at http://hub-77b44f7655-ndwqr:36959/user/admin/ didn’t respond in 30 seconds”)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/tornado/gen.py”, line 618, in error_callback
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py”, line 907, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py”, line 748, in spawn
await self._wait_up(spawner)
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py”, line 795, in _wait_up
raise e
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py”, line 762, in _wait_up
resp = await server.wait_up(
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py”, line 236, in wait_for_http_server
re = await exponential_backoff(
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py”, line 184, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://hub-77b44f7655-ndwqr:36959/user/admin/ didn’t respond in 30 seconds

[I 2023-11-15 08:04:22.505 JupyterHub log:189] 200 GET /hub/api/users/admin/server/progress (admin@ 51585.47ms
[D 2023-11-15 08:04:22.786 JupyterHub log:189] 200 GET /hub/health (@ 0.83ms
[D 2023-11-15 08:04:24.274 JupyterHub log:189] 200 GET /hub/health (@ 0.77ms
[D 2023-11-15 08:04:24.786 JupyterHub log:189] 200 GET /hub/health (@ 0.86ms

1、I have executed: pip install notebook
2、Why is the requested address this and where can it be configured from? “http://hub-77b44f7655-ndwqr:36959/user/admin/”
3、Why can’t I ping the hub pod’s IP from another pod? is this normal?

The first thing I’d do is check for updates. What version of the helm chart? There is no jupyterhub 1.1.4, so I’m assuming it was 1.4.1, which has been out of date for some years now.

The current version of the chart is 3.1.0 which deploys jupyterhub 4.0.2.