Spawn failed timeout 300 seconds

JupyterHub
1 
I 2025-02-24 16:04:42.078 JupyterHub log:191] 200 POST /hub/api/users/xxxxx/activity (xxxx@10.1.2.128) 50.95ms

[E 2025-02-24 16:04:51.449 JupyterHub gen:630] Exception in Future <Task finished name='Task-500' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError('pod jupyterhub/jupyter-xxx did not start in 300 seconds!')> after timeout

    Traceback (most recent call last):

      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback

        future.result()

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn

        await spawn_future

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 902, in spawn

        raise e

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn

        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)

              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2801, in _start

        await exponential_backoff(

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff

        raise asyncio.TimeoutError(fail_message)

    TimeoutError: pod jupyterhub/jupyter-xxxxx did not start in 300 seconds!
2

repoURL: ‘JupyterHub helm charts | JupyterHub’s Helm chart repository’
targetRevision: 3.2.1

2025-02-24T16:54:16.490194Z [Warning] 0/4 nodes are available: 1 Insufficient cpu, 1 Too many pods, 3 Insufficient memory. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.

2025-02-24T16:54:24Z [Normal] pod triggered scale-up: [{aks-xxxxx-91477950-vmss 3->4 (max: 10)}]

2025-02-24T16:55:28.320968Z [Warning] 0/5 nodes are available: 1 Insufficient cpu, 1 Too many pods, 1 node(s) had volume node affinity conflict, 3 Insufficient memory. preemption: 0/5 nodes are available: 1 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.

2025-02-24T16:55:30.447599Z [Normal] Successfully assigned jupyterhub/jupyter-txxxxx to aks-xxxx-91477950-vmss000008

2025-02-24T16:55:41Z [Normal] AttachVolume.Attach succeeded for volume “pvc-0642ad12-3979-41f8-9f59-7d938974c825”

2025-02-24T16:55:46Z [Normal] Pulling image “xxxxxx”

Spawn failed: Timeout

3 
[W 2025-02-24 16:59:40.250 JupyterHub base:1057] 2 consecutive spawns failed.  Hub will exit if failure count reaches 5 before succeeding

[E 2025-02-24 16:59:40.251 JupyterHub gen:630] Exception in Future <Task finished name='Task-9443' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError('Timeout')> after timeout

    Traceback (most recent call last):

      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback

        future.result()

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn

        await spawn_future

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 902, in spawn

        raise e

      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn

        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)

              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    TimeoutError: Timeout
4

It sounds like your K8s cluster has run out of resources, and/or you’ve got nodes in multiple zones but your volumes are only valid in a single zone.

2 Likes
5

I’ve increase the startTimeout: 600 and i was able to pull the image

1 Like