Hey everyone. We have been using sudospawner but wish to migrate to systemdspawner to take advantage of cpu/memory limits per user.
For now, I do NOT have any limits in place yet. Pretty much default jupyterhub_config.py file except for SSL cert things etc; no issues with the old spawner.
We are on Centos 8 (not 7…), and the systemd version is new enough to be correct per the requirements.
When a user logs in, the session tries to spawn but never competes; it times. out. Error below. I wholly appreciate any ideas or insight!
May 3 10:39:59 mcgivers jupyterhub[353934]: [I 2023-05-03 10:39:59.402 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (kcb@141.106.28.10) 10.75ms
May 3 10:39:59 mcgivers jupyterhub[353934]: [I 2023-05-03 10:39:59.474 JupyterHub provider:659] Creating oauth client jupyterhub-user-kcb
May 3 10:39:59 mcgivers jupyterhub[353967]: Unknown assignment: OOMPolicy=continue
May 3 10:40:00 mcgivers jupyterhub[353934]: [I 2023-05-03 10:40:00.463 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/kcb (kcb@141.106.28.10) 1003.50ms
May 3 10:40:00 mcgivers jupyterhub[353934]: [I 2023-05-03 10:40:00.515 JupyterHub pages:398] kcb is pending spawn
May 3 10:40:00 mcgivers jupyterhub[353934]: [I 2023-05-03 10:40:00.539 JupyterHub log:191] 200 GET /hub/spawn-pending/kcb (kcb@141.106.28.10) 25.39ms
May 3 10:40:09 mcgivers jupyterhub[353934]: [W 2023-05-03 10:40:09.463 JupyterHub base:1088] User kcb is slow to start (timeout=10)
May 3 10:40:59 mcgivers jupyterhub[353934]: [W 2023-05-03 10:40:59.484 JupyterHub user:876] kcb's server failed to start in 60 seconds, giving up.
May 3 10:40:59 mcgivers jupyterhub[353934]:
May 3 10:40:59 mcgivers jupyterhub[353934]: Common causes of this timeout, and debugging tips:
May 3 10:40:59 mcgivers jupyterhub[353934]:
May 3 10:40:59 mcgivers jupyterhub[353934]: 1. Everything is working, but it took too long.
May 3 10:40:59 mcgivers jupyterhub[353934]: To fix: increase `Spawner.start_timeout` configuration
May 3 10:40:59 mcgivers jupyterhub[353934]: to a number of seconds that is enough for spawners to finish starting.
May 3 10:40:59 mcgivers jupyterhub[353934]: 2. The server didn't finish starting,
May 3 10:40:59 mcgivers jupyterhub[353934]: or it crashed due to a configuration issue.
May 3 10:40:59 mcgivers jupyterhub[353934]: Check the single-user server's logs for hints at what needs fixing.
May 3 10:40:59 mcgivers jupyterhub[353934]:
May 3 10:40:59 mcgivers jupyterhub[353934]: [E 2023-05-03 10:40:59.506 JupyterHub gen:630] Exception in Future <Task finished name='Task-28' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.10/site-packages/jupyterhub/handlers/base.py:979> exception=TimeoutError('Timeout')> after timeout
May 3 10:40:59 mcgivers jupyterhub[353934]: Traceback (most recent call last):
May 3 10:40:59 mcgivers jupyterhub[353934]: File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
May 3 10:40:59 mcgivers jupyterhub[353934]: future.result()
May 3 10:40:59 mcgivers jupyterhub[353934]: File "/usr/local/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 986, in finish_user_spawn
May 3 10:40:59 mcgivers jupyterhub[353934]: await spawn_future
May 3 10:40:59 mcgivers jupyterhub[353934]: File "/usr/local/lib/python3.10/site-packages/jupyterhub/user.py", line 902, in spawn
May 3 10:40:59 mcgivers jupyterhub[353934]: raise e
May 3 10:40:59 mcgivers jupyterhub[353934]: File "/usr/local/lib/python3.10/site-packages/jupyterhub/user.py", line 798, in spawn
May 3 10:40:59 mcgivers jupyterhub[353934]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
May 3 10:40:59 mcgivers jupyterhub[353934]: asyncio.exceptions.TimeoutError: Timeout
May 3 10:40:59 mcgivers jupyterhub[353934]:
May 3 10:40:59 mcgivers jupyterhub[353934]: [I 2023-05-03 10:40:59.507 JupyterHub log:191] 200 GET /hub/api/users/kcb/server/progress?_xsrf=[secret] (kcb@141.106.28.10) 58886.03ms