After server reset, TLJH is not able to spawn the user server

JupyterHub
1

Hi,
I needed to reboot the on-premise server (Ubuntu20.04, x86) that hosts the TLJH service, and after that I cannot spawn the users (or admin) servers. The TLJH itself is running. Prior to the reboot I managed to change the frontend to by JupyterLab and change culling timeout to be 1800sec.
I’ve tried stopping it and starting manualy with:

sudo systemctl stop jupyterhub.service
sudo systemctl start jupyterhub.service

This didn’t do a thing :frowning:

I’m seeing this error when checking the logs
sudo journalctl -u jupyterhub

Jul 03 11:28:56 had-1142-lnv python3[5891]: [E 2023-07-03 11:28:56.274 JupyterHub gen:630] Exception in Future <Task finished name=‘Task-8213’ coro=<BaseHandler.spawn_single_user..finis>
Jul 03 11:28:56 had-1142-lnv python3[5891]: Traceback (most recent call last):
Jul 03 11:28:56 had-1142-lnv python3[5891]: File “/opt/tljh/hub/lib/python3.8/site-packages/tornado/gen.py”, line 625, in error_callback
Jul 03 11:28:56 had-1142-lnv python3[5891]: future.result()
Jul 03 11:28:56 had-1142-lnv python3[5891]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py”, line 913, in finish_user_spawn
Jul 03 11:28:56 had-1142-lnv python3[5891]: await spawn_future
Jul 03 11:28:56 had-1142-lnv python3[5891]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py”, line 736, in spawn
Jul 03 11:28:56 had-1142-lnv python3[5891]: raise e
Jul 03 11:28:56 had-1142-lnv python3[5891]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py”, line 635, in spawn
Jul 03 11:28:56 had-1142-lnv python3[5891]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
Jul 03 11:28:56 had-1142-lnv python3[5891]: asyncio.exceptions.TimeoutError: Timeout

The Traefik logs looks without an issue. Any idea what went wrong?

2

Are you sure that JupyterLab is installed in your env? I am not familiar with TLJH but from this comment there are multiple envs and you should ensure that JupyterLab is installed in User env

3

Thanks, yes, I’m sure, before the server reboot we’ve validated that it was the Lab frontend. Do you think that this what might cause the issue?

4

Could you share more logs? For me the TimeOut errors say that hub is trying to start a single-user server but it is failing for some reason. One possible reason is that it cannot find jupyter-lab installed in the user’s env.

5

Lab is installed, this is what I get after activating /opt/tljh/user:
image

What other logs might be usefull?

6

The logs preceding these lines. There were no errors reporting before this? Maybe hub spitted some tracebacks which can be helpful to find the issue.

If you rollback the change that you have made, does hub spawns single user notebooks?

7

Timeout in spawn means the launch likely failed. The unit with the likely useful logs are the singleuser server, e.g. journalctl -u jupyter-username. That should lead you to the right error, which may mean an installation/upgrade/etc.