Unable to use systemd spawner on RHEL 7.9

JupyterHub
, , , ,
1

Hello, everyone. We’ve been using sudospawner, but we’d like to switch to systemdspawner to take advantage of per-user CPU/memory constraints.

We’re on RHEL 7.9, and the systemd version is young enough to meet the needs.
tried with systedDspawner 0.17.0 getting these errors

When a user logs in, the session attempts to launch but never competes; instead, it times out. Below is an error. Any suggestions or insights are greatly appreciated!

[E 2023-09-14 04:38:57.851 JupyterHub log:174] 500 GET /hub/spawn-pending/sptvnt (sptvnt@10.229.202.228) 9.20ms
Unknown assignment DynamicUser=yes.
Failed to create bus message: No such device or address
[I 2023-09-14 04:39:00.308 JupyterHub log:174] 302 GET /hub/spawn/sptvnt -> /hub/spawn-pending/sptvnt (sptvnt@10.229.202.228) 1007.43ms
[I 2023-09-14 04:39:00.322 JupyterHub pages:303] sptvnt is pending spawn
[I 2023-09-14 04:39:00.323 JupyterHub log:174] 200 GET /hub/spawn-pending/sptvnt (sptvnt@10.229.202.228) 7.63ms
[W 2023-09-14 04:39:09.308 JupyterHub base:900] User sptvnt is slow to start (timeout=10)
04:42:19.694 [ConfigProxy] info: 200 GET /api/routes 
[I 2023-09-14 04:42:19.696 JupyterHub proxy:319] Checking routes
[W 2023-09-14 04:43:59.337 JupyterHub user:618] sptvnt's server failed to start in 300 seconds, giving up
[E 2023-09-14 04:43:59.388 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /users/anaconda37/lib/python3.7/site-packages/jupyterhub/handlers/base.py:800> exception=TimeoutError('Timeout')> after timeout
    Traceback (most recent call last):
      File "/users/anaconda37/lib/python3.7/site-packages/tornado/gen.py", line 970, in error_callback
        future.result()
      File "/users/anaconda37/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
        await spawn_future
      File "/users/anaconda37/lib/python3.7/site-packages/jupyterhub/user.py", line 642, in spawn
        raise e
      File "/users/anaconda37/lib/python3.7/site-packages/jupyterhub/user.py", line 546, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
    tornado.util.TimeoutError: Timeout

downgraded the systemd spawner version to 0.14.0 yet facing the same issue

2

What version of systemd you are using? I see that RHEL 7.9 uses systemd v219 and it is too old for systemd spawner. It is clear from the error message that directive DynamicUser is unknown to systemd which confirms that your systemd version is old.

1 Like
3

I am using systemd version 219, if i remove this command from c.SystemdSpawner.dynamic_users = True from config file. i am getting this “Unknown assignment: OOMPolicy=continue” error and same timeout error.

i tried downgrading the systemdspawer to 0.14 getting these errors

Screenshot (242)
Screenshot (242)1920×1080 230 KB

4

The errors you have got nothing to do with systemd spawner. Your systemd is too old. You will need to upgrade your OS to RHEL 8/9 for systemd spawner to work.

1 Like
5

is there any other alternate way? does custom resources work to limit resources (cpu and memory) for jupyterhub?. According to official documentations they have mentioned to use systemdspawner.
currently I am using these versions
jupyterhub==1.0.0
jupyterlab==1.0.2

6

If you’re unable to upgrade your OS you might be better off using a containerised deployment, for example GitHub - jupyterhub/dockerspawner: Spawns JupyterHub single user servers in Docker containers

These are very old unsupported versions. Is there a particular reason for using these?

7

Yes, these are very old versions of JupyterHub and JupyterLab. It is better to upgrade them to latest versions.

If you want to impose same restrictions of CPU and memory on all users, maybe you can define a user slice in cgroups that will be applied to all users on the machine?