Hello!
I’m currently going through the process of trying to get systemdspawner running on a CentOS 7 instance.
# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
# systemctl --version
systemd 219
# uname -r
3.10.0-1160.15.2.el7.x86_64
# python --version
Python 3.8.6
# pip list
jupyterhub 1.3.0
jupyterhub-systemdspawner 0.15.0
jupyterlab 2.2.5
I understand that there’s limited functionality due to the version of systemd that’s most likely to remain on the OS till EoL, but I haven’t been able to successfully completely log in against local accounts or those authenticating against LDAP.
We’re coming from a working setup on CentOS 7, but instead using SudoSpawner. We’re looking to switch away from this to take advantage of the memory and CPU cgroup implementation here.
As a note, for testing purposes I’ve disabed SELinux till everything is up and running, then will fight that fight when the time comes. Apologies for what is likely to be a mess here.
Out of the box, I receive the below error after authenticating against LDAP credentials:
jupyterhub: [E 2021-02-11 18:22:20.455 JupyterHub user:638] Unhandled error starting ldap-user's server: [Errno 13] Permission denied: '/run/jupyter-ldap-user-singleuser/jupyter-ldap-user-singleuser.env'
jupyterhub: [W 2021-02-11 18:22:20.482 JupyterHub web:1787] 500 GET /hub/spawn/ldap-user (::ffff:192.168.56.1): Error in Authenticator.pre_spawn_start: PermissionError [Errno 13] Permission denied: '/run/jupyter-ldap-user-singleuser/jupyter-ldap-user-singleuser.env'
And this if I use a local account:
jupyterhub: [W 2021-02-11 18:28:41.744 JupyterHub auth:919] PAM Authentication failed (local-user@::ffff:192.168.56.1): [PAM Error 7] Authentication failure
jupyterhub: [W 2021-02-11 18:28:41.747 JupyterHub base:713] Failed login for local-user
To authenticate with the local account, I need to comment out the CapabilityBoundingSet line in the service file and at that stage, I then get the same Error in Authenticator.pre_spawn_start error above.
After adding /run to ReadWriteDirectories= in the service file, I’m able to authenticate with both accounts, but get stuck at spawning. On the UI Spawn failed: Timeout eventually appears.
jupyterhub: [I 2021-02-11 18:34:44.081 JupyterHub pages:347] local-user is pending spawn
jupyterhub: [I 2021-02-11 18:34:44.086 JupyterHub log:174] 200 GET /hub/spawn-pending/local-user (vagrant@::ffff:192.168.56.1) 10.46ms
jupyterhub: [W 2021-02-11 18:34:53.028 JupyterHub base:947] User local-user is slow to start (timeout=10)
I’ve added some parameters to the JupyterHub config to see if that’ll resolve.
With c.SystemdSpawner.readwrite_paths = ['/home/{USERNAME}'] configured, I see:
Feb 11 18:11:09 sysdspawner jupyterhub: Unknown assignment ReadWriteDirectories=/home/local-user.
Feb 11 18:11:09 sysdspawner jupyterhub: Failed to create bus message: No such device or address
c.SystemdSpawner.user_workingdir = '/home/{USERNAME}' is also specified with no change.
Also tried are
c.PAMAuthenticator.open_sessions = False
c.SystemdSpawner.dynamic_users = True
But all this results in is the below appearing in the logs. I’ve enabled debugging in various parameters, but nothing comes to light.
Feb 11 17:53:22 sysdspawner jupyterhub: Unknown assignment DynamicUser=yes.
Feb 11 17:53:22 sysdspawner jupyterhub: Failed to create bus message: No such device or address
Is there a step that I’m missing here at all? I’ve been able to get things working on Debian Buster fine using the same setup, but our infrastructure is built on CentOS for just now.
Thanks!