Error writing /proc/self/loginuid: Operation not permitted / set_loginuid failed

JupyterHub
1

After following Install JupyterHub and JupyterLab from the ground up line for line, I successfully manage to navigate to JupyterHub in my browser, and to log in, but then I get:

Spawn failed: Server at http://127.0.0.1:48387/jupyter/user/ubuntu/ didn't respond in 30 seconds

Experiment 1

I dug into systemctl status logs:

Dec 02 18:55:49 ip-172-31-16-128 python[2385]: pam_loginuid(login:session): Error writing /proc/self/loginuid: Operation not permitted
Dec 02 18:55:49 ip-172-31-16-128 python[2385]: pam_loginuid(login:session): set_loginuid failed
Dec 02 18:55:50 ip-172-31-16-128 python[2385]: pam_unix(login:session): session opened for user ubuntu by (uid=0)
Dec 02 18:55:50 ip-172-31-16-128 jupyterhub[2385]: [W 2020-12-02 18:55:50.014 JupyterHub auth:1065] Failed to open PAM session for ubuntu: [PAM Error 14] Cannot make/remove an entry for the specified session
Dec 02 18:55:50 ip-172-31-16-128 jupyterhub[2385]: [W 2020-12-02 18:55:50.014 JupyterHub auth:1066] Disabling PAM sessions from now on.
Dec 02 18:55:50 ip-172-31-16-128 jupyterhub[2385]: [I 2020-12-02 18:55:50.025 JupyterHub spawner:1455] Spawning jupyterhub-singleuser --port=48387
Dec 02 18:55:50 ip-172-31-16-128 jupyterhub[2385]: [I 2020-12-02 18:55:50.451 JupyterHub log:181] 302 GET /jupyter/hub/spawn/ubuntu -> /jupyter/hub/spawn-pending/ubuntu (ubuntu@87.110.171.29) 1004.63ms

It seems there is an issue with the spawner / PAM / permissions? I’m completley new to this so I’m struggling to untangle on my own.

Experiment 2

I also tried turning off the service and instead running jupyterhub from the terminal with venv enabled. This leads to a different error:

[C 2020-12-03 07:24:18.606 SingleUserNotebookApp application:89] Bad config encountered during initialization: The 'kernel_spec_manager_class' trait of <jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.SingleUserNotebookApp object at 0x7f3ed7340090> instance must be a type, but 'environment_kernels.EnvironmentKernelSpecManager' could not be imported

Are the 2 related?

Experiment 3

To make things even weirder, running a global install of jupyterhub:

python3 -m pip install jupyterhub

…and then starting the server not using systemctl but by simply typing jupyterhub into terminal - solves the issue.

What is happening?

My setup:

Ubuntu 18.04

My local environment:

jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.7.0
jupyter-telemetry==0.1.0
jupyterhub==1.2.2
jupyterlab==2.2.9
jupyterlab-pygments==0.1.2
jupyterlab-server==1.2.0

My service config file:

[Unit]
Description=JupyterHub
After=syslog.target network.target

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/ubuntu/jhub/.venv/bin"
ExecStart=/home/ubuntu/jhub/.venv/bin/jupyterhub -f /home/ubuntu/jhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target
2

Solved the issue.

The crux was:

[C 2020-12-03 07:24:18.606 SingleUserNotebookApp application:89] Bad config encountered during initialization: The 'kernel_spec_manager_class' trait of <jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.SingleUserNotebookApp object at 0x7f3ed7340090> instance must be a type, but 'environment_kernels.EnvironmentKernelSpecManager' could not be imported

It sounded like config was messing things up. I’m not working on a fresh machine and have had jupyter installed before I started trying jupyterhub. Deleting ~/.jupyter solved the issue.

It sounds like some of the previous settings were preventing JH from working properly. Hope helps anyone with this issue in the future.

The only mystery left unsolved is: I’m still getting the below errors in logs:

Error writing /proc/self/loginuid: Operation not permitted / set_loginuid failed

Even though everything seems to be working fine…

3

same issue here, in one machine cleaning previous setting files works (opensuse TW), but in other doesn’t spawn the hub (fedora 35). Same error in both:

Nov 17 21:34:14 localhost python3[2490]: pam_loginuid(login:session): Error writing /proc/self/loginuid: Operation not permitted 
Nov 17 21:34:14 localhost python3[2490]: pam_loginuid(login:session): set_loginuid failed

any solution?