Facing issues when started Jupyterhub with systemd or supervisorctl

JupyterHub
1

I have setup Jupyterhub on my local system and it configured to use gitlab for authentication , whenever I start jupyter on screen it’s working fine without ant issue.
But now I am trying to run Jupyterhub with systemd or supervisor. I configured .service and .ini files also and these are working fine too, when I went on JH UI then I faced 500 internal server error, and I am not able to understand why
Below is the jupyterhub.service file
[Unit]
Description=Jupyterhub

[Service]
Type=simple
User=root
ExecStart=/datadrive/miniconda3/bin/jupyterhub --config /datadrive/jupyter/jupyterhub_config.py --port 5001 --ip 127.0.0.1
WorkingDirectory=/datadrive/jupyter/
Restart=always

[Install]
WantedBy=multi-user.target

Error massage I am getting

500 : Internal Server Error

Unhandled error starting server pramod.lomte.work

You can try restarting your server from the home page.

2

Could you share the JupyterHub systemd service logs? Without logs, it is hard to identify the issue

3

Hi @mahendrapaipuri Below are the logs
[I 2023-11-28 08:30:02.697 JupyterHub log:186] 302 GET /hub/ → /hub/spawn (pramod.lomte.work@106.209.208.125) 21.30ms
/datadrive/miniconda3/lib/python3.11/site-packages/jupyterhub/oauth/provider.py:652: SAWarning: The argument signature for the “ConnectionEvents.engine_connect” event listener has changed as of version 2.0, and conversion for the old argument signature will be removed in a future release. The new signature is “def engine_connect(conn)” (This warning originated from the Session ‘autoflush’ process, which was invoked automatically in response to a user-initiated operation.)
self.db.query(orm.OAuthClient).filter_by(identifier=client_id).one_or_none()
[I 2023-11-28 08:30:02.836 JupyterHub provider:659] Creating oauth client jupyterhub-user-pramod.lomte.work
[I 2023-11-28 08:30:02.908 JupyterHub spawner:1672] Spawning jupyterhub-singleuser
[E 2023-11-28 08:30:02.932 JupyterHub user:833] Unhandled error starting pramod.lomte.work’s server: [Errno 2] No such file or directory: ‘jupyterhub-singleuser’
Traceback (most recent call last):
File “/datadrive/miniconda3/lib/python3.11/site-packages/jupyterhub/user.py”, line 748, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/datadrive/miniconda3/lib/python3.11/site-packages/jupyterhub/spawner.py”, line 1682, in start
self.proc = Popen(cmd, **popen_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/datadrive/miniconda3/lib/python3.11/subprocess.py”, line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File “/datadrive/miniconda3/lib/python3.11/subprocess.py”, line 1950, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: ‘jupyterhub-singleuser’

/datadrive/miniconda3/lib/python3.11/site-packages/jupyterhub/orm.py:664: SAWarning: The argument signature for the “ConnectionEvents.engine_connect” event listener has changed as of version 2.0, and conversion for the old argument signature will be removed in a future release. The new signature is “def engine_connect(conn)” (This warning originated from the Session ‘autoflush’ process, which was invoked automatically in response to a user-initiated operation.)
for orm_token in prefix_match:
[E 2023-11-28 08:30:03.039 JupyterHub pages:311] Error starting server pramod.lomte.work: [Errno 2] No such file or directory: ‘jupyterhub-singleuser’
Traceback (most recent call last):
NoneType: None

[W 2023-11-28 08:30:03.040 JupyterHub web:1869] 500 GET /hub/spawn (106.209.208.125): Unhandled error starting server pramod.lomte.work
[E 2023-11-28 08:30:03.078 JupyterHub log:178] {
“X-Forwarded-Host”: “98.70.25.111”,
“X-Forwarded-Proto”: “http”,
“X-Forwarded-Port”: “80”,
“Cookie”: “jupyterhub-hub-login=[secret]; ajs_anonymous_id=[secret]; ajs_user_id=[secret]; _gcl_au=[secret]; token=[secret]; jupyterhub-session-id=[secret]”,
“Accept-Language”: “en-US,en;q=0.9”,
“Accept-Encoding”: “gzip, deflate”,
“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7”,
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0”,
“Upgrade-Insecure-Requests”: “1”,
“X-Scheme”: “http”,
“Connection”: “close”,
“Host”: “98.70.25.111”,
“X-Forwarded-For”: “106.209.208.125,127.0.0.1”,
“X-Real-Ip”: “106.209.208.125”
}
[E 2023-11-28 08:30:03.078 JupyterHub log:186] 500 GET /hub/spawn (pramod.lomte.work@106.209.208.125) 296.44ms

4

Here’s your problem:

services generally start with an extremely bare PATH, and /datadrive/miniconda3/bin won’t be on it. You likely need to set PATH for the service as well.

For this reason, I usually like to point services to a shell script entrypoint like this instead of the actual program I want to start, so I can more easily load environment variables, profiles, etc. like this.

1 Like
5

Hi @minrk thanks for the reply,

But still I am not getting clear idea how to resolve this

6

For a start, you can use c.Spawner.cmd = ['/datadrive/miniconda3/bin/jupyterhub-singleuser'] in your JupyterHub config.

What @minrk was saying is, when you start services with systemd, it does not have /datadrive/miniconda3/bin on your PATH. So, the service cannot find jupyterhub-singleuser script which is in /datadrive/miniconda3/bin. By using above config, you are providing the full path to jupyterhub-singleuser and in theory it should work. You can use systemd environment variables to set PATH as well. And there are several other ways to do it like @minrk suggested.

1 Like
7

@mahendrapaipuri Thanks, its working now

1 Like