Cannot spawn a server for new users

jimbo · February 14, 2021, 1:46am

when creating a new user I can log in to jupyterhub, but I can’t spawn a server. I have 2 users that can spawn servers

I’m using jupyterhub installed with pip3 on a remote cluster.

OS:
Ubuntu 18.04.3
Version(s):
jupyterhub 1.3.0, Python 3.6.9,

I’ve tried using both the local spawner and the local simple spawner and have had no luck. I do have a respective user on the linux system for the user on the jupyterhub install

[I 2021-02-12 11:28:44.331 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600
[I 2021-02-12 11:28:49.966 JupyterHub log:181] 302 GET /hub/spawn-pending/rossg → /hub/login?next=%2Fhub%2Fspawn-pending%2Frossg (@::ffff:127.0.0.1) 2.91ms
[I 2021-02-12 11:28:50.033 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2Fspawn-pending%2Frossg (@::ffff:127.0.0.1) 3.61ms
[I 2021-02-12 11:29:00.068 JupyterHub base:757] User logged in: rossg
[I 2021-02-12 11:29:00.069 JupyterHub log:181] 302 POST /hub/login?next=%2Fhub%2Fspawn-pending%2Frossg → /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 35.67ms
[E 2021-02-12 11:29:00.335 JupyterHub pages:380] Previous spawn for rossg failed: Server at http://127.0.0.1:34769/user/rossg/ didn’t respond in 30 seconds
[E 2021-02-12 11:29:00.350 JupyterHub log:173] {
[E 2021-02-12 11:29:00.350 JupyterHub log:181] 500 GET /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 31.69ms
[I 2021-02-12 11:29:03.561 JupyterHub spawner:1451] Spawning jupyterhub-singleuser --port=54521 --notebook-dir=/notebooks
[I 2021-02-12 11:29:04.518 JupyterHub log:181] 302 GET /hub/spawn/rossg → /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 1008.48ms
[I 2021-02-12 11:29:04.581 JupyterHub pages:402] rossg is pending spawn
[I 2021-02-12 11:29:04.584 JupyterHub log:181] 200 GET /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 9.08ms
[I 2021-02-12 11:29:13.187 JupyterHub log:181] 200 POST /hub/api/users/vertex/activity (vertex@127.0.0.1) 19.45ms
[E 2021-02-12 11:29:14.345 JupyterHub service:380] Service idle-culler exited with status 1
[I 2021-02-12 11:29:14.345 JupyterHub service:339] Starting service ‘idle-culler’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’]
[I 2021-02-12 11:29:14.349 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600
[W 2021-02-12 11:29:34.203 JupyterHub user:743] rossg’s server never showed up at http://127.0.0.1:54521/user/rossg/ after 30 seconds. Giving up
[E 2021-02-12 11:29:34.234 JupyterHub gen:624] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user…finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:895> exception=TimeoutError(“Server at http://127.0.0.1:54521/user/rossg/ didn’t respond in 30 seconds”,)> after timeout
[I 2021-02-12 11:29:34.237 JupyterHub log:181] 200 GET /hub/api/users/rossg/server/progress (rossg@::ffff:127.0.0.1) 29554.06ms
[E 2021-02-12 11:29:44.362 JupyterHub service:380] Service idle-culler exited with status 1
[I 2021-02-12 11:29:44.362 JupyterHub service:339] Starting service ‘idle-culler’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’]
[I 2021-02-12 11:29:44.366 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600
[E 2021-02-12 11:30:14.379 JupyterHub service:380] Service idle-culler exited with status 1
[I 2021-02-12 11:30:14.379 JupyterHub service:339] Starting service ‘idle-culler’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’]
[I 2021-02-12 11:30:14.383 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600

1kastner · February 14, 2021, 11:42am

Which authenticator are you using? Which spawner are you using? What does your config look like?

Have you checked the logs? You might have a look at Looking at Logs — The Littlest JupyterHub v0.1 documentation - the suggestiosn are also valid for non-tljh setups.

When I encountered a similar error the last time, the reason was that the user did not have a home directory. As the default spawner wanted to start there but couldn’t, it just ran into a timeout as it can be seen in your example.

jimbo · February 15, 2021, 5:01pm

my config looks like this

import sys

c.Jupyterhub.extra_log_file = ‘/var/log/jupyterhub.log’
c.Spawner.notebook_dir = ‘/notebooks’
c.JupyterHub.spawner_class = ‘jupyterhub.spawner.LocalProcessSpawner’

c.JupyterHub.services = [
{
‘name’: ‘idle-culler’,
‘admin’: True,
‘command’: [sys.executable, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’],
}
]

So im using the local process spawner and (I believe?) the PAM authenticator. I do definitely have users and home directories setup for the users for the users that cant spawn a server. I was thinking, is it possible that my linux and jupyterhub users of the same name might not be properly synced?

when I check that users logs I get

– Logs begin at Tue 2019-09-17 10:25:57 EDT, end at Mon 2021-02-15 12:00:32 EST. –
– No entries –

manics · February 15, 2021, 7:40pm

Can you turn on debug logging, and show us your JupyterHub logs?

jimbo · February 16, 2021, 3:27pm

yes!

[D 2021-02-16 10:23:19.734 JupyterHub base:525] Setting cookie for rossg: jupyterhub-hub-login
[D 2021-02-16 10:23:19.734 JupyterHub base:521] Setting cookie jupyterhub-hub-login: {‘httponly’: True, ‘path’: ‘/hub/’}
[I 2021-02-16 10:23:19.735 JupyterHub base:757] User logged in: rossg
[I 2021-02-16 10:23:19.736 JupyterHub log:181] 302 POST /hub/login?next= → /hub/spawn (rossg@::ffff:127.0.0.1) 38.02ms
[D 2021-02-16 10:23:19.806 JupyterHub pages:218] Triggering spawn with default options for rossg
[D 2021-02-16 10:23:19.807 JupyterHub base:875] Initiating spawn for rossg
[D 2021-02-16 10:23:19.807 JupyterHub base:882] 0/100 concurrent spawns
[D 2021-02-16 10:23:19.807 JupyterHub base:887] 3 active servers
[D 2021-02-16 10:23:19.840 JupyterHub user:602] Calling Spawner.start for rossg
[I 2021-02-16 10:23:19.841 JupyterHub spawner:1451] Spawning jupyterhub-singleuser --port=56907 --notebook-dir=/notebooks --debug
[D 2021-02-16 10:23:19.861 JupyterHub spawner:1147] Polling subprocess every 30s
[I 2021-02-16 10:23:20.810 JupyterHub log:181] 302 GET /hub/spawn → /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 1011.64ms
[I 2021-02-16 10:23:20.868 JupyterHub pages:402] rossg is pending spawn
[D 2021-02-16 10:23:20.871 JupyterHub log:181] 304 GET /hub/spawn-pending/rossg (rossg@::ffff:127.0.0.1) 8.99ms
[E 2021-02-16 10:23:42.901 JupyterHub service:380] Service idle-culler exited with status 1
[I 2021-02-16 10:23:42.902 JupyterHub service:339] Starting service ‘idle-culler’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’]
[I 2021-02-16 10:23:42.905 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600
[D 2021-02-16 10:23:42.913 JupyterHub spawner:1147] Polling subprocess every 30s
[W 2021-02-16 10:23:52.541 JupyterHub user:743] rossg’s server never showed up at http://127.0.0.1:56907/user/rossg/ after 30 seconds. Giving up
[D 2021-02-16 10:23:52.542 JupyterHub user:790] Stopping rossg
[D 2021-02-16 10:23:52.555 JupyterHub user:816] Deleting oauth client jupyterhub-user-rossg
[D 2021-02-16 10:23:52.566 JupyterHub user:819] Finished stopping rossg
[E 2021-02-16 10:23:52.577 JupyterHub gen:624] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:895> exception=TimeoutError(“Server at http://127.0.0.1:56907/user/rossg/ didn’t respond in 30 seconds”,)> after timeout
[I 2021-02-16 10:23:52.579 JupyterHub log:181] 200 GET /hub/api/users/rossg/server/progress (rossg@::ffff:127.0.0.1) 31623.56ms
[D 2021-02-16 10:24:12.812 JupyterHub proxy:795] Proxy: Fetching GET http:// 127.0.0.1:8001/api/routes
[I 2021-02-16 10:24:12.839 JupyterHub proxy:319] Checking routes
[E 2021-02-16 10:24:12.915 JupyterHub service:380] Service idle-culler exited with status 1
[I 2021-02-16 10:24:12.916 JupyterHub service:339] Starting service ‘idle-culler’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’]
[I 2021-02-16 10:24:12.919 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=3600
[D 2021-02-16 10:24:12.928 JupyterHub spawner:1147] Polling subprocess every 30s

1kastner · February 16, 2021, 3:38pm

@jimbo could you also show the logs for the user rossg? See the link of my first answer for an explanation of how to do it.

jimbo · February 16, 2021, 3:53pm

yes

– Logs begin at Mon 2019-09-23 03:08:19 EDT, end at Tue 2021-02-16 10:52:57 EST. –
– No entries –

1kastner · February 16, 2021, 3:58pm

What happens if you try to start the Notebook Server directly, i.e. by running
jupyterhub-singleuser --port=56907 --notebook-dir=/notebooks --debug? Another idea: Do you really have the /notebooks directory?

jimbo · February 16, 2021, 4:46pm

I will try that! and where should the /notebooks directory be? on the root of the system?

1kastner · February 16, 2021, 4:58pm

I’d expect it right there.

jimbo · February 16, 2021, 5:14pm

okay so I dont have that

1kastner · February 16, 2021, 5:28pm

So either you create that folder or you change your config
I guess you just copied the config from somewhere else?

jimbo · February 16, 2021, 5:32pm

yeah, i’ve been having this issue for a while and this was something people suggested. do I need specific permissions on the folder? or just root is okay?

1kastner · February 16, 2021, 5:40pm

I believe it is better to just remove that line. That way every user should end up in their home directories if I remember correctly. If you have a single/notebooks directory, everybody will share the same Notebooks. It will turn into a mess of two or more people edit one of them concurrently.

jimbo · April 19, 2021, 3:37pm

okay i am back to working on this problem, i tried to setup a new user and this is the error that comes up in the debug logs when I try to log in with the new user on my jupyter setup!

I think the error may have to do with new users permissions to access something to spawn a server?

any suggestions

manics · April 19, 2021, 8:14pm

Can you show us your current configuration file with secrets redacted?

jimbo · April 20, 2021, 2:28pm

import sys

c.Jupyterhub.extra_log_file = ‘/var/log/jupyterhub.log’

c.JupyterHub.spawner_class = ‘jupyterhub.spawner.LocalProcessSpawner’

c.JupyterHub.services = [
{
‘name’: ‘idle-culler’,
‘admin’: True,
‘command’: [sys.executable, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=3600’],
}
]

it should probably be noted that the idle culler is not working lol and ive tried using a few different spawners and nothing has changed. also the user IS a user on the linux system and I know jupyterhub recognizes that

manics · April 20, 2021, 3:33pm

Could you stop your JupyterHub server, delete all state including the log file, database, cookie secret, etc. Then start JupyterHub, try and spawn a user, and show us the entire log file?

jimbo · April 20, 2021, 4:41pm

I think so, by delete the database, what do you mean?

manics · April 20, 2021, 7:19pm

JupyterHub creates an SQLite database in its working directory jupyterhub.sqlite as well as a cookie file jupyterhub_cookie_secret

Since this is still a testing environment it’s easiest to delete these files to get things back to a clean start whilst you’re trying to get things to work.

Topic		Replies	Views
Struggling to get the spawners to work JupyterHub jupyterhub , how-to , help-wanted	5	2164	August 30, 2021
Problems spawning single-user-servers (using groups) JupyterHub jupyterhub , help-wanted	6	345	November 30, 2023
SwarmSpawner -- JupyterHub can't connect to user servers JupyterHub help-wanted	1	604	November 5, 2021
Jupyterhub spawn failed error after upgrade JupyterHub	0	537	October 27, 2021
Jupyterhub fails to spawn, singleuser logs indicate a read permission error JupyterHub	5	107	September 23, 2024

Cannot spawn a server for new users

Related topics