Description
I am trying to start Notebook via Slurm on JupyterHub.
The Slurm job is running and the Notebook is up and running, but the Hub screen still says “Cluster job Running … Waiting to connect” is still displayed on the Hub screen.
It then times out and exits.
Note that JupyterHub, Slurm, and Slurm compute nodes are all on the same server.
The log and config file are shown below. It would be helpful if you could tell me if there are any missing or incorrect configurations.
What does the following message in the Slurm log have to do with this?
/usr/local/lib/python3.8/dist-packages/batchspawner/singleuser.py:17: RuntimeWarning: coroutine ‘HubAuth._api_request’ was never awaited
Version
jupyterhub 4.0.1
batchspawner 1.2.0
wrapspawner 1.0.1
Config (Use almost defaults)
c = get_config()
c.Application.log_level = ‘DEBUG’
c.ConfigurableHTTPProxy.debug = True
c.JupyterHub.spawner_class = ‘wrapspawner.ProfilesSpawner’
import batchspawner
c.Spawner.start_timeout=300
c.Spawner.http_timeout = 300
c.ProfilesSpawner.profiles = [
(“Local server”, ‘local’, ‘jupyterhub.spawner.LocalProcessSpawner’, {‘ip’:‘0.0.0.0’} ),
(‘batch_Slurm’, ‘singleuser’, ‘batchspawner.SlurmSpawner’ , dict(req_nprocs=‘2’, req_partition=‘workq’, req_runtime=‘0:05:00’, req_memory=‘4gb’)),
(‘batch_Slurm_lab’, ‘lab’, ‘batchspawner.SlurmSpawner’ , dict(req_nprocs=‘2’, req_partition=‘workq’, req_runtime=‘0:05:00’, req_memory=‘4gb’))
]
c.Spawner.args = [ ‘–debug’ ]
c.Spawner.debug = True
LOG
JupyterHub Log
root@r7525ububtu20:/home/hpcs/setup# jupyterhub --debug
[D 2023-07-11 13:28:03.714 JupyterHub application:902] Looking for jupyterhub_config in /home/hpcs/setup
[D 2023-07-11 13:28:03.723 JupyterHub application:923] Loaded config file: /home/hpcs/setup/jupyterhub_config.py
[I 2023-07-11 13:28:03.734 JupyterHub app:2859] Running JupyterHub version 4.0.1
[I 2023-07-11 13:28:03.734 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.0.1
[I 2023-07-11 13:28:03.734 JupyterHub app:2889] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2023-07-11 13:28:03.734 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.1
[I 2023-07-11 13:28:03.744 JupyterHub app:1664] Loading cookie_secret from /home/hpcs/setup/jupyterhub_cookie_secret
[D 2023-07-11 13:28:03.744 JupyterHub app:1833] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2023-07-11 13:28:03.756 JupyterHub orm:1016] database schema version found: 0eee8c825d24
[I 2023-07-11 13:28:03.822 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2023-07-11 13:28:03.822 JupyterHub app:2104] Loading roles into database
[I 2023-07-11 13:28:03.828 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[D 2023-07-11 13:28:03.831 JupyterHub app:2343] Purging expired APITokens
[D 2023-07-11 13:28:03.834 JupyterHub app:2343] Purging expired OAuthCodes
[D 2023-07-11 13:28:03.835 JupyterHub app:2179] Loading role assignments from config
[D 2023-07-11 13:28:03.841 JupyterHub app:2502] Initializing spawners
[D 2023-07-11 13:28:03.843 JupyterHub app:2633] Loaded users:
[I 2023-07-11 13:28:03.843 JupyterHub app:2928] Initialized 0 spawners in 0.001 seconds
[I 2023-07-11 13:28:03.847 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-07-11 13:28:03.847 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-07-11 13:28:03.848 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[W 2023-07-11 13:28:03.848 JupyterHub proxy:625] Found proxy pid file: /home/hpcs/setup/jupyterhub-proxy.pid
[W 2023-07-11 13:28:03.848 JupyterHub proxy:642] Proxy still running at pid=38693
04:28:03.848 [ConfigProxy] warn: Terminated
[W 2023-07-11 13:28:04.849 JupyterHub proxy:662] Stopped proxy at pid=38693
[D 2023-07-11 13:28:04.849 JupyterHub proxy:678] Removing proxy pid file jupyterhub-proxy.pid
[W 2023-07-11 13:28:04.850 JupyterHub proxy:746] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else…
[I 2023-07-11 13:28:04.850 JupyterHub proxy:750] Starting proxy @ http://:8000
[D 2023-07-11 13:28:04.850 JupyterHub proxy:751] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘’, ‘–port’, ‘8000’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http://127.0.0.1:8081/hub/error’, ‘–log-level’, ‘debug’]
[D 2023-07-11 13:28:04.854 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
04:28:05.074 [ConfigProxy] info: Proxying http://*:8000 to (no default)
04:28:05.075 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2023-07-11 13:28:05.086 JupyterHub proxy:787] Proxy started and appears to be up
[D 2023-07-11 13:28:05.087 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:28:05.092 [ConfigProxy] info: 200 GET /api/routes
[I 2023-07-11 13:28:05.092 JupyterHub app:3178] Hub API listening on http://127.0.0.1:8081/hub/
[D 2023-07-11 13:28:05.092 JupyterHub proxy:389] Fetching routes to check
[D 2023-07-11 13:28:05.092 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:28:05.093 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:28:05.093 JupyterHub proxy:392] Checking routes
[I 2023-07-11 13:28:05.093 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081
[D 2023-07-11 13:28:05.093 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
04:28:05.094 [ConfigProxy] info: Adding route / → http://127.0.0.1:8081
04:28:05.095 [ConfigProxy] info: Route added / → http://127.0.0.1:8081
04:28:05.095 [ConfigProxy] info: 201 POST /api/routes/
[I 2023-07-11 13:28:05.095 JupyterHub app:3245] JupyterHub is now running at http://:8000
[D 2023-07-11 13:28:05.095 JupyterHub app:2852] It took 1.384 seconds for the Hub to start
04:28:54.214 [ConfigProxy] debug: PROXY WEB / to http://127.0.0.1:8081
[I 2023-07-11 13:28:54.219 JupyterHub log:191] 302 GET / → /hub/ (@::ffff:192.168.11.11) 0.65ms
04:28:54.220 [ConfigProxy] debug: Not recording activity for status 302 on /
04:28:54.231 [ConfigProxy] debug: PROXY WEB /hub/ to http://127.0.0.1:8081
[D 2023-07-11 13:28:54.239 JupyterHub base:342] Refreshing auth for hpcs
[D 2023-07-11 13:28:54.242 JupyterHub user:431] Creating <class ‘wrapspawner.wrapspawner.ProfilesSpawner’> for hpcs:
[W 2023-07-11 13:28:54.243 JupyterHub spawner:170]
The shared database session at Spawner.db is deprecated, and will be removed.
Please manage your own database and connections.
Contact JupyterHub at Revoke db access from Authenticator, Spawner · Issue #3700 · jupyterhub/jupyterhub · GitHub
if you have questions or ideas about direct database needs for your Spawner.
[I 2023-07-11 13:28:54.245 JupyterHub log:191] 302 GET /hub/ → /hub/spawn (hpcs@::ffff:192.168.11.11) 13.13ms
04:28:54.245 [ConfigProxy] debug: Not recording activity for status 302 on /
04:28:54.281 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://127.0.0.1:8081
[D 2023-07-11 13:28:54.283 JupyterHub scopes:863] Checking access via scope servers
[D 2023-07-11 13:28:54.283 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-07-11 13:28:54.283 JupyterHub pages:209] Serving options form for hpcs
[I 2023-07-11 13:28:54.302 JupyterHub log:191] 200 GET /hub/spawn (hpcs@::ffff:192.168.11.11) 20.40ms
04:28:57.857 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://127.0.0.1:8081
[D 2023-07-11 13:28:57.859 JupyterHub scopes:863] Checking access via scope servers
[D 2023-07-11 13:28:57.859 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2023-07-11 13:28:57.859 JupyterHub pages:257] Triggering spawn with supplied form options for hpcs
[D 2023-07-11 13:28:57.860 JupyterHub base:959] Initiating spawn for hpcs
[D 2023-07-11 13:28:57.860 JupyterHub base:963] 0/100 concurrent spawns
[D 2023-07-11 13:28:57.860 JupyterHub base:968] 0 active servers
[I 2023-07-11 13:28:57.873 JupyterHub provider:659] Creating oauth client jupyterhub-user-hpcs
[D 2023-07-11 13:28:57.880 JupyterHub user:794] Calling Spawner.start for hpcs
[I 2023-07-11 13:28:57.884 JupyterHub batchspawner:291] Spawner submitting job using sudo -E -u hpcs sbatch --parsable
[I 2023-07-11 13:28:57.884 JupyterHub batchspawner:292] Spawner submitted script:
#!/bin/bash
#SBATCH --output=/home/hpcs/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH --chdir=/home/hpcs
#SBATCH --export=PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_DEBUG,USER,HOME,SHELL
#SBATCH --get-user-env=L
#SBATCH --partition=workq
#SBATCH --time=0:05:00
#SBATCH --mem=4gb
#SBATCH --cpus-per-task=2
set -euo pipefail
trap ‘echo SIGTERM received’ TERM
which jupyterhub-singleuser
srun batchspawner-singleuser jupyterhub-singleuser --debug
echo “jupyterhub-singleuser ended gracefully”
[I 2023-07-11 13:28:57.897 JupyterHub batchspawner:295] Job submitted. cmd: sudo -E -u hpcs sbatch --parsable output: 73
[D 2023-07-11 13:28:57.897 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
[D 2023-07-11 13:28:57.908 JupyterHub batchspawner:436] Job 73 still pending
[D 2023-07-11 13:28:58.409 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
[D 2023-07-11 13:28:58.419 JupyterHub batchspawner:436] Job 73 still pending
[W 2023-07-11 13:28:58.861 JupyterHub base:183] Rolling back dirty objects IdentitySet([<Server(:0)>])
04:28:58.863 [ConfigProxy] debug: Not recording activity for status 302 on /
[I 2023-07-11 13:28:58.865 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] → /hub/spawn-pending/hpcs?_xsrf=[secret] (hpcs@::ffff:192.168.11.11) 1004.83ms
04:28:58.873 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/hpcs to http://127.0.0.1:8081
[D 2023-07-11 13:28:58.876 JupyterHub scopes:863] Checking access via scope servers
[D 2023-07-11 13:28:58.876 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/hpcs via servers
[I 2023-07-11 13:28:58.877 JupyterHub pages:398] hpcs is pending spawn
[I 2023-07-11 13:28:58.880 JupyterHub log:191] 200 GET /hub/spawn-pending/hpcs?_xsrf=[secret] (hpcs@::ffff:192.168.11.11) 6.50ms
[D 2023-07-11 13:28:58.920 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
04:28:58.944 [ConfigProxy] debug: PROXY WEB /hub/api/users/hpcs/server/progress to http://127.0.0.1:8081
[D 2023-07-11 13:28:58.947 JupyterHub scopes:863] Checking access via scope read:servers
[D 2023-07-11 13:28:58.947 JupyterHub scopes:690] Argument-based access to /hub/api/users/hpcs/server/progress via read:servers
[I 2023-07-11 13:29:01.255 JupyterHub log:191] 200 GET /hub/api (@127.0.0.1) 0.40ms
[D 2023-07-11 13:29:01.271 JupyterHub base:297] Recording first activity for <APIToken(‘10ed…’, user=‘hpcs’, client_id=‘jupyterhub’)>
[D 2023-07-11 13:29:01.275 JupyterHub scopes:863] Checking access via scope users:activity
[D 2023-07-11 13:29:01.275 JupyterHub scopes:690] Argument-based access to /hub/api/users/hpcs/activity via users:activity
[D 2023-07-11 13:29:01.276 JupyterHub users:874] Activity for user hpcs: 2023-07-11T04:29:01.241685Z
[D 2023-07-11 13:29:01.276 JupyterHub users:892] Activity on server hpcs/: 2023-07-11T04:29:01.241685Z
[I 2023-07-11 13:29:01.279 JupyterHub log:191] 200 POST /hub/api/users/hpcs/activity (hpcs@127.0.0.1) 9.70ms
[W 2023-07-11 13:29:07.862 JupyterHub base:1088] User hpcs is slow to start (timeout=10)
[D 2023-07-11 13:33:05.096 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:33:05.097 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:33:05.097 JupyterHub proxy:392] Checking routes
[W 2023-07-11 13:33:57.881 JupyterHub user:876] hpcs’s server failed to start in 300 seconds, giving up.
Common causes of this timeout, and debugging tips:
1. Everything is working, but it took too long.
To fix: increase Spawner.start_timeout configuration
to a number of seconds that is enough for spawners to finish starting.
2. The server didn’t finish starting,
or it crashed due to a configuration issue.
Check the single-user server’s logs for hints at what needs fixing.
[D 2023-07-11 13:33:57.882 JupyterHub user:982] Stopping hpcs
[D 2023-07-11 13:33:57.882 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
[I 2023-07-11 13:33:57.894 JupyterHub batchspawner:477] Stopping server job 73
[I 2023-07-11 13:33:57.894 JupyterHub batchspawner:355] Cancelling job 73: sudo -E -u hpcs scancel 73
[D 2023-07-11 13:33:57.907 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
[D 2023-07-11 13:33:58.919 JupyterHub batchspawner:322] Spawner querying job: sudo -E -u hpcs squeue -h -j 73 -o ‘%T %B’
[D 2023-07-11 13:33:58.939 JupyterHub user:1002] Deleting oauth client jupyterhub-user-hpcs
[D 2023-07-11 13:33:58.943 JupyterHub user:1005] Finished stopping hpcs
[E 2023-07-11 13:33:58.946 JupyterHub gen:630] Exception in Future <Task finished name=‘Task-79’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:979> exception=TimeoutError(‘Timeout’)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/tornado/gen.py”, line 625, in error_callback
future.result()
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py”, line 986, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py”, line 902, in spawn
raise e
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py”, line 798, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
asyncio.exceptions.TimeoutError: Timeout
[I 2023-07-11 13:33:58.947 JupyterHub log:191] 200 GET /hub/api/users/hpcs/server/progress?_xsrf=[secret] (hpcs@::ffff:192.168.11.11) 300002.02ms
[D 2023-07-11 13:38:05.096 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:38:05.097 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:38:05.098 JupyterHub proxy:392] Checking routes
[D 2023-07-11 13:43:05.096 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:43:05.097 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:43:05.097 JupyterHub proxy:392] Checking routes
[D 2023-07-11 13:48:05.095 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:48:05.096 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:48:05.096 JupyterHub proxy:392] Checking routes
[D 2023-07-11 13:53:05.096 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
04:53:05.097 [ConfigProxy] info: 200 GET /api/routes
[D 2023-07-11 13:53:05.097 JupyterHub proxy:392] Checking routes
Slurm Log
hpcs@r7525ububtu20:~$ cat jupyterhub_slurmspawner_73.log
/usr/local/bin/jupyterhub-singleuser
/usr/local/lib/python3.8/dist-packages/batchspawner/singleuser.py:17: RuntimeWarning: coroutine ‘HubAuth._api_request’ was never awaited
hub_auth._api_request(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[D 2023-07-11 13:29:00.634 ServerApp] Searching [‘/home/hpcs/.jupyter’, ‘/home/hpcs/.local/etc/jupyter’, ‘/usr/etc/jupyter’, ‘/usr/local/etc/jupyter’, ‘/etc/jupyter’] for config files
[D 2023-07-11 13:29:00.634 ServerApp] Looking for jupyter_config in /etc/jupyter
[D 2023-07-11 13:29:00.634 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-07-11 13:29:00.634 ServerApp] Looking for jupyter_config in /usr/etc/jupyter
[D 2023-07-11 13:29:00.634 ServerApp] Looking for jupyter_config in /home/hpcs/.local/etc/jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_config in /home/hpcs/.jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_server_config in /etc/jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_server_config in /usr/etc/jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_server_config in /home/hpcs/.local/etc/jupyter
[D 2023-07-11 13:29:00.635 ServerApp] Looking for jupyter_server_config in /home/hpcs/.jupyter
[D 2023-07-11 13:29:00.637 ServerApp] Paths used for configuration of jupyter_server_config:
/etc/jupyter/jupyter_server_config.json
[D 2023-07-11 13:29:00.637 ServerApp] Paths used for configuration of jupyter_server_config:
/usr/local/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
/usr/local/etc/jupyter/jupyter_server_config.d/jupyter_server_fileid.json
/usr/local/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
/usr/local/etc/jupyter/jupyter_server_config.d/jupyter_server_ydoc.json
/usr/local/etc/jupyter/jupyter_server_config.d/jupyterlab.json
/usr/local/etc/jupyter/jupyter_server_config.d/nbclassic.json
/usr/local/etc/jupyter/jupyter_server_config.d/notebook_shim.json
/usr/local/etc/jupyter/jupyter_server_config.json
[D 2023-07-11 13:29:00.638 ServerApp] Paths used for configuration of jupyter_server_config:
/usr/etc/jupyter/jupyter_server_config.json
[D 2023-07-11 13:29:00.638 ServerApp] Paths used for configuration of jupyter_server_config:
/home/hpcs/.local/etc/jupyter/jupyter_server_config.json
[D 2023-07-11 13:29:00.638 ServerApp] Paths used for configuration of jupyter_server_config:
/home/hpcs/.jupyter/jupyter_server_config.json
[I 2023-07-11 13:29:00.641 ServerApp] Package jupyterhub took 0.0000s to import
[I 2023-07-11 13:29:00.648 ServerApp] Package jupyter_lsp took 0.0067s to import
[W 2023-07-11 13:29:00.648 ServerApp] A _jupyter_server_extension_points function was not found in jupyter_lsp. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-07-11 13:29:00.650 ServerApp] Package jupyter_server_fileid took 0.0016s to import
[I 2023-07-11 13:29:00.653 ServerApp] Package jupyter_server_terminals took 0.0029s to import
[I 2023-07-11 13:29:00.672 ServerApp] Package jupyter_server_ydoc took 0.0179s to import
[I 2023-07-11 13:29:00.711 ServerApp] Package jupyterlab took 0.0393s to import
[I 2023-07-11 13:29:01.027 ServerApp] Package nbclassic took 0.0000s to import
[W 2023-07-11 13:29:01.028 ServerApp] A _jupyter_server_extension_points function was not found in nbclassic. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-07-11 13:29:01.028 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-07-11 13:29:01.028 ServerApp] A _jupyter_server_extension_points function was not found in notebook_shim. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-07-11 13:29:01.029 ServerApp] jupyter_lsp | extension was successfully linked.
[D 2023-07-11 13:29:01.032 FileIdExtension] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue {‘update’: {‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[D 2023-07-11 13:29:01.033 ServerApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[I 2023-07-11 13:29:01.034 ServerApp] jupyter_server_fileid | extension was successfully linked.
[D 2023-07-11 13:29:01.037 TerminalsExtensionApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[D 2023-07-11 13:29:01.037 ServerApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[I 2023-07-11 13:29:01.038 ServerApp] jupyter_server_terminals | extension was successfully linked.
[D 2023-07-11 13:29:01.041 YDocExtension] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[D 2023-07-11 13:29:01.041 ServerApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[I 2023-07-11 13:29:01.042 ServerApp] jupyter_server_ydoc | extension was successfully linked.
[I 2023-07-11 13:29:01.042 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.1
[I 2023-07-11 13:29:01.042 JupyterHubSingleUser] Using default url from server extension lab: /lab
[D 2023-07-11 13:29:01.043 ServerApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[D 2023-07-11 13:29:01.046 JupyterHubSingleUser] Config changed: {‘ServerApp’: {‘identity_provider_class’: <class ‘jupyterhub.singleuser.extension.JupyterHubIdentityProvider’>, ‘allow_remote_access’: True, ‘open_browser’: False, ‘trust_xheaders’: True, ‘quit_button’: False, ‘port_retries’: 0, ‘answer_yes’: True, ‘port’: 37191, ‘ip’: ‘0.0.0.0’, ‘base_url’: ‘/user/hpcs/’, ‘default_url’: ‘/lab’, ‘keyfile’: ‘’, ‘certfile’: ‘’, ‘client_ca’: ‘’, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘FileContentsManager’: {‘delete_to_trash’: False}, ‘NotebookApp’: {‘extra_template_paths’: <LazyConfigValue {‘extend’: [‘/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/templates’]}>}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
[D 2023-07-11 13:29:01.046 ServerApp] Config changed: {‘ServerApp’: {‘port’: 37191, ‘log_level’: ‘DEBUG’, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterhub’: True, ‘jupyter_lsp’: True, ‘jupyter_server_fileid’: True, ‘jupyter_server_terminals’: True, ‘jupyter_server_ydoc’: True, ‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>, ‘identity_provider_class’: <class ‘jupyterhub.singleuser.extension.JupyterHubIdentityProvider’>, ‘allow_remote_access’: True, ‘open_browser’: False, ‘trust_xheaders’: True, ‘quit_button’: False, ‘port_retries’: 0, ‘answer_yes’: True, ‘ip’: ‘0.0.0.0’, ‘base_url’: ‘/user/hpcs/’, ‘default_url’: ‘/lab’, ‘keyfile’: ‘’, ‘certfile’: ‘’, ‘client_ca’: ‘’}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}, ‘FileContentsManager’: {‘delete_to_trash’: False}}
[I 2023-07-11 13:29:01.047 ServerApp] jupyterhub | extension was successfully linked.
[W 2023-07-11 13:29:01.048 LabApp] ‘extra_template_paths’ was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
…Omitted
[I 2023-07-11 13:29:01.265 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-07-11 13:29:01.265 ServerApp] Serving notebooks from local directory: /home/hpcs
[I 2023-07-11 13:29:01.265 ServerApp] Jupyter Server 2.7.0 is running at:
[I 2023-07-11 13:29:01.265 ServerApp] http://r7525ububtu20:37191/user/hpcs/lab?token=…
[I 2023-07-11 13:29:01.266 ServerApp] http://127.0.0.1:37191/user/hpcs/lab?token=…
[I 2023-07-11 13:29:01.266 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 2023-07-11 13:29:01.268 JupyterHubSingleUser] Notifying Hub of activity 2023-07-11T04:29:01.241685Z
[D 2023-07-11 13:29:01.626 ServerApp] Checking for /home/hpcs/node_modules/bash-language-server/out/cli.js
…Omitted
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** JOB 73 ON r7525ububtu20 CANCELLED AT 2023-07-11T13:33:57 ***
slurmstepd: error: *** STEP 73.0 ON r7525ububtu20 CANCELLED AT 2023-07-11T13:33:57 ***
[C 2023-07-11 13:33:57.907 ServerApp] received signal 15, stopping
[I 2023-07-11 13:33:57.907 ServerApp] Shutting down 8 extensions
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_fileid | extension app ‘jupyter_server_fileid’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_fileid | extension app ‘jupyter_server_fileid’ stopped
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_terminals | extension app ‘jupyter_server_terminals’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_terminals | extension app ‘jupyter_server_terminals’ stopped
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_ydoc | extension app ‘jupyter_server_ydoc’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] jupyter_server_ydoc | extension app ‘jupyter_server_ydoc’ stopped
[D 2023-07-11 13:33:57.908 ServerApp] jupyterhub | extension app ‘jupyterhub-singleuser’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] jupyterhub | extension app ‘jupyterhub-singleuser’ stopped
[D 2023-07-11 13:33:57.908 ServerApp] jupyterlab | extension app ‘lab’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] jupyterlab | extension app ‘lab’ stopped
[D 2023-07-11 13:33:57.908 ServerApp] nbclassic | extension app ‘nbclassic’ stopping
[D 2023-07-11 13:33:57.908 ServerApp] nbclassic | extension app ‘nbclassic’ stopped
SIGTERM received
jupyterhub-singleuser ended gracefully