SlurmSpawner doesnt work 4.1.0

JupyterHub
1

Hello,

I am reaching out to you because we are setting up a new infrastructure and the migration of JupyterHub from version 1 to version 4.1.0 is not working at all. We have a Slurm cluster that runs Singularity containers in which we have added Jupyterhub & batchspawner.

Today, even without the Singularity part, hub not working.

jupyterhub_config.py :

import os
import importlib.machinery

c = get_config()


c.JupyterHub.ip = 'r9jupyter.domain.fr'
c.JupyterHub.hub_ip = 'r9jupyter.domain.fr'
c.JupyterHub.reset_db = True

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
import batchspawner
c.BatchSpawnerBase.req_host = 'r9jupyter.domain.fr'
c.BatchSpawnerBase.req_queue = 'r9jupyter'
c.BatchSpawnerBase.req_runtime = '12:00:00'


bscript = '''#!/bin/bash
#SBATCH --time={runtime}
#SBATCH --output={homedir}/jupyterlab-%j.output
#SBATCH --error={homedir}/jupyterlab-%j.error
#SBATCH --job-name=jupyterlab
#SBATCH --export={keepvars}
#SBATCH --mail-user=william.vincent@domain.fr
#SBATCH --ntasks=1
#SBATCH --partition={partition}
#SBATCH --chdir={homedir}
#SBATCH --get-user-env=L

set -x
#set -euo pipefail

trap 'echo SIGTERM received' TERM

{prologue}

#echo "# With Singularity "
export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/pytorch2-will.sif
#srun singularity exec --env JUPYTER_PORT=0 --bind $PWD:/run/user $SINGULARITYENV_CONTAINER_PATH {cmd} > jupyter.debug 2>&1

#echo "# Without Singularity "
module load Python/3.12.2
{cmd} > jupyter.debug 2>&1

echo "jupyterhub-singleuser ended gracefully"
{epilogue}
echo "# end"
'''

c.ProfilesSpawner.profiles = [
   ('24CPUNodes', '24CPUNodes', 'batchspawner.SlurmSpawner',dict(req_partition='24CPUNodes',batch_script=bscript)),
   ('GPUNodes1080-dev', 'GPUNodes1080-dev', 'batchspawner.SlurmSpawner',dict(req_partition='GPUNodes1080-dev',batch_script=bscript)),
]

# c.ProfilesSpawner.ip = '$HOSTNAME'

#c.Spawner.args = ["--debug" ]
c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~'
# c.Spawner.ip = '$HOSTNAME'
# c.Spawner.port = 0
# c.Spawner.hub_connect_url = "https://r9jupyter.domain.fr" # that not work
c.Spawner.start_timeout = 300
c.Spawner.http_timeout = 300
#c.Spawner.debug = True

First issue : without singularity

If I run it like this, here is the log:

jupyterhub -f /etc/jupyterhub/jupyterhub_config.py --debug
[D 2024-03-20 16:06:16.593 JupyterHub application:908] Looking for /etc/jupyterhub/jupyterhub_config in /etc/jupyterhub
[D 2024-03-20 16:06:16.602 JupyterHub application:929] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2024-03-20 16:06:16.612 JupyterHub app:2885] Running JupyterHub version 4.1.0
[I 2024-03-20 16:06:16.612 JupyterHub app:2915] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.1.0
[I 2024-03-20 16:06:16.613 JupyterHub app:2915] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2024-03-20 16:06:16.613 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.0
[D 2024-03-20 16:06:16.617 JupyterHub app:2842] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2024-03-20 16:06:16.618 JupyterHub app:1683] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[D 2024-03-20 16:06:16.618 JupyterHub app:1852] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2024-03-20 16:06:16.771 JupyterHub orm:997] Stamping empty database with alembic revision 0eee8c825d24
[I 2024-03-20 16:06:16.773 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
[I 2024-03-20 16:06:16.774 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
[I 2024-03-20 16:06:17.193 JupyterHub proxy:557] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2024-03-20 16:06:17.193 JupyterHub app:2125] Loading roles into database
[I 2024-03-20 16:06:17.467 JupyterHub app:2005] Not using allowed_users. Any authenticated user will be allowed.
[D 2024-03-20 16:06:17.472 JupyterHub app:2364] Purging expired APITokens
[D 2024-03-20 16:06:17.477 JupyterHub app:2364] Purging expired OAuthCodes
[D 2024-03-20 16:06:17.479 JupyterHub app:2200] Loading role assignments from config
[D 2024-03-20 16:06:17.486 JupyterHub app:2523] Initializing spawners
[D 2024-03-20 16:06:17.487 JupyterHub app:2658] Loaded users:

[I 2024-03-20 16:06:17.488 JupyterHub app:2954] Initialized 0 spawners in 0.002 seconds
[I 2024-03-20 16:06:17.492 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-03-20 16:06:17.492 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2024-03-20 16:06:17.493 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2024-03-20 16:06:17.493 JupyterHub proxy:747] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2024-03-20 16:06:17.493 JupyterHub proxy:751] Starting proxy @ http://r9jupyter.domain.fr:8000/
[D 2024-03-20 16:06:17.493 JupyterHub proxy:752] Proxy cmd: ['configurable-http-proxy', '--ip', 'r9jupyter.domain.fr', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://r9jupyter.domain.fr:8081/hub/error', '--log-level', 'info']
[D 2024-03-20 16:06:17.498 JupyterHub proxy:671] Writing proxy pid file: jupyterhub-proxy.pid
[D 2024-03-20 16:06:17.499 JupyterHub utils:264] Waiting 10s for server at r9jupyter.domain.fr:8000
[D 2024-03-20 16:06:17.501 JupyterHub utils:111] Server at r9jupyter.domain.fr:8000 not ready: [Errno 111] Connection refused
[D 2024-03-20 16:06:17.501 JupyterHub utils:264] Waiting 10s for server at 127.0.0.1:8001
[D 2024-03-20 16:06:17.501 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-03-20 16:06:17.521 JupyterHub utils:111] Server at r9jupyter.domain.fr:8000 not ready: [Errno 111] Connection refused
[D 2024-03-20 16:06:17.523 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-03-20 16:06:17.552 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
16:06:17.725 [ConfigProxy] info: Proxying http://r9jupyter.domain.fr:8000 to (no default)
16:06:17.727 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-03-20 16:06:17.918 JupyterHub utils:274] Server at r9jupyter.domain.fr:8000 responded in 0.42s
[D 2024-03-20 16:06:17.944 JupyterHub utils:274] Server at 127.0.0.1:8001 responded in 0.44s
[D 2024-03-20 16:06:17.944 JupyterHub proxy:831] Proxy started and appears to be up
[D 2024-03-20 16:06:17.948 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:06:17.965 [ConfigProxy] info: 200 GET /api/routes
[I 2024-03-20 16:06:17.965 JupyterHub app:3204] Hub API listening on http://r9jupyter.domain.fr:8081/hub/
[D 2024-03-20 16:06:17.965 JupyterHub proxy:390] Fetching routes to check
[D 2024-03-20 16:06:17.965 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:06:17.967 [ConfigProxy] info: 200 GET /api/routes
[D 2024-03-20 16:06:17.967 JupyterHub proxy:393] Checking routes
[I 2024-03-20 16:06:17.967 JupyterHub proxy:478] Adding route for Hub: / => http://r9jupyter.domain.fr:8081
[D 2024-03-20 16:06:17.968 JupyterHub proxy:924] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
16:06:17.970 [ConfigProxy] info: Adding route / -> http://r9jupyter.domain.fr:8081
16:06:17.971 [ConfigProxy] info: Route added / -> http://r9jupyter.domain.fr:8081
16:06:17.971 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-03-20 16:06:17.971 JupyterHub app:3271] JupyterHub is now running at http://r9jupyter.domain.fr:8000/
[D 2024-03-20 16:06:17.972 JupyterHub app:2878] It took 1.382 seconds for the Hub to start
[I 2024-03-20 16:06:22.387 JupyterHub log:192] 302 GET / -> /hub/ (@xxx.xxx.4.106) 2.09ms
[W 2024-03-20 16:06:22.467 JupyterHub base:480] Invalid cookie token
[I 2024-03-20 16:06:22.469 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@xxx.xxx.4.106) 9.17ms
[W 2024-03-20 16:06:22.516 JupyterHub base:480] Invalid cookie token
[D 2024-03-20 16:06:22.517 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:1b87285ca9b349d1a8093e334af56e3c' != b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY='
[I 2024-03-20 16:06:22.517 JupyterHub _xsrf_utils:159] Setting new xsrf cookie for b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY=' {'path': '/hub/', 'max_age': 3600}
[I 2024-03-20 16:06:22.544 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@xxx.xxx.4.106) 31.54ms
[D 2024-03-20 16:06:22.965 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 9.43ms
[W 2024-03-20 16:06:31.481 JupyterHub base:480] Invalid cookie token
[D 2024-03-20 16:06:31.599 JupyterHub roles:282] Assigning default role to User wvincent
[I 2024-03-20 16:06:31.606 JupyterHub roles:239] Adding role user for User: wvincent
[D 2024-03-20 16:06:31.670 JupyterHub roles:282] Assigning default role to User wvincent
[W 2024-03-20 16:06:31.681 JupyterHub base:480] Invalid cookie token
[D 2024-03-20 16:06:31.681 JupyterHub base:667] Setting cookie for wvincent: jupyterhub-hub-login
[D 2024-03-20 16:06:31.681 JupyterHub base:663] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-03-20 16:06:31.682 JupyterHub base:925] User logged in: wvincent
[I 2024-03-20 16:06:31.682 JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (wvincent@xxx.xxx.4.106) 204.91ms
[D 2024-03-20 16:06:31.765 JupyterHub base:357] Recording first activity for <User(wvincent 0/1 running)>
[D 2024-03-20 16:06:31.779 JupyterHub user:431] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for wvincent:
[W 2024-03-20 16:06:31.781 JupyterHub spawner:172]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.

    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

[I 2024-03-20 16:06:31.784 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (wvincent@xxx.xxx.4.106) 21.28ms
[D 2024-03-20 16:06:31.850 JupyterHub scopes:884] Checking access to /hub/spawn via scope servers
[D 2024-03-20 16:06:31.850 JupyterHub scopes:697] Argument-based access to /hub/spawn via servers
[D 2024-03-20 16:06:31.851 JupyterHub pages:210] Serving options form for wvincent
[D 2024-03-20 16:06:31.851 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY=' != b'd7d8defe6da747e897e18966ea64dac6:d7aa9d8e0be3471ca0e3c3533feb1b74'
[I 2024-03-20 16:06:31.852 JupyterHub _xsrf_utils:159] Setting new xsrf cookie for b'd7d8defe6da747e897e18966ea64dac6:d7aa9d8e0be3471ca0e3c3533feb1b74' {'path': '/hub/'}
[I 2024-03-20 16:06:31.861 JupyterHub log:192] 200 GET /hub/spawn (wvincent@xxx.xxx.4.106) 14.35ms
[D 2024-03-20 16:06:32.087 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 1.71ms
[D 2024-03-20 16:06:39.246 JupyterHub scopes:884] Checking access to /hub/spawn via scope servers
[D 2024-03-20 16:06:39.246 JupyterHub scopes:697] Argument-based access to /hub/spawn via servers
[D 2024-03-20 16:06:39.247 JupyterHub pages:258] Triggering spawn with supplied form options for wvincent
[D 2024-03-20 16:06:39.248 JupyterHub base:1049] Initiating spawn for wvincent
[D 2024-03-20 16:06:39.248 JupyterHub base:1053] 0/100 concurrent spawns
[D 2024-03-20 16:06:39.248 JupyterHub base:1058] 0 active servers
[I 2024-03-20 16:06:39.476 JupyterHub provider:660] Creating oauth client jupyterhub-user-wvincent
[D 2024-03-20 16:06:39.621 JupyterHub user:797] Calling Spawner.start for wvincent
[I 2024-03-20 16:06:39.628 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cluster': '', 'epilogue': '', 'gres': '', 'homedir': '/home/wvincent', 'host': 'r9jupyter.domain.fr', 'keepvars': 'PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,USER,HOME,SHELL', 'keepvars_extra': '', 'memory': '', 'ngpus': '', 'nprocs': '2', 'options': '', 'partition': '24CPUNodes', 'prologue': '', 'qos': '', 'queue': 'r9jupyter', 'reservation': '', 'runtime': '12:00:00', 'srun': 'srun', 'username': 'wvincent', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser', 'profile': '24CPUNodes'}
[I 2024-03-20 16:06:39.628 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u wvincent sbatch --parsable
[D 2024-03-20 16:06:39.628 JupyterHub batchspawner:283] Spawner submitting script:
    #!/bin/bash
    #SBATCH --time=12:00:00
    #SBATCH --output=/home/wvincent/jupyterlab-%j.output
    #SBATCH --error=/home/wvincent/jupyterlab-%j.error
    #SBATCH --job-name=jupyterlab
    #SBATCH --export=PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,USER,HOME,SHELL
    #SBATCH --mail-user=william.vincent@domain.fr
    #SBATCH --ntasks=1
    #SBATCH --partition=24CPUNodes
    #SBATCH --chdir=/home/wvincent
    #SBATCH --get-user-env=L

    set -x
    #set -euo pipefail

    trap 'echo SIGTERM received' TERM



    #echo "# With Singularity
    #export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
    #export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
    #export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/pytorch2-will.sif
    #singularity exec --env JUPYTER_PORT=0 --bind $PWD:/run/user $SINGULARITYENV_CONTAINER_PATH batchspawner-singleuser jupyterhub-singleuser > jupyter.debugsing 2>&1


    echo "# Without Singularity "
    module load Python/3.12.2
    batchspawner-singleuser jupyterhub-singleuser > jupyter.debug 2>&1


    echo "jupyterhub-singleuser ended gracefully"


    echo "# fin"

[D 2024-03-20 16:06:39.629 JupyterHub batchspawner:284] Spawner submitting environment: {'PATH': '/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin', 'LANG': 'en_US.UTF-8', 'JUPYTERHUB_API_TOKEN': 'd11931814c9e47fc8bc824d0698980cb', 'JPY_API_TOKEN': 'd11931814c9e47fc8bc824d0698980cb', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-wvincent', 'JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED': '0', 'JUPYTERHUB_HOST': '', 'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/wvincent/oauth_callback', 'JUPYTERHUB_OAUTH_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]', 'JUPYTERHUB_USER': 'wvincent', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL': 'http://r9jupyter.domain.fr:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://r9jupyter.domain.fr:8081/hub/api/users/wvincent/activity', 'JUPYTERHUB_BASE_URL': '/', 'JUPYTERHUB_SERVICE_PREFIX': '/user/wvincent/', 'JUPYTERHUB_SERVICE_URL': 'http://0.0.0.0:0/user/wvincent/', 'JUPYTERHUB_ROOT_DIR': '~', 'JUPYTERHUB_DEFAULT_URL': '/lab', 'USER': 'wvincent', 'HOME': '/home/wvincent', 'SHELL': '/bin/bash'}
[I 2024-03-20 16:06:39.744 JupyterHub batchspawner:287] Job submitted. output: 163
[D 2024-03-20 16:06:39.745 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:06:39.805 JupyterHub batchspawner:428] Job 163 still pending
[W 2024-03-20 16:06:40.249 JupyterHub base:192] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I 2024-03-20 16:06:40.259 JupyterHub log:192] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@xxx.xxx.4.106) 1012.19ms
[D 2024-03-20 16:06:40.306 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:06:40.399 JupyterHub scopes:884] Checking access to /hub/spawn-pending/wvincent via scope servers
[D 2024-03-20 16:06:40.400 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/wvincent via servers
[I 2024-03-20 16:06:40.402 JupyterHub pages:399] wvincent is pending spawn
[I 2024-03-20 16:06:40.406 JupyterHub log:192] 200 GET /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@xxx.xxx.4.106) 16.70ms
[D 2024-03-20 16:06:40.578 JupyterHub scopes:884] Checking access to /hub/api/users/wvincent/server/progress via scope read:servers
[D 2024-03-20 16:06:40.578 JupyterHub scopes:697] Argument-based access to /hub/api/users/wvincent/server/progress via read:servers
[D 2024-03-20 16:06:40.877 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:06:41.456 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:06:41.505 JupyterHub base:357] Recording first activity for <APIToken('d119...', user='wvincent', client_id='jupyterhub')>
[I 2024-03-20 16:06:41.574 JupyterHub log:192] 200 POST /hub/api/batchspawner (wvincent@xxx.xxx.102.71) 77.97ms
[I 2024-03-20 16:06:41.576 JupyterHub batchspawner:462] Notebook server job 163 started at r9nc-24-1:50313
[D 2024-03-20 16:06:41.687 JupyterHub spawner:1388] Polling subprocess every 30s
[D 2024-03-20 16:06:41.732 JupyterHub utils:286] Waiting 300s for server at http://r9jupyter:0/user/wvincent/
[I 2024-03-20 16:06:42.953 JupyterHub log:192] 200 GET /hub/api (@xxx.xxx.102.71) 1.57ms
[D 2024-03-20 16:06:43.014 JupyterHub scopes:884] Checking access to /hub/api/users/wvincent/activity via scope users:activity
[D 2024-03-20 16:06:43.015 JupyterHub scopes:697] Argument-based access to /hub/api/users/wvincent/activity via users:activity
[D 2024-03-20 16:06:43.019 JupyterHub users:882] Activity for user wvincent: 2024-03-20T15:06:42.916014Z
[D 2024-03-20 16:06:43.019 JupyterHub users:900] Activity on server wvincent/: 2024-03-20T15:06:42.916014Z
[I 2024-03-20 16:06:43.095 JupyterHub log:192] 200 POST /hub/api/users/wvincent/activity (wvincent@xxx.xxx.102.71) 84.58ms
[D 2024-03-20 16:06:49.249 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[W 2024-03-20 16:06:49.346 JupyterHub base:1210] User wvincent is slow to become responsive (timeout=10)
[D 2024-03-20 16:06:49.346 JupyterHub base:1215] Expecting server for wvincent at: http://r9jupyter:0/user/wvincent/
[D 2024-03-20 16:07:11.690 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:07:41.688 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:08:11.690 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:08:41.690 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:09:11.691 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:09:41.688 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:10:11.689 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'
[D 2024-03-20 16:10:41.688 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 163 -o '%T %B'

jupyter.debug

[I 2024-03-20 16:06:42.434 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-03-20 16:06:42.438 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-03-20 16:06:42.438 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.0
[I 2024-03-20 16:06:42.438 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
[I 2024-03-20 16:06:42.441 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-03-20 16:06:42.443 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-03-20 16:06:42.446 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-03-20 16:06:42.909 ServerApp] notebook_shim | extension was successfully linked.
/apps/Python/Python-3.12.2/lib/python3.12/site-packages/jupyter_server/serverapp.py:2235: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/wvincent/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/wvincent/oauth_callback
  self.web_app = ServerWebApplication(
[I 2024-03-20 16:06:42.941 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-03-20 16:06:42.943 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-03-20 16:06:42.944 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-03-20 16:06:42.954 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-03-20 16:06:42.954 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-03-20 16:06:42.958 LabApp] JupyterLab extension loaded from /apps/Python/Python-3.12.2/lib/python3.12/site-packages/jupyterlab
[I 2024-03-20 16:06:42.958 LabApp] JupyterLab application directory is /apps/Python/Python-3.12.2/share/jupyter/lab
[I 2024-03-20 16:06:42.959 LabApp] Extension Manager is 'pypi'.
[I 2024-03-20 16:06:42.998 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-03-20 16:06:42.998 ServerApp] Serving notebooks from local directory: /home/wvincent
[I 2024-03-20 16:06:42.998 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-03-20 16:06:42.998 ServerApp] http://r9nc-24-1:50313/user/wvincent/lab?token=...
[I 2024-03-20 16:06:42.998 ServerApp]     http://127.0.0.1:50313/user/wvincent/lab?token=...
[I 2024-03-20 16:06:42.998 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-03-20 16:06:43.007 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

My ~/jupyterlab-xxx.output is always empty

As you can see, once everything is launched, I get “Expecting server for wvincent at: http://r9jupyter:0/user/wvincent/”. I think the problem comes from there, it does not give the correct URL, and nothing happens.

Second issue : with singularity

When I run it through Singularity, the ports are not in agreement between the Jupyter server and the Slurm node. For example, if I look at the server log, I get

[I 2024-03-20 09:06:34.651 JupyterHub batchspawner:462] Notebook server job 144 started at r9nc-24-1:35303
[D 2024-03-20 09:06:34.658 JupyterHub spawner:1384] Polling subprocess every 30s
[I 2024-03-20 09:06:37.850 JupyterHub log:191] 200 GET /hub/api (@xxx.xxx.102.71) 1.88ms
[D 2024-03-20 09:06:37.863 JupyterHub scopes:877] Checking access to /hub/api/users/wvincent/activity via scope users:activity
[D 2024-03-20 09:06:37.863 JupyterHub scopes:690] Argument-based access to /hub/api/users/wvincent/activity via users:activity
[D 2024-03-20 09:06:37.866 JupyterHub users:874] Activity for user wvincent: 2024-03-20T08:06:35.457853Z
[D 2024-03-20 09:06:37.866 JupyterHub users:892] Activity on server wvincent/: 2024-03-20T08:06:35.457853Z
[I 2024-03-20 09:06:37.908 JupyterHub log:191] 200 POST /hub/api/users/wvincent/activity (wvincent@xxx.xxx.102.71) 48.00ms
[D 2024-03-20 09:06:41.716 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[W 2024-03-20 09:06:41.812 JupyterHub base:1122] User wvincent is slow to become responsive (timeout=10)
[D 2024-03-20 09:06:41.813 JupyterHub base:1127] Expecting server for wvincent at: http://r9jupyter:0/user/wvincent/
[D 2024-03-20 09:07:04.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:07:34.662 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:08:04.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:08:34.662 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:09:04.660 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:09:34.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:10:04.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:10:34.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:11:04.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[D 2024-03-20 09:11:13.529 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
09:11:13.533 [ConfigProxy] info: 200 GET /api/routes
[D 2024-03-20 09:11:13.535 JupyterHub proxy:392] Checking routes
[D 2024-03-20 09:11:31.489 JupyterHub base:344] Refreshing auth for wvincent
[D 2024-03-20 09:11:31.490 JupyterHub scopes:877] Checking access to /hub/api/users/wvincent/activity via scope users:activity
[D 2024-03-20 09:11:31.491 JupyterHub scopes:690] Argument-based access to /hub/api/users/wvincent/activity via users:activity
[D 2024-03-20 09:11:31.495 JupyterHub users:879] Not updating activity for <User(wvincent 1/1 running)>: 2024-03-20T08:06:35.457853Z < 2024-03-20T08:06:35.457853Z
[D 2024-03-20 09:11:31.495 JupyterHub users:900] Not updating server activity on wvincent/: 2024-03-20T08:06:35.457853Z < 2024-03-20T08:06:35.457853Z
[I 2024-03-20 09:11:31.496 JupyterHub log:191] 200 POST /hub/api/users/wvincent/activity (wvincent@xxx.xxx.102.71) 21.91ms
[D 2024-03-20 09:11:34.661 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 144 -o '%T %B'
[W 2024-03-20 09:11:37.259 JupyterHub user:933] wvincent's server never showed up at http://r9jupyter:0/user/wvincent/ after 300 seconds. Giving up.

In job log node :

[I 2024-03-20 09:06:37.850 SingleUserNotebookApp notebookapp:2329] Serving notebooks from local directory: /home/wvincent
[I 2024-03-20 09:06:37.850 SingleUserNotebookApp notebookapp:2329] Jupyter Notebook 6.4.10 is running at:
[I 2024-03-20 09:06:37.850 SingleUserNotebookApp notebookapp:2329] http://hostname:0/

And if I look at the open ports node , I have this:

ss -ltpn
LISTEN 0 128 0.0.0.0:36447 0.0.0.0:* users:(("batchspawner-si",pid=9940,fd=10))

Ports are never same.
And i have same error with ‘http://r9jupyter:0/user/wvincent/’ url

Install

On node slurm and jupyterhub server, I install latest version batchspawner & jupyterhub with

/usr/local/bin/pip3 install jupyterhub
/usr/local/bin/pip3 install https://github.com/jupyterhub/batchspawner/archive/main.zip

Modules list

pip list -v
batchspawner              1.3.1.dev0      /usr/local/lib/python3.9/site-packages   pip
jupyter_client            8.6.0                                     /usr/local/lib/python3.9/site-packages   pip
jupyter_core              5.7.1                                     /usr/local/lib/python3.9/site-packages   pip
jupyter-events            0.9.0                                     /usr/local/lib/python3.9/site-packages   pip
jupyter-lsp               2.2.3                                     /usr/local/lib/python3.9/site-packages   pip
jupyter_server            2.13.0                                    /usr/local/lib/python3.9/site-packages   pip
jupyter_server_terminals  0.5.2                                     /usr/local/lib/python3.9/site-packages   pip
jupyter-telemetry         0.1.0                                     /usr/local/lib/python3.9/site-packages   pip
jupyterhub                4.1.0                                     /usr/local/lib/python3.9/site-packages   pip
jupyterlab                4.1.3                                     /usr/local/lib/python3.9/site-packages   pip
jupyterlab_pygments       0.3.0                                     /usr/local/lib/python3.9/site-packages   pip
jupyterlab_server         2.25.3                                    /usr/local/lib/python3.9/site-packages   pip
jupyterlab_widgets        3.0.10                                    /usr/local/lib/python3.9/site-packages   pip
wrapspawner               1.0.2.dev0      /usr/local/lib/python3.9/site-packages   pip

Thank you in advance for your help.

2

You need to remove this config

c.ProfilesSpawner.ip = '$HOSTNAME'
c.Spawner.ip = '$HOSTNAME'
c.Spawner.port = 0

It is spawner’s who will set the hostname and port of single user server during spawning. The ip will be the SLURM compute node where job will be scheduled and port will be a free port on that node batchspawner will find before launching single user server. This is the reason you are seeing this line.

Could you try removing those config options?

2 Likes
3

Thank you, but it doesn’t change anything with and without singularity :confused:
We had set it up that way because it was the old configuration that worked.

4

Could you please share your updated config, JupyterHub logs and also logs of the SLURM job? Cheers

5

I have updated my first post with the new config/logs.
Thanks

6

Thanks for the logs. I guess the issue is coming from ProfileSpawner. Can you try using SlurmSpawner directly just to reduce the variables?

Here is the config that you use

import os
import importlib.machinery
import batchspawner

c = get_config()

c.JupyterHub.ip = 'r9jupyter.domain.fr'
c.JupyterHub.hub_ip = 'r9jupyter.domain.fr'
c.JupyterHub.reset_db = True

c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
c.BatchSpawnerBase.req_partition = '24CPUNodes'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.BatchSpawnerBase.batch_script = '''#!/bin/bash
#SBATCH --time={runtime}
#SBATCH --output={homedir}/jupyterlab-%j.output
#SBATCH --error={homedir}/jupyterlab-%j.error
#SBATCH --job-name=jupyterlab
#SBATCH --export={keepvars}
#SBATCH --mail-user=william.vincent@domain.fr
#SBATCH --ntasks=1
#SBATCH --partition={partition}
#SBATCH --chdir={homedir}
#SBATCH --get-user-env=L

set -x
#set -euo pipefail

trap 'echo SIGTERM received' TERM

{prologue}

#echo "# With Singularity "
export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/pytorch2-will.sif
#srun singularity exec --env JUPYTER_PORT=0 --bind $PWD:/run/user $SINGULARITYENV_CONTAINER_PATH {cmd} > jupyter.debug 2>&1

#echo "# Without Singularity "
module load Python/3.12.2
{cmd} > jupyter.debug 2>&1

echo "jupyterhub-singleuser ended gracefully"
{epilogue}
echo "# end"
'''

c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~'
c.Spawner.start_timeout = 300
c.Spawner.http_timeout = 300
c.Spawner.debug = True
7

Is this with a recent version of batchspawner? 1.3.0 should support jupyterhub 4, but 1.2.0 didnt work with jupyterhub 3+

1 Like
8

Hello,

@consideRatio I have batchspawner==1.3.1.dev0 installed on both the server and nodes.

Without using Singularity, the JupyterHub is working fine after removing ProfilesSpawner as suggested by @mahendrapaipuri . Here is the log:

[I 2024-03-21 09:12:59.284 JupyterHub batchspawner:462] Notebook server job 169 started at r9nc-24-1:37557
[D 2024-03-21 09:12:59.407 JupyterHub spawner:1388] Polling subprocess every 30s
[D 2024-03-21 09:12:59.486 JupyterHub utils:286] Waiting 300s for server at http://r9nc-24-1:37557/user/wvincent/

However, when using Singularity, it still doesn’t work, and there is still “http://hostname:0/” in the log on the node.

I was wondering if there is a way to give users the choice of the Singularity image and therefore the Nvidia PyTorch, TensorFlow, Kaldi image?

I just tried with our old configuration using optionsspawner.OptionsFormSpawner, but there is still the same issue with the URL:

[I 2024-03-21 09:10:09.244 JupyterHub batchspawner:462] Notebook server job 168 started at r9nc-24-1:54977
[D 2024-03-21 09:10:09.252 JupyterHub spawner:1388] Polling subprocess every 30s
[D 2024-03-21 09:10:09.260 JupyterHub utils:286] Waiting 300s for server at http://r9jupyter:0/user/wvincent/
9

Ok, that is what I suspected. It is an issue from ProfileSpawner. Could you try with the following patched ProfileSpawner:

import os
import importlib.machinery
import batchspawner

c = get_config()

import os
import json
import re
import urllib.request

from tornado import gen, concurrent

from jupyterhub.spawner import LocalProcessSpawner, Spawner
from traitlets import (
    Instance, Type, Tuple, List, Dict, Integer, Unicode, Float, Any
)
from traitlets import directional_link

# Only needed for DockerProfilesSpawner
try:
    import docker
except ImportError:
    pass

# Utility to create dummy Futures to return values through yields
def _yield_val(x=None):
    f = concurrent.Future()
    f.set_result(x)
    return f

class WrapSpawner(Spawner):

    # Grab this from constructor args in case some Spawner ever wants it
    config = Any()

    child_class = Type(LocalProcessSpawner, Spawner,
        config=True,
        help="""The class to wrap for spawning single-user servers.
                Should be a subclass of Spawner.
                """
        )

    child_config = Dict(default_value={},
        config=True,
        help="Dictionary of config values to apply to wrapped spawner class."
        )

    child_state = Dict(default_value={})

    child_spawner = Instance(Spawner, allow_none=True)

    def construct_child(self):
        if self.child_spawner is None:
            self.child_spawner = self.child_class(
                user = self.user,
                db   = self.db,
                hub  = self.hub,
                authenticator = self.authenticator,
                oauth_client_id = self.oauth_client_id,
                server = self._server,
                config = self.config,
                **self.child_config
                )
            # initial state will always be wrong since it will see *our* state
            self.child_spawner.clear_state()
            if self.child_state:
                self.child_spawner.load_state(self.child_state)

            # link traits common between self and child
            common_traits = (
                set(self.trait_names()) &
                set(self.child_spawner.trait_names()) -
                set(self.child_config.keys())
            )
            for trait in common_traits:
                directional_link((self, trait), (self.child_spawner, trait))
        return self.child_spawner

    def load_child_class(self, state):
        # Subclasses must arrange for correct child_class setting from load_state
        pass

    def load_state(self, state):
        super().load_state(state)
        self.load_child_class(state)
        self.child_config.update(state.get('child_conf', {}))
        self.child_state = state.get('child_state', {})
        self.construct_child()

    def get_state(self):
        state = super().get_state()
        state['child_conf'] = self.child_config
        if self.child_spawner:
            self.child_state = state['child_state'] = self.child_spawner.get_state()
        return state

    def clear_state(self):
        super().clear_state()
        if self.child_spawner:
            self.child_spawner.clear_state()
        self.child_state = {}
        self.child_config = {}
        self.child_spawner = None

    # proxy functions for start/poll/stop
    # pass back the child's Future, or create a dummy if needed

    def start(self):
        if not self.child_spawner:
            self.construct_child()
        return self.child_spawner.start()

    def stop(self, now=False):
        if self.child_spawner:
            return self.child_spawner.stop(now)
        else:
            return _yield_val()

    def poll(self):
        if self.child_spawner:
            return self.child_spawner.poll()
        else:
            return _yield_val(1)

    if hasattr(Spawner, 'progress'):
        @property
        def progress(self):
            if self.child_spawner:
                return self.child_spawner.progress
            else:
                raise RuntimeError("No child spawner yet exists - can not get progress yet")

class ProfilesSpawner(WrapSpawner):

    """ProfilesSpawner - leverages the Spawner options form feature to allow user-driven
        configuration of Spawner classes while permitting:
        1) configuration of Spawner classes that don't natively implement options_form
        2) administrator control of allowed configuration changes
        3) runtime choice of which Spawner backend to launch
    """

    profiles = List(
        trait = Tuple( Unicode(), Unicode(), Type(Spawner), Dict() ),
        default_value = [ ( 'Local Notebook Server', 'local', LocalProcessSpawner,
                            {'start_timeout': 15, 'http_timeout': 10} ) ],
        minlen = 1,
        config = True,
        help = """List of profiles to offer for selection. Signature is:
            List(Tuple( Unicode, Unicode, Type(Spawner), Dict )) corresponding to
            profile display name, unique key, Spawner class, dictionary of spawner config options.

            The first three values will be exposed in the input_template as {display}, {key}, and {type}"""
        )

    child_profile = Unicode()

    form_template = Unicode(
        """<label for="profile">Select a job profile:</label>
        <select class="form-control" name="profile" required autofocus>
        {input_template}
        </select>
        """,
        config = True,
        help = """Template to use to construct options_form text. {input_template} is replaced with
            the result of formatting input_template against each item in the profiles list."""
        )

    first_template = Unicode('selected',
        config=True,
        help="Text to substitute as {first} in input_template"
        )

    input_template = Unicode("""
        <option value="{key}" {first}>{display}</option>""",
        config = True,
        help = """Template to construct {input_template} in form_template. This text will be formatted
            against each item in the profiles list, in order, using the following key names:
            ( display, key, type ) for the first three items in the tuple, and additionally
            first = "checked" (taken from first_template) for the first item in the list, so that
            the first item starts selected."""
        )

    def _options_form_default(self):
        temp_keys = [ dict(display=p[0], key=p[1], type=p[2], first='') for p in self.profiles ]
        temp_keys[0]['first'] = self.first_template
        text = ''.join([ self.input_template.format(**tk) for tk in temp_keys ])
        return self.form_template.format(input_template=text)

    def options_from_form(self, formdata):
        # Default to first profile if somehow none is provided
        return dict(profile=formdata.get('profile', [self.profiles[0][1]])[0])

    # load/get/clear : save/restore child_profile (and on load, use it to update child class/config)

    def select_profile(self, profile):
        # Select matching profile, or do nothing (leaving previous or default config in place)
        for p in self.profiles:
            if p[1] == profile:
                self.child_class = p[2]
                self.child_config = p[3]
                break

    def construct_child(self):
        self.child_profile = self.user_options.get('profile', "")
        self.select_profile(self.child_profile)
        super().construct_child()

    def load_child_class(self, state):
        try:
            self.child_profile = state['profile']
        except KeyError:
            self.child_profile = ''
        self.select_profile(self.child_profile)

    def get_state(self):
        state = super().get_state()
        state['profile'] = self.child_profile
        return state

    def clear_state(self):
        super().clear_state()
        self.child_profile = ''

c.JupyterHub.ip = 'r9jupyter.domain.fr'
c.JupyterHub.hub_ip = 'r9jupyter.domain.fr'
c.JupyterHub.reset_db = True

c.JupyterHub.spawner_class = 'ProfilesSpawner'
c.ProfilesSpawner.profiles = [
   ('24CPUNodes', '24CPUNodes', 'batchspawner.SlurmSpawner',dict(req_partition='24CPUNodes',batch_script=bscript)),
   ('GPUNodes1080-dev', 'GPUNodes1080-dev', 'batchspawner.SlurmSpawner',dict(req_partition='GPUNodes1080-dev',batch_script=bscript)),
]

c.BatchSpawnerBase.req_partition = '24CPUNodes'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.BatchSpawnerBase.batch_script = '''#!/bin/bash
#SBATCH --time={runtime}
#SBATCH --output={homedir}/jupyterlab-%j.output
#SBATCH --error={homedir}/jupyterlab-%j.error
#SBATCH --job-name=jupyterlab
#SBATCH --export={keepvars}
#SBATCH --mail-user=william.vincent@domain.fr
#SBATCH --ntasks=1
#SBATCH --partition={partition}
#SBATCH --chdir={homedir}
#SBATCH --get-user-env=L

set -x
#set -euo pipefail

trap 'echo SIGTERM received' TERM

{prologue}

#echo "# With Singularity "
export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/pytorch2-will.sif
#srun singularity exec --env JUPYTER_PORT=0 --bind $PWD:/run/user $SINGULARITYENV_CONTAINER_PATH {cmd} > jupyter.debug 2>&1

#echo "# Without Singularity "
module load Python/3.12.2
{cmd} > jupyter.debug 2>&1

echo "jupyterhub-singleuser ended gracefully"
{epilogue}
echo "# end"
'''

c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~'
c.Spawner.start_timeout = 300
c.Spawner.http_timeout = 300
c.Spawner.debug = True

Could you try with and without singularity and post the logs here? Please do not edit existing posts with updated logs. Create new post with updated config and logs so that we can compare the changes.

@consideRatio I think the issue is the server property on WrapSpawner. This change has been made following this issue. I think what is happening is when the WrapSpawner class gets instantiated, self.server is from LocalSpawner as child_class is of type LocalSpawner. And my guess is that it never gets updated to that of child spawner’s server object. That is the reason why we see the single server’s URL http://r9jupyter:0/user/wvincent/ where r9jupyter is the host where JupyterHub is deployed and default value of port is always 0.

We use our own WrapSpawner that works with BatchSpawner and SSHSpawner and I remember I had to remove the server property on WrapSpawner to make it work. I will try to look into the internals to see what is exactly happening.

@minrk Any thoughts? Maybe the way server property is handled on spawner has changed after that PR on WrapSpawner?

1 Like
10

I just merged the removal of the server property in wrapspawner, so it may work if you update your wrapspawner.

That was always a very tricky bit, and I don’t understand why it didn’t work, but I’ll try to dig in to see if I can figure it out better.

1 Like
11

Ok, thank you very much, we are making progress!
With the update of wrapspawner, I confirm that profilespawner and optionsspawner.OptionsFormSpawner works !
Now we need to understand why with singularity, it does not use a random port and why there is not the same port between the server and the node.

I deleted the JUPYTER_PORT environment variable that was set to 8888 in the container.

Node log :

[I 2024-03-21 13:14:35.961 SingleUserNotebookApp mixins:541] Starting jupyterhub single-user server version 4.1.0
[I 2024-03-21 13:14:35.961 SingleUserNotebookApp mixins:555] Extending notebook.notebookapp.NotebookApp from notebook 6.4.10
[W 2024-03-21 13:14:35.976 SingleUserNotebookApp configurable:200] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2024-03-21 13:14:36.205 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[I 2024-03-21 13:14:38.575 SingleUserNotebookApp handlers:52] jupyter_tensorboard extension loaded.
[I 2024-03-21 13:14:38.582 SingleUserNotebookApp extension:162] JupyterLab extension loaded from /usr/local/lib/python3.10/dist-packages/jupyterlab
[I 2024-03-21 13:14:38.583 SingleUserNotebookApp extension:163] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-03-21 13:14:38.586 SingleUserNotebookApp __init__:14] [Jupytext Server Extension] NotebookApp.contents_manager_class is (a subclass of) jupytext.TextFileContentsManager already - OK
[I 2024-03-21 13:14:38.590 SingleUserNotebookApp mixins:629] Starting jupyterhub-singleuser server version 4.1.0
[I 2024-03-21 13:14:38.598 SingleUserNotebookApp notebookapp:2329] Serving notebooks from local directory: /home/wvincent
[I 2024-03-21 13:14:38.598 SingleUserNotebookApp notebookapp:2329] Jupyter Notebook 6.4.10 is running at:
[I 2024-03-21 13:14:38.598 SingleUserNotebookApp notebookapp:2329] http://hostname:8888/
[I 2024-03-21 13:14:38.598 SingleUserNotebookApp notebookapp:2330] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-03-21 13:14:38.603 SingleUserNotebookApp mixins:523] Updating Hub with activity every 300 seconds

Server log :

[D 2024-03-21 13:13:51.522 JupyterHub application:908] Looking for /etc/jupyterhub/jupyterhub_config in /etc/jupyterhub
[D 2024-03-21 13:13:51.531 JupyterHub application:929] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2024-03-21 13:13:51.541 JupyterHub app:2885] Running JupyterHub version 4.1.0
[I 2024-03-21 13:13:51.541 JupyterHub app:2915] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.1.0
[I 2024-03-21 13:13:51.541 JupyterHub app:2915] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2024-03-21 13:13:51.541 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.0
[D 2024-03-21 13:13:51.545 JupyterHub app:2842] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2024-03-21 13:13:51.545 JupyterHub app:1683] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[D 2024-03-21 13:13:51.545 JupyterHub app:1852] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2024-03-21 13:13:52.218 JupyterHub orm:997] Stamping empty database with alembic revision 0eee8c825d24
[I 2024-03-21 13:13:52.220 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
[I 2024-03-21 13:13:52.220 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
[I 2024-03-21 13:13:52.316 JupyterHub proxy:557] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2024-03-21 13:13:52.316 JupyterHub app:2125] Loading roles into database
[I 2024-03-21 13:13:52.418 JupyterHub app:2005] Not using allowed_users. Any authenticated user will be allowed.
[D 2024-03-21 13:13:52.426 JupyterHub app:2364] Purging expired APITokens
[D 2024-03-21 13:13:52.431 JupyterHub app:2364] Purging expired OAuthCodes
[D 2024-03-21 13:13:52.432 JupyterHub app:2200] Loading role assignments from config
[D 2024-03-21 13:13:52.441 JupyterHub app:2523] Initializing spawners
[D 2024-03-21 13:13:52.442 JupyterHub app:2658] Loaded users:

[I 2024-03-21 13:13:52.442 JupyterHub app:2954] Initialized 0 spawners in 0.002 seconds
[I 2024-03-21 13:13:52.446 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-03-21 13:13:52.447 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2024-03-21 13:13:52.447 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2024-03-21 13:13:52.448 JupyterHub proxy:747] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2024-03-21 13:13:52.448 JupyterHub proxy:751] Starting proxy @ http://r9jupyter.domain.fr:8000/
[D 2024-03-21 13:13:52.448 JupyterHub proxy:752] Proxy cmd: ['configurable-http-proxy', '--ip', 'r9jupyter.domain.fr', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://r9jupyter.domain.fr:8081/hub/error', '--log-level', 'info']
[D 2024-03-21 13:13:52.453 JupyterHub proxy:671] Writing proxy pid file: jupyterhub-proxy.pid
[D 2024-03-21 13:13:52.454 JupyterHub utils:264] Waiting 10s for server at r9jupyter.domain.fr:8000
[D 2024-03-21 13:13:52.455 JupyterHub utils:111] Server at r9jupyter.domain.fr:8000 not ready: [Errno 111] Connection refused
[D 2024-03-21 13:13:52.456 JupyterHub utils:264] Waiting 10s for server at 127.0.0.1:8001
[D 2024-03-21 13:13:52.456 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-03-21 13:13:52.515 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-03-21 13:13:52.640 JupyterHub utils:111] Server at r9jupyter.domain.fr:8000 not ready: [Errno 111] Connection refused
13:13:52.692 [ConfigProxy] info: Proxying http://r9jupyter.domain.fr:8000 to (no default)
13:13:52.694 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-03-21 13:13:52.762 JupyterHub utils:274] Server at r9jupyter.domain.fr:8000 responded in 0.31s
[D 2024-03-21 13:13:52.791 JupyterHub utils:274] Server at 127.0.0.1:8001 responded in 0.34s
[D 2024-03-21 13:13:52.791 JupyterHub proxy:831] Proxy started and appears to be up
[D 2024-03-21 13:13:52.797 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
13:13:52.815 [ConfigProxy] info: 200 GET /api/routes
[I 2024-03-21 13:13:52.816 JupyterHub app:3204] Hub API listening on http://r9jupyter.domain.fr:8081/hub/
[D 2024-03-21 13:13:52.817 JupyterHub proxy:390] Fetching routes to check
[D 2024-03-21 13:13:52.817 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
13:13:52.820 [ConfigProxy] info: 200 GET /api/routes
[D 2024-03-21 13:13:52.820 JupyterHub proxy:393] Checking routes
[I 2024-03-21 13:13:52.821 JupyterHub proxy:478] Adding route for Hub: / => http://r9jupyter.domain.fr:8081
[D 2024-03-21 13:13:52.821 JupyterHub proxy:924] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
13:13:52.824 [ConfigProxy] info: Adding route / -> http://r9jupyter.domain.fr:8081
13:13:52.825 [ConfigProxy] info: Route added / -> http://r9jupyter.domain.fr:8081
13:13:52.825 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-03-21 13:13:52.826 JupyterHub app:3271] JupyterHub is now running at http://r9jupyter.domain.fr:8000/
[D 2024-03-21 13:13:52.826 JupyterHub app:2878] It took 1.308 seconds for the Hub to start
[W 2024-03-21 13:13:54.074 JupyterHub web:1873] 403 POST /hub/api/users/wvincent/activity (xxx.xxx.102.71): Missing or invalid credentials.
[W 2024-03-21 13:13:54.075 JupyterHub log:192] 403 POST /hub/api/users/wvincent/activity (@xxx.xxx.102.71) 9.15ms
[I 2024-03-21 13:14:01.851 JupyterHub log:192] 302 GET /user/wvincent/terminals/websocket/1 -> /hub/user/wvincent/terminals/websocket/1 (@xxx.xxx.4.106) 1.59ms
[I 2024-03-21 13:14:04.028 JupyterHub log:192] 302 GET /user/wvincent/api/kernels/a7da2131-a978-4699-b85e-d54a3a62bca4?1711023234312 -> /hub/user/wvincent/api/kernels/a7da2131-a978-4699-b85e-d54a3a62bca4?1711023234312 (@xxx.xxx.4.106) 1.49ms
[W 2024-03-21 13:14:04.043 JupyterHub base:480] Invalid cookie token
[I 2024-03-21 13:14:04.045 JupyterHub log:192] 302 GET /hub/user/wvincent/api/kernels/a7da2131-a978-4699-b85e-d54a3a62bca4?1711023234312 -> /hub/login?next=%2Fhub%2Fuser%2Fwvincent%2Fapi%2Fkernels%2Fa7da2131-a978-4699-b85e-d54a3a62bca4%3F1711023234312 (@xxx.xxx.4.106) 8.50ms
[W 2024-03-21 13:14:04.056 JupyterHub base:480] Invalid cookie token
[D 2024-03-21 13:14:04.057 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:3e49d812a0154eeab5de5cf11eb4edce' != b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY='
[I 2024-03-21 13:14:04.088 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fuser%2Fwvincent%2Fapi%2Fkernels%2Fa7da2131-a978-4699-b85e-d54a3a62bca4%3F1711023234312 (@xxx.xxx.4.106) 34.97ms
[I 2024-03-21 13:14:11.042 JupyterHub log:192] 302 GET /user/wvincent/api/kernels/786691a0-5799-426a-ad18-97d30f0e74f3?1711023241342 -> /hub/user/wvincent/api/kernels/786691a0-5799-426a-ad18-97d30f0e74f3?1711023241342 (@xxx.xxx.4.106) 1.90ms
[W 2024-03-21 13:14:11.056 JupyterHub base:480] Invalid cookie token
[I 2024-03-21 13:14:11.057 JupyterHub log:192] 302 GET /hub/user/wvincent/api/kernels/786691a0-5799-426a-ad18-97d30f0e74f3?1711023241342 -> /hub/login?next=%2Fhub%2Fuser%2Fwvincent%2Fapi%2Fkernels%2F786691a0-5799-426a-ad18-97d30f0e74f3%3F1711023241342 (@xxx.xxx.4.106) 4.28ms
[W 2024-03-21 13:14:11.070 JupyterHub base:480] Invalid cookie token
[D 2024-03-21 13:14:11.071 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:3e49d812a0154eeab5de5cf11eb4edce' != b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY='
[I 2024-03-21 13:14:11.074 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fuser%2Fwvincent%2Fapi%2Fkernels%2F786691a0-5799-426a-ad18-97d30f0e74f3%3F1711023241342 (@xxx.xxx.4.106) 6.09ms
[I 2024-03-21 13:14:16.571 JupyterHub log:192] 302 GET /user/wvincent/api/events/subscribe?token=[secret] -> /hub/user/wvincent/api/events/subscribe?token=[secret] (@xxx.xxx.4.106) 0.72ms
[I 2024-03-21 13:14:24.852 JupyterHub log:192] 302 GET / -> /hub/ (@xxx.xxx.4.106) 1.29ms
[W 2024-03-21 13:14:24.865 JupyterHub base:480] Invalid cookie token
[I 2024-03-21 13:14:24.866 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@xxx.xxx.4.106) 2.51ms
[W 2024-03-21 13:14:24.876 JupyterHub base:480] Invalid cookie token
[D 2024-03-21 13:14:24.877 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:3e49d812a0154eeab5de5cf11eb4edce' != b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY='
[I 2024-03-21 13:14:24.877 JupyterHub _xsrf_utils:159] Setting new xsrf cookie for b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY=' {'path': '/hub/', 'max_age': 3600}
[I 2024-03-21 13:14:24.878 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@xxx.xxx.4.106) 3.50ms
[D 2024-03-21 13:14:25.047 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=01598a5386176f0279952a3b9632a07e7fce9a12aa53108973c83be9ec3473e7a59354876fab64bfeb01892eb503870183707aa03f207d7a94845ca7980c3819 (@xxx.xxx.4.106) 10.27ms
[D 2024-03-21 13:14:25.052 JupyterHub log:192] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@xxx.xxx.4.106) 1.04ms
[D 2024-03-21 13:14:25.054 JupyterHub log:192] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=bf6089ed4698cb8270a8b0c8ad9508ff886a7a842278e98064d5c1790ca3a36d5d69d9f047ef196882554fc104da2c88eb5395f1ee8cf0f3f6ff8869408350fe (@xxx.xxx.4.106) 1.24ms
[D 2024-03-21 13:14:25.055 JupyterHub log:192] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@xxx.xxx.4.106) 1.01ms
[D 2024-03-21 13:14:25.056 JupyterHub log:192] 200 GET /hub/logo (@xxx.xxx.4.106) 1.37ms
[D 2024-03-21 13:14:25.408 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 1.49ms
[D 2024-03-21 13:14:25.416 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 1.43ms
[W 2024-03-21 13:14:28.338 JupyterHub base:480] Invalid cookie token
[D 2024-03-21 13:14:28.342 JupyterHub log:192] 200 GET /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 (@xxx.xxx.4.106) 1.27ms
[D 2024-03-21 13:14:28.455 JupyterHub roles:282] Assigning default role to User wvincent
[I 2024-03-21 13:14:28.462 JupyterHub roles:239] Adding role user for User: wvincent
[D 2024-03-21 13:14:28.688 JupyterHub roles:282] Assigning default role to User wvincent
[W 2024-03-21 13:14:28.695 JupyterHub base:480] Invalid cookie token
[D 2024-03-21 13:14:28.695 JupyterHub base:667] Setting cookie for wvincent: jupyterhub-hub-login
[D 2024-03-21 13:14:28.695 JupyterHub base:663] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-03-21 13:14:28.695 JupyterHub base:925] User logged in: wvincent
[I 2024-03-21 13:14:28.696 JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (wvincent@xxx.xxx.4.106) 361.20ms
[D 2024-03-21 13:14:28.710 JupyterHub base:357] Recording first activity for <User(wvincent 0/1 running)>
[D 2024-03-21 13:14:28.823 JupyterHub user:431] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for wvincent:
[W 2024-03-21 13:14:28.826 JupyterHub spawner:172]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.

    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

[I 2024-03-21 13:14:28.829 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (wvincent@xxx.xxx.4.106) 121.29ms
[D 2024-03-21 13:14:28.841 JupyterHub scopes:884] Checking access to /hub/spawn via scope servers
[D 2024-03-21 13:14:28.841 JupyterHub scopes:697] Argument-based access to /hub/spawn via servers
[D 2024-03-21 13:14:28.841 JupyterHub pages:210] Serving options form for wvincent
[D 2024-03-21 13:14:28.841 JupyterHub _xsrf_utils:137] xsrf id mismatch b'd7d8defe6da747e897e18966ea64dac6:ibeGjLpkpp9LNy9w5jDbRC1kfnNkIdAghLSQt2OZvPY=' != b'd7d8defe6da747e897e18966ea64dac6:3adf768a7c444433aa0ee8c2e30d772b'
[I 2024-03-21 13:14:28.841 JupyterHub _xsrf_utils:159] Setting new xsrf cookie for b'd7d8defe6da747e897e18966ea64dac6:3adf768a7c444433aa0ee8c2e30d772b' {'path': '/hub/'}
[I 2024-03-21 13:14:28.846 JupyterHub log:192] 200 GET /hub/spawn (wvincent@xxx.xxx.4.106) 6.74ms
[D 2024-03-21 13:14:28.998 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=01598a5386176f0279952a3b9632a07e7fce9a12aa53108973c83be9ec3473e7a59354876fab64bfeb01892eb503870183707aa03f207d7a94845ca7980c3819 (@xxx.xxx.4.106) 1.99ms
[D 2024-03-21 13:14:29.073 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 1.63ms
[D 2024-03-21 13:14:31.706 JupyterHub scopes:884] Checking access to /hub/spawn via scope servers
[D 2024-03-21 13:14:31.706 JupyterHub scopes:697] Argument-based access to /hub/spawn via servers
[D 2024-03-21 13:14:31.707 JupyterHub pages:258] Triggering spawn with supplied form options for wvincent
[D 2024-03-21 13:14:31.707 JupyterHub base:1049] Initiating spawn for wvincent
[D 2024-03-21 13:14:31.708 JupyterHub base:1053] 0/100 concurrent spawns
[D 2024-03-21 13:14:31.708 JupyterHub base:1058] 0 active servers
[I 2024-03-21 13:14:32.071 JupyterHub provider:660] Creating oauth client jupyterhub-user-wvincent
[D 2024-03-21 13:14:32.406 JupyterHub user:797] Calling Spawner.start for wvincent
[I 2024-03-21 13:14:32.413 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cluster': '', 'epilogue': '', 'gres': '', 'homedir': '/home/wvincent', 'host': 'r9jupyter.domain.fr', 'keepvars': 'PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,USER,HOME,SHELL', 'keepvars_extra': '', 'memory': '', 'ngpus': '', 'nprocs': '2', 'options': '', 'partition': '24CPUNodes', 'prologue': '', 'qos': '', 'queue': 'r9jupyter', 'reservation': '', 'runtime': '12:00:00', 'srun': 'srun', 'username': 'wvincent', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser', 'profile': '24CPUNodes'}
[I 2024-03-21 13:14:32.413 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u wvincent sbatch --parsable
[D 2024-03-21 13:14:32.413 JupyterHub batchspawner:283] Spawner submitting script:
    #!/bin/bash
    #SBATCH --time=12:00:00
    #SBATCH --output=/home/wvincent/jupyterlab-%j.output
    #SBATCH --error=/home/wvincent/jupyterlab-%j.error
    #SBATCH --job-name=jupyterlab
    #SBATCH --export=PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,USER,HOME,SHELL
    #SBATCH --mail-user=william.vincent@domain.fr
    #SBATCH --ntasks=1
    #SBATCH --partition=24CPUNodes
    #SBATCH --chdir=/home/wvincent
    #SBATCH --get-user-env=L

    set -x
    #set -euo pipefail

    trap 'echo SIGTERM received' TERM

    echo "# With Singularity"
    export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
    export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
    singularity exec --bind $PWD:/run/user /apps/containerCollections/CUDA12/pytorch2-will.sif batchspawner-singleuser jupyterhub-singleuser > jupyter.debugsing 2>&1


    echo "jupyterhub-singleuser ended gracefully"
    echo "# fin"

[D 2024-03-21 13:14:32.414 JupyterHub batchspawner:284] Spawner submitting environment: {'PATH': '/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin', 'LANG': 'en_US.UTF-8', 'JUPYTERHUB_API_TOKEN': '1d7813c5a8df49fa8805a14690c8d9fc', 'JPY_API_TOKEN': '1d7813c5a8df49fa8805a14690c8d9fc', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-wvincent', 'JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED': '0', 'JUPYTERHUB_HOST': '', 'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/wvincent/oauth_callback', 'JUPYTERHUB_OAUTH_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]', 'JUPYTERHUB_USER': 'wvincent', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL': 'http://r9jupyter.domain.fr:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://r9jupyter.domain.fr:8081/hub/api/users/wvincent/activity', 'JUPYTERHUB_BASE_URL': '/', 'JUPYTERHUB_SERVICE_PREFIX': '/user/wvincent/', 'JUPYTERHUB_SERVICE_URL': 'http://0.0.0.0:0/user/wvincent/', 'JUPYTERHUB_ROOT_DIR': '~', 'JUPYTERHUB_DEFAULT_URL': '/lab', 'USER': 'wvincent', 'HOME': '/home/wvincent', 'SHELL': '/bin/bash'}
[I 2024-03-21 13:14:32.483 JupyterHub batchspawner:287] Job submitted. output: 179
[D 2024-03-21 13:14:32.484 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:14:32.551 JupyterHub batchspawner:428] Job 179 still pending
[W 2024-03-21 13:14:32.709 JupyterHub base:192] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I 2024-03-21 13:14:32.718 JupyterHub log:192] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@xxx.xxx.4.106) 1009.52ms
[D 2024-03-21 13:14:32.726 JupyterHub scopes:884] Checking access to /hub/spawn-pending/wvincent via scope servers
[D 2024-03-21 13:14:32.726 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/wvincent via servers
[I 2024-03-21 13:14:32.729 JupyterHub pages:399] wvincent is pending spawn
[I 2024-03-21 13:14:32.732 JupyterHub log:192] 200 GET /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@xxx.xxx.4.106) 9.60ms
[D 2024-03-21 13:14:32.868 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=01598a5386176f0279952a3b9632a07e7fce9a12aa53108973c83be9ec3473e7a59354876fab64bfeb01892eb503870183707aa03f207d7a94845ca7980c3819 (@xxx.xxx.4.106) 2.22ms
[D 2024-03-21 13:14:32.963 JupyterHub scopes:884] Checking access to /hub/api/users/wvincent/server/progress via scope read:servers
[D 2024-03-21 13:14:32.964 JupyterHub scopes:697] Argument-based access to /hub/api/users/wvincent/server/progress via read:servers
[D 2024-03-21 13:14:33.015 JupyterHub log:192] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@xxx.xxx.4.106) 1.72ms
[D 2024-03-21 13:14:33.053 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:14:33.124 JupyterHub batchspawner:428] Job 179 still pending
[D 2024-03-21 13:14:33.626 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:14:34.232 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:14:34.809 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:14:35.139 JupyterHub base:357] Recording first activity for <APIToken('1d78...', user='wvincent', client_id='jupyterhub')>
[I 2024-03-21 13:14:35.254 JupyterHub log:192] 200 POST /hub/api/batchspawner (wvincent@xxx.xxx.102.71) 121.02ms
[D 2024-03-21 13:14:35.381 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[I 2024-03-21 13:14:35.454 JupyterHub batchspawner:462] Notebook server job 179 started at r9nc-24-1:54049
[D 2024-03-21 13:14:35.535 JupyterHub spawner:1388] Polling subprocess every 30s
[D 2024-03-21 13:14:35.658 JupyterHub utils:286] Waiting 300s for server at http://r9nc-24-1:54049/user/wvincent/
[I 2024-03-21 13:14:38.597 JupyterHub log:192] 200 GET /hub/api (@xxx.xxx.102.71) 1.69ms
[D 2024-03-21 13:14:38.610 JupyterHub scopes:884] Checking access to /hub/api/users/wvincent/activity via scope users:activity
[D 2024-03-21 13:14:38.610 JupyterHub scopes:697] Argument-based access to /hub/api/users/wvincent/activity via users:activity
[D 2024-03-21 13:14:38.613 JupyterHub users:882] Activity for user wvincent: 2024-03-21T12:14:36.205701Z
[D 2024-03-21 13:14:38.613 JupyterHub users:900] Activity on server wvincent/: 2024-03-21T12:14:36.205701Z
[I 2024-03-21 13:14:38.623 JupyterHub log:192] 200 POST /hub/api/users/wvincent/activity (wvincent@xxx.xxx.102.71) 15.38ms
[D 2024-03-21 13:14:41.710 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[W 2024-03-21 13:14:41.801 JupyterHub base:1210] User wvincent is slow to become responsive (timeout=10)
[D 2024-03-21 13:14:41.801 JupyterHub base:1215] Expecting server for wvincent at: http://r9nc-24-1:54049/user/wvincent/
[D 2024-03-21 13:15:05.537 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:15:35.537 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:16:05.537 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:16:35.539 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:17:05.538 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:17:35.538 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
[D 2024-03-21 13:18:05.539 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 179 -o '%T %B'
13

Sorry, my previous message is an error, the process was launched locally, I’m still investigating.

14

These lines suggest that single user server found a random port and batchspawner sent that port information back to JupyterHub. So, technically the jupyter notebook in the singularity container should start at the port 54049. If it is not, I think you should check your singularity def file. Probably the port is set in the singularity def file. Could you share the def file of the singularity container image?

15

I don’t have anything special in my .def files, but what I assume is that the environment variables are not being passed to the container. So I’m trying to take all the variables and pass them through an env file, but it’s complicated because there are a lot of characters that need to be escaped.

16

It would be good to test if things work on 4.0.2, and then do 4.1.0 tests. Then you can rule out very recently released things cause issues first.

1 Like
17

I guess all JUPYTERHUB_* env vars are being passed into Singularity container (not sure why though) because the single user server authenticated itself with hub to send the port information back to hub. It is not possible to do it without the env var JUPYTERHUB_API_URL, JUPYTERHUB_OAUTH_SCOPES, JUPYTERHUB_OAUTH_ACCESS_SCOPES being set.

18

I have restored version 4.0.2 and it’s still the same.
Indeed, the variables seem to pass through the container properly, so the issue wouldn’t come from there.
Why is there a port on the server side, but on the client side and only through Singularity, the default port is used?
It would have been helpful in debugging to see what the node receives from the API and the environment variables.

19

These are the things you can do:

  • Upgrade JupyterLab to 4.x and Notebook to 7.x in the singularity container
  • Ensure that $HOME/.jupyter folder is empty inside the container where jupyter server might be getting some config from and redo the image
20

Ah yes, good catch!
I didn’t notice that the docker:nvcr.io/nvidia/pytorch:24.01-py3 image already had notebook installed.
I thought it was my jupyterhub installation that installed notebook, and it’s too bad it doesn’t update the dependency.
Installing notebook==7.1.2 in the container worked!

image
image1441×710 39.3 KB

[I 2024-03-22 12:41:49.231 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-03-22 12:41:49.232 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-03-22 12:41:49.254 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-03-22 12:41:49.254 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-03-22 12:41:49.256 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/dist-packages/jupyterlab
[I 2024-03-22 12:41:49.256 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-03-22 12:41:49.257 LabApp] Extension Manager is 'pypi'.
[I 2024-03-22 12:41:49.300 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-03-22 12:41:49.385 ServerApp] jupyterlab | extension was successfully loaded.
[W 2024-03-22 12:41:49.386 ServerApp] [Jupytext Server Extension] Async contents managers like AsyncLargeFileManager are not supported at the moment (https://github.com/mwouts/jupytext/issues/1020). We will derive a contents manager from LargeFileManager instead.
[I 2024-03-22 12:41:49.386 ServerApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[I 2024-03-22 12:41:49.387 ServerApp] jupyterlab_jupytext | extension was successfully loaded.
[I 2024-03-22 12:41:49.389 ServerApp] notebook | extension was successfully loaded.
[I 2024-03-22 12:41:49.389 ServerApp] Serving notebooks from local directory: /home/wvincent
[I 2024-03-22 12:41:49.389 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-03-22 12:41:49.389 ServerApp] http://r9nc-24-1:52467/user/wvincent/lab?token=...
[I 2024-03-22 12:41:49.389 ServerApp]     http://127.0.0.1:52467/user/wvincent/lab?token=...
[I 2024-03-22 12:41:49.389 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-03-22 12:41:49.599 ServerApp] 302 GET /user/wvincent/ -> /user/wvincent/lab? (@xxx.xxx.4.106) 0.32ms
[I 2024-03-22 12:41:49.643 ServerApp] 302 GET /user/wvincent/ -> /user/wvincent/lab? (@xxx.xxx.4.106) 0.30ms
[W 2024-03-22 12:41:49.665 ServerApp] No Hub user identified for request
[W 2024-03-22 12:41:49.665 ServerApp] Token stored in cookie may have expired
[I 2024-03-22 12:41:49.666 ServerApp] 302 GET /user/wvincent/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-wvincent&redirect_uri=%2Fuser%2Fwvincent%2Foauth_callback&response_type=code&state=[secret] (@xxx.xxx.4.106) 9.08ms
[I 2024-03-22 12:41:50.021 ServerApp] Logged-in user {'kind': 'user', 'name': 'wvincent', 'admin': False, 'groups': [], 'session_id': '8c6da92f4fa6429db660fbc6bdf8c0af', 'scopes': ['access:servers!server=wvincent/', 'read:users:groups!user=wvincent', 'read:users:name!user=wvincent']}
[W 2024-03-22 12:41:50.022 ServerApp] Token stored in cookie may have expired
[I 2024-03-22 12:41:50.022 ServerApp] 302 GET /user/wvincent/oauth_callback?code=[secret]&state=[secret] -> /user/wvincent/lab? (@xxx.xxx.4.106) 313.59ms
[I 2024-03-22 12:41:50.041 ServerApp] 200 GET /user/wvincent/lab? (wvincent@xxx.xxx.4.106) 8.37ms
[I 2024-03-22 12:41:50.886 ServerApp] 200 GET /user/wvincent/api/kernelspecs?1711107700733 (wvincent@xxx.xxx.4.106) 360.70ms
[I 2024-03-22 12:41:50.892 ServerApp] 200 GET /user/wvincent/api/me?1711107700733 (wvincent@xxx.xxx.4.106) 4.14ms
[I 2024-03-22 12:41:50.958 ServerApp] 200 GET /user/wvincent/lab/api/settings?1711107700737 (wvincent@xxx.xxx.4.106) 70.11ms
[I 2024-03-22 12:41:50.959 ServerApp] 200 GET /user/wvincent/api/kernels?1711107700742 (wvincent@xxx.xxx.4.106) 68.89ms
[I 2024-03-22 12:41:50.959 ServerApp] 200 GET /user/wvincent/api/terminals?1711107700742 (wvincent@xxx.xxx.4.106) 69.19ms
[I 2024-03-22 12:41:50.963 ServerApp] 200 GET /user/wvincent/api/sessions?1711107700742 (wvincent@xxx.xxx.4.106) 72.32ms
[I 2024-03-22 12:41:50.963 ServerApp] 101 GET /user/wvincent/api/events/subscribe?token=[secret] (wvincent@xxx.xxx.4.106) 72.69ms
[I 2024-03-22 12:41:50.965 ServerApp] 200 GET /user/wvincent/api/kernelspecs?1711107701106 (wvincent@xxx.xxx.4.106) 1.17ms
[I 2024-03-22 12:41:50.966 ServerApp] 200 GET /user/wvincent/api/me?1711107701110 (wvincent@xxx.xxx.4.106) 0.54ms
[I 2024-03-22 12:41:51.072 ServerApp] 200 GET /user/wvincent/lab/api/translations/default?1711107701255 (wvincent@xxx.xxx.4.106) 22.40ms
[I 2024-03-22 12:41:51.093 ServerApp] 200 GET /user/wvincent/api/config/jupyterlabapputilsextensionannouncements?1711107701298 (wvincent@xxx.xxx.4.106) 2.51ms
[W 2024-03-22 12:41:51.108 LabApp] Could not determine jupyterlab build status without nodejs
[I 2024-03-22 12:41:51.108 ServerApp] 200 GET /user/wvincent/lab/api/build?1711107701314 (wvincent@xxx.xxx.4.106) 1.30ms
[I 2024-03-22 12:41:51.150 ServerApp] 200 GET /user/wvincent/lsp/status?1711107701354 (wvincent@xxx.xxx.4.106) 1.97ms
[I 2024-03-22 12:41:51.330 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107701373 (wvincent@xxx.xxx.4.106) 149.68ms
[I 2024-03-22 12:41:51.332 ServerApp] 200 GET /user/wvincent/lab/api/translations?1711107701368 (wvincent@xxx.xxx.4.106) 164.99ms
[I 2024-03-22 12:41:51.338 ServerApp] 200 GET /user/wvincent/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1711107701425 (wvincent@xxx.xxx.4.106) 4.83ms
[I 2024-03-22 12:41:51.340 ServerApp] 200 GET /user/wvincent/lab/api/settings/@jupyterlab/notebook-extension:panel?1711107701431 (wvincent@xxx.xxx.4.106) 5.90ms
[I 2024-03-22 12:41:51.394 ServerApp] 200 GET /user/wvincent/lab/api/workspaces/default?1711107701598 (wvincent@xxx.xxx.4.106) 2.48ms
[I 2024-03-22 12:41:51.470 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107701653 (wvincent@xxx.xxx.4.106) 23.68ms
[I 2024-03-22 12:41:51.481 ServerApp] 200 GET /user/wvincent/api/terminals?1711107701685 (wvincent@xxx.xxx.4.106) 0.48ms
[I 2024-03-22 12:41:51.482 ServerApp] 200 GET /user/wvincent/api/terminals?1711107701685 (wvincent@xxx.xxx.4.106) 0.45ms
[I 2024-03-22 12:41:51.553 ServerApp] 200 GET /user/wvincent/api/sessions?1711107701749 (wvincent@xxx.xxx.4.106) 0.53ms
[I 2024-03-22 12:41:51.575 ServerApp] New terminal with automatic name: 1
[I 2024-03-22 12:41:51.576 ServerApp] 200 POST /user/wvincent/api/terminals?1711107701772 (wvincent@xxx.xxx.4.106) 9.22ms
[I 2024-03-22 12:41:51.588 ServerApp] New terminal with automatic name: 2
[I 2024-03-22 12:41:51.589 ServerApp] 200 POST /user/wvincent/api/terminals?1711107701773 (wvincent@xxx.xxx.4.106) 12.54ms
[I 2024-03-22 12:41:51.635 ServerApp] 200 GET /user/wvincent/api/terminals?1711107701793 (wvincent@xxx.xxx.4.106) 0.61ms
[I 2024-03-22 12:41:51.653 ServerApp] Kernel started: 5caac952-08c6-4243-a69e-e1bfe09739fe
[I 2024-03-22 12:41:51.653 ServerApp] 201 POST /user/wvincent/api/sessions?1711107701781 (wvincent@xxx.xxx.4.106) 76.07ms
[I 2024-03-22 12:41:51.690 ServerApp] 101 GET /user/wvincent/terminals/websocket/1 (wvincent@xxx.xxx.4.106) 0.85ms
[I 2024-03-22 12:41:51.717 ServerApp] 101 GET /user/wvincent/terminals/websocket/2 (wvincent@xxx.xxx.4.106) 0.54ms
[I 2024-03-22 12:41:51.731 ServerApp] 200 GET /user/wvincent/lab/api/settings?ids_only=true&1711107701918 (wvincent@xxx.xxx.4.106) 5.59ms
[I 2024-03-22 12:41:51.787 ServerApp] 200 GET /user/wvincent/api/sessions?1711107701993 (wvincent@xxx.xxx.4.106) 0.65ms
[I 2024-03-22 12:41:51.865 ServerApp] 200 PATCH /user/wvincent/api/sessions/474a2da7-6050-4b5d-9edf-c5f4e7ff1a87?1711107702070 (wvincent@xxx.xxx.4.106) 1.46ms
[I 2024-03-22 12:41:51.874 ServerApp] 200 GET /user/wvincent/api/kernels?1711107702080 (wvincent@xxx.xxx.4.106) 0.77ms
[I 2024-03-22 12:41:51.880 ServerApp] 200 PATCH /user/wvincent/api/sessions/474a2da7-6050-4b5d-9edf-c5f4e7ff1a87?1711107702085 (wvincent@xxx.xxx.4.106) 0.94ms
[I 2024-03-22 12:41:51.881 ServerApp] 200 GET /user/wvincent/api/sessions?1711107702086 (wvincent@xxx.xxx.4.106) 0.56ms
[I 2024-03-22 12:41:52.416 ServerApp] 204 PUT /user/wvincent/lab/api/workspaces/default?1711107702619 (wvincent@xxx.xxx.4.106) 2.26ms
[I 2024-03-22 12:41:52.498 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107702666 (wvincent@xxx.xxx.4.106) 39.85ms
[I 2024-03-22 12:41:53.506 ServerApp] 101 GET /user/wvincent/api/kernels/5caac952-08c6-4243-a69e-e1bfe09739fe/channels?session_id=fe177d02-338a-49b9-9174-7af861851633 (wvincent@xxx.xxx.4.106) 1639.83ms
[I 2024-03-22 12:41:53.507 ServerApp] Connecting to kernel 5caac952-08c6-4243-a69e-e1bfe09739fe.
[I 2024-03-22 12:41:53.538 ServerApp] Starting buffering for 5caac952-08c6-4243-a69e-e1bfe09739fe:fe177d02-338a-49b9-9174-7af861851633
[I 2024-03-22 12:41:53.544 ServerApp] 200 GET /user/wvincent/api/sessions?1711107703750 (wvincent@xxx.xxx.4.106) 0.75ms
[I 2024-03-22 12:41:53.545 ServerApp] 200 GET /user/wvincent/api/kernels?1711107703750 (wvincent@xxx.xxx.4.106) 0.59ms
[I 2024-03-22 12:41:53.594 ServerApp] 101 GET /user/wvincent/api/kernels/5caac952-08c6-4243-a69e-e1bfe09739fe/channels?session_id=507f5097-a44d-4fd8-b91f-84af9b083a7a (wvincent@xxx.xxx.4.106) 1.28ms
[I 2024-03-22 12:41:53.594 ServerApp] Connecting to kernel 5caac952-08c6-4243-a69e-e1bfe09739fe.
[I 2024-03-22 12:41:53.628 ServerApp] 101 GET /user/wvincent/api/kernels/5caac952-08c6-4243-a69e-e1bfe09739fe/channels?session_id=ecc943ab-ef94-4980-b225-1efc2556683e (wvincent@xxx.xxx.4.106) 3.30ms
[I 2024-03-22 12:41:53.628 ServerApp] Connecting to kernel 5caac952-08c6-4243-a69e-e1bfe09739fe.
[I 2024-03-22 12:41:53.665 ServerApp] 200 GET /user/wvincent/api/kernels?1711107703871 (wvincent@xxx.xxx.4.106) 0.52ms
[I 2024-03-22 12:41:53.666 ServerApp] 200 GET /user/wvincent/api/sessions?1711107703870 (wvincent@xxx.xxx.4.106) 0.57ms
[I 2024-03-22 12:42:01.665 ServerApp] 200 GET /user/wvincent/api/terminals?1711107711859 (wvincent@xxx.xxx.4.106) 0.56ms
[I 2024-03-22 12:42:02.568 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107712739 (wvincent@xxx.xxx.4.106) 36.58ms
[I 2024-03-22 12:42:03.685 ServerApp] 200 GET /user/wvincent/api/kernels?1711107713889 (wvincent@xxx.xxx.4.106) 0.52ms
[I 2024-03-22 12:42:03.686 ServerApp] 200 GET /user/wvincent/api/sessions?1711107713889 (wvincent@xxx.xxx.4.106) 0.55ms
[I 2024-03-22 12:42:11.699 ServerApp] 200 GET /user/wvincent/api/terminals?1711107721889 (wvincent@xxx.xxx.4.106) 0.50ms
[I 2024-03-22 12:42:12.626 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107722798 (wvincent@xxx.xxx.4.106) 34.52ms
[I 2024-03-22 12:42:13.702 ServerApp] 200 GET /user/wvincent/api/kernels?1711107723908 (wvincent@xxx.xxx.4.106) 0.54ms
[I 2024-03-22 12:42:13.703 ServerApp] 200 GET /user/wvincent/api/sessions?1711107723909 (wvincent@xxx.xxx.4.106) 0.53ms
[I 2024-03-22 12:42:21.729 ServerApp] 200 GET /user/wvincent/api/terminals?1711107731919 (wvincent@xxx.xxx.4.106) 0.52ms
[I 2024-03-22 12:42:22.686 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107732860 (wvincent@xxx.xxx.4.106) 32.71ms
[I 2024-03-22 12:42:23.724 ServerApp] 200 GET /user/wvincent/api/kernels?1711107733929 (wvincent@xxx.xxx.4.106) 0.52ms
[I 2024-03-22 12:42:23.725 ServerApp] 200 GET /user/wvincent/api/sessions?1711107733929 (wvincent@xxx.xxx.4.106) 0.53ms
[I 2024-03-22 12:42:31.759 ServerApp] 200 GET /user/wvincent/api/terminals?1711107741949 (wvincent@xxx.xxx.4.106) 0.48ms
[I 2024-03-22 12:42:32.748 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107742920 (wvincent@xxx.xxx.4.106) 34.32ms
[I 2024-03-22 12:42:33.745 ServerApp] 200 GET /user/wvincent/api/kernels?1711107743949 (wvincent@xxx.xxx.4.106) 0.51ms
[I 2024-03-22 12:42:33.746 ServerApp] 200 GET /user/wvincent/api/sessions?1711107743950 (wvincent@xxx.xxx.4.106) 0.53ms
[I 2024-03-22 12:42:41.786 ServerApp] 200 GET /user/wvincent/api/terminals?1711107751978 (wvincent@xxx.xxx.4.106) 0.48ms
[I 2024-03-22 12:42:42.805 ServerApp] 200 GET /user/wvincent/api/contents?content=1&hash=0&1711107752988 (wvincent@xxx.xxx.4.106) 23.27ms
[I 2024-03-22 12:42:43.765 ServerApp] 200 GET /user/wvincent/api/kernels?1711107753968 (wvincent@xxx.xxx.4.106) 0.50ms
[I 2024-03-22 12:42:43.765 ServerApp] 200 GET /user/wvincent/api/sessions?1711107753969 (wvincent@xxx.xxx.4.106) 0.53ms
[I 2024-03-22 12:42:51.820 ServerApp] 200 GET /user/wvincent/api/terminals?1711107762009 (wvincent@xxx.xxx.4.106) 0.47ms
[I 2024-03-22 12:42:52.056 ServerApp] 200 GET /user/wvincent/api/kernelspecs?1711107762259 (wvincent@xxx.xxx.4.106) 1.98ms
[I 2024-03-22 12:42:52.057 ServerApp] 200 GET /user/wvincent/api/me?1711107762260 (wvincent@xxx.xxx.4.106) 0.52ms

To summarize, in the Singularity images, I rebuild them with the following dependencies:

/usr/local/bin/pip3 install jupyterhub==4.0.2 || true
/usr/local/bin/pip3 install notebook==7.1.2 || true
/usr/local/bin/pip3 install https://github.com/jupyterhub/batchspawner/archive/main.zip

Thank you again for all the time you have given me.
Have a good day, William

2 Likes
21

Hello
I updated the packages and it broke a dependency. Since singularity images no longer run correctly jupyterhub-singleuser
I reinstalled everything, jupyterhub server, and the singularity images, in version 4.2.5, 4.1.2 and 4.0.2, nothing works

My server log :

[I 2024-04-11 13:11:31.517 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2024-04-11 13:11:31.517 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticat>
[I 2024-04-11 13:11:31.517 JupyterHub app:2889] Using Spawner: optionsspawner.optionsspawner.Option>
[I 2024-04-11 13:11:31.517 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy>
[I 2024-04-11 13:11:31.521 JupyterHub app:1664] Loading cookie_secret from /etc/jupyterhub/jupyterh>
[I 2024-04-11 13:11:31.607 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
[I 2024-04-11 13:11:31.607 alembic.runtime.migration migration:219] Will assume non-transactional D>
[I 2024-04-11 13:11:31.759 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2024-04-11 13:11:31.802 JupyterHub roles:238] Adding role admin for User: jthomaze
[I 2024-04-11 13:11:31.809 JupyterHub roles:238] Adding role user for User: jthomaze
[I 2024-04-11 13:11:31.821 JupyterHub roles:238] Adding role admin for User: wvincent
[I 2024-04-11 13:11:31.827 JupyterHub roles:238] Adding role user for User: wvincent
[I 2024-04-11 13:11:31.832 JupyterHub app:1984] Not using allowed_users. Any authenticated user wil>
[I 2024-04-11 13:11:31.874 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2024-04-11 13:11:31.908 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPerio>
[I 2024-04-11 13:11:31.909 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPerio>
[I 2024-04-11 13:11:31.910 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPerio>
[W 2024-04-11 13:11:31.910 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is S>
[I 2024-04-11 13:11:31.910 JupyterHub proxy:750] Starting proxy @ http://0.0.0.0:8000
[I 2024-04-11 13:11:32,093 configurable_http_proxy] Proxying http://0.0.0.0:8000 to (no default)
[I 2024-04-11 13:11:32,093 configurable_http_proxy] Proxy API at http://127.0.0.1:8001/api/routes
[I 2024-04-11 13:11:32,432 configurable_http_proxy] 200 GET /api/routes
[I 2024-04-11 13:11:32.433 JupyterHub app:3178] Hub API listening on http://r9jupyter.irit.fr:8081/>
[I 2024-04-11 13:11:32,435 configurable_http_proxy] 200 GET /api/routes
[I 2024-04-11 13:11:32.436 JupyterHub proxy:477] Adding route for Hub: / => http://r9jupyter.irit.f>
[I 2024-04-11 13:11:32,438 configurable_http_proxy] Adding route / -> http://r9jupyter.irit.fr:8081
[I 2024-04-11 13:11:32,438 configurable_http_proxy] Route added / -> http://r9jupyter.irit.fr:8081
[I 2024-04-11 13:11:32,438 configurable_http_proxy] 201 POST /api/routes/
[I 2024-04-11 13:11:32.439 JupyterHub app:3245] JupyterHub is now running at http://0.0.0.0:8000
[W 2024-04-11 13:11:48.713 JupyterHub base:422] Invalid cookie token
[I 2024-04-11 13:11:48.714 JupyterHub log:191] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (>
[I 2024-04-11 13:11:48.775 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2Fhome (@xxx.3.28>
_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=wvincent
[I 2024-04-11 13:11:57.852 JupyterHub base:837] User logged in: wvincent
[I 2024-04-11 13:11:57.853 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2Fhome -> /hub/home >
[W 2024-04-11 13:11:57.908 JupyterHub spawner:170]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.
    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.
[I 2024-04-11 13:11:57.920 JupyterHub log:191] 200 GET /hub/home (wvincent@xxx.3.28) 49.00ms
[I 2024-04-11 13:12:00.454 JupyterHub log:191] 200 GET /hub/spawn/wvincent (wvincent@xxx.3.28) >
[I 2024-04-11 13:12:04.187 JupyterHub provider:659] Creating oauth client jupyterhub-user-wvincent
[I 2024-04-11 13:12:04.373 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cl>
[I 2024-04-11 13:12:04.373 JupyterHub batchspawner:282] Spawner submitting command:  if [ "GPUNodes>
    080-dev"]
    then
    sudo -E -u wvincent sbatch --parsable --gres-flags=enforce-binding --gres=gpu:1
    else
    sudo -E -u wvincent sbatch --parsable
    fi
ot : PWD=/etc/jupyterhub ; USER=wvincent ; COMMAND=/bin/sbatch --parsable
[I 2024-04-11 13:12:04.709 JupyterHub batchspawner:287] Job submitted. output: 335
[W 2024-04-11 13:12:05.039 JupyterHub base:183] Rolling back dirty objects IdentitySet([<Server(:0)>
[I 2024-04-11 13:12:05.048 JupyterHub log:191] 302 POST /hub/spawn/wvincent?_xsrf=[secret] -> /hub/>
[I 2024-04-11 13:12:05.068 JupyterHub pages:398] wvincent is pending spawn
[I 2024-04-11 13:12:05.074 JupyterHub log:191] 200 GET /hub/spawn-pending/wvincent?_xsrf=[secret] (>
[I 2024-04-11 13:12:08.076 JupyterHub log:191] 200 POST /hub/api/batchspawner (wvincent@xxx.102>
[I 2024-04-11 13:12:08.263 JupyterHub batchspawner:455] Notebook server job 335 started at r9ng-108>
[I 2024-04-11 13:12:09.575 JupyterHub log:191] 200 GET /hub/api (@xxx.102.68) 1.35ms
[I 2024-04-11 13:12:09.659 JupyterHub log:191] 200 POST /hub/api/users/wvincent/activity (wvincent@>
[W 2024-04-11 13:12:14.059 JupyterHub base:1122] User wvincent is slow to become responsive (timeou>
[E 2024-04-11 13:12:25,249 configurable_http_proxy] 503 GET /hub/api/users/wvincent/server/progress>
[I 2024-04-11 13:12:25.266 JupyterHub log:191] 200 GET /hub/error/503?url=/hub/api/users/wvincent/s>
[E 2024-04-11 13:12:25,267 tornado.access] 503 GET /hub/api/users/wvincent/server/progress?_xsrf=2%>
[I 2024-04-11 13:16:32,445 configurable_http_proxy] 200 GET /api/routes
[I 2024-04-11 13:16:50.719 JupyterHub log:191] 200 POST /hub/api/users/wvincent/activity (wvincent@>
[W 2024-04-11 13:17:00.449 JupyterHub user:933] wvincent's server never showed up at http://r9ng-10>
    Common causes of this timeout, and debugging tips:
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.
[I 2024-04-11 13:17:00.471 JupyterHub batchspawner:469] Stopping server job 335
[I 2024-04-11 13:17:00.472 JupyterHub batchspawner:347] Cancelling job 335:  scancel 335
[W 2024-04-11 13:17:10.716 JupyterHub batchspawner:479] Notebook server job 335 at r9ng-1080-8:6063>
[E 2024-04-11 13:17:10.826 JupyterHub gen:630] Exception in Future <Task finished name='Task-40' co>
    Traceback (most recent call last):
      File "/usr/local/lib64/python3.9/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 988, in finis>
        await spawn_future
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/user.py", line 914, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/user.py", line 958, in _wait_up
        raise e
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/user.py", line 928, in _wait_up
        resp = await server.wait_up(
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/utils.py", line 289, in wait_for_http>
        re = await exponential_backoff(
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/utils.py", line 237, in exponential_b>
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Server at http://r9ng-1080-8:60631/user/wvincent/ didn't respo>
[W 2024-04-11 13:17:10.830 JupyterHub users:666] Stream closed while handling /hub/api/users/wvince>
[I 2024-04-11 13:17:10.831 JupyterHub log:191] 200 GET /hub/api/users/wvincent/server/progress?_xsr>
[I 2024-04-11 13:21:32,444 configurable_http_proxy] 200 GET /api/routes

Singularity / slurm log :

+ trap 'echo SIGTERM received' TERM
+ '[' /apps/containerCollections/CUDA12/tf2-NGC-24-02.sif = autre ']'
+ export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/tf2-NGC-24-02.sif
+ SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/tf2-NGC-24-02.sif
+ singularity exec --bind /home/wvincent:/run/user /apps/containerCollections/CUDA12/tf2-NGC-24-02.sif batchspawner-singleuser jupyterhub-singleuser
[I 2024-04-11 15:45:58.977 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-04-11 15:45:58.983 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-04-11 15:45:58.983 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2024-04-11 15:45:58.983 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
[I 2024-04-11 15:45:58.987 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-04-11 15:45:58.990 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-04-11 15:45:58.994 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-04-11 15:45:58.994 ServerApp] jupyterlab_jupytext | extension was successfully linked.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'allow_root' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'custom_display_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'custom_display_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'terminado_settings' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'contents_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-04-11 15:45:59.002 ServerApp] notebook | extension was successfully linked.
[I 2024-04-11 15:45:59.554 ServerApp] notebook_shim | extension was successfully linked.
/usr/local/lib/python3.10/dist-packages/jupyter_server/serverapp.py:2235: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/wvincent/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/wvincent/oauth_callback
  self.web_app = ServerWebApplication(
[I 2024-04-11 15:45:59.622 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-04-11 15:45:59.625 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-04-11 15:45:59.627 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-04-11 15:45:59.636 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-04-11 15:45:59.637 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-04-11 15:45:59.644 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/dist-packages/jupyterlab
[I 2024-04-11 15:45:59.644 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-04-11 15:45:59.645 LabApp] Extension Manager is 'pypi'.
[I 2024-04-11 15:45:59.677 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-04-11 15:45:59.725 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-04-11 15:45:59.725 ServerApp] [Jupytext Server Extension] NotebookApp.contents_manager_class is (a subclass of) jupytext.TextFileContentsManager already - OK
[I 2024-04-11 15:45:59.725 ServerApp] jupyterlab_jupytext | extension was successfully loaded.
[I 2024-04-11 15:45:59.732 ServerApp] notebook | extension was successfully loaded.
[I 2024-04-11 15:45:59.732 ServerApp] Serving notebooks from local directory: /home/wvincent
[I 2024-04-11 15:45:59.732 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-04-11 15:45:59.733 ServerApp] http://hostname:8888/user/wvincent/lab?token=...
[I 2024-04-11 15:45:59.733 ServerApp]     http://127.0.0.1:34387/user/wvincent/lab?token=...
[I 2024-04-11 15:45:59.733 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Thank you for your help