Jupyter hub issue on lxd container

JupyterHub
, ,
1

I’m installing jupyterhub with jupyter-lab into a lxd container, i have a nginx frontend that receive https request then route them to the port 8000.
The same configuration works in a physical host, but in the lxd container, with the same linux distro (ubuntu 22.04), i got a lot of error:

“message”: “Action is not authorized with current scopes; requires any of [read:servers]”

The only differences is that i use on the frontend a different port for https, but i can see the login and i can login into the hub.

Anyone has this issue with containers ?

igor

2

There shouldn’t be anything specific to containers, but more log context would help.

My guess is it’s XSRF or cookie related, and ‘current scopes’ is really ‘no permissions at all’ because some credential is not accepted. Sometimes this has to do with the proxy headers.

Make sure to include the JupyterHub startup logs, which include some things about versions, and ideally pip list or conda list, and (redacted) jupyterhub configuration.

3

I use the same reverse proxy for both jupyterhub, the only difference is the port, the first is working, the second not…

4

Please can you provide the information requested earlier, i.e. your package versions and JupyterHub configuration?

It would also be helpful to see your reverse proxy configuration.

5

This is my reverse proxy:

map $http_upgrade2 $connection_upgrade2 {
    default upgrade;
    '' close;
}

server {
    listen 7443 ssl http2;

    server_name         gpu1.retis.local.santannapisa.it;
    ssl_certificate     /etc/nginx/ssl/gpu1.crt;
    ssl_certificate_key /etc/nginx/ssl/gpu1.key;
    add_header Strict-Transport-Security max-age=15768000;
    
    location / {
 	proxy_pass http://192.168.171.20:8000;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade2;
        proxy_set_header Connection $connection_upgrade2;
        proxy_set_header X-Scheme $scheme;

	proxy_buffering off;
    }
}

in my jupyterhub_config.py i changed only the line:

c.Spawner.default_url = ‘/lab’

the config file is shipped with ubuntu 22.04 jupytherhub package

igor

6

can you share the logs for jupyterhub startup and surrounding the error? Ideally with --debug?

7 
$ sudo jupyterhub --debug
[D 2025-02-27 18:37:50.491 JupyterHub application:731] Looking for jupyterhub_config in /var/log
[I 2025-02-27 18:37:50.492 JupyterHub app:2717] Running JupyterHub version 2.0.0
[I 2025-02-27 18:37:50.492 JupyterHub app:2747] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.0.0
[I 2025-02-27 18:37:50.492 JupyterHub app:2747] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.0.0
[I 2025-02-27 18:37:50.492 JupyterHub app:2747] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.0.0
[D 2025-02-27 18:37:50.494 JupyterHub app:2678] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[D 2025-02-27 18:37:50.494 JupyterHub app:1636] Generating new cookie_secret
[I 2025-02-27 18:37:50.494 JupyterHub app:1641] Writing cookie_secret to /var/log/jupyterhub_cookie_secret
[D 2025-02-27 18:37:50.495 JupyterHub app:1763] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2025-02-27 18:37:50.510 JupyterHub orm:924] Stamping empty database with alembic revision 833da8570507
[I 2025-02-27 18:37:50.513 alembic.runtime.migration migration:201] Context impl SQLiteImpl.
[I 2025-02-27 18:37:50.513 alembic.runtime.migration migration:204] Will assume non-transactional DDL.
[I 2025-02-27 18:37:50.519 alembic.runtime.migration migration:615] Running stamp_revision  -> 833da8570507
[D 2025-02-27 18:37:50.519 alembic.runtime.migration migration:815] new branch insert 833da8570507
[I 2025-02-27 18:37:50.623 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2025-02-27 18:37:50.623 JupyterHub app:2007] Loading default roles to database
[I 2025-02-27 18:37:50.645 JupyterHub app:1913] Not using allowed_users. Any authenticated user will be allowed.
[D 2025-02-27 18:37:50.649 JupyterHub app:2229] Purging expired APITokens
[D 2025-02-27 18:37:50.651 JupyterHub app:2229] Purging expired OAuthCodes
[D 2025-02-27 18:37:50.652 JupyterHub app:2080] Loading predefined roles from config file to database
[W 2025-02-27 18:37:50.655 JupyterHub app:2152] No admin role found; assuming hub upgrade. Initializing default roles for all entities
[D 2025-02-27 18:37:50.667 JupyterHub app:2375] Initializing spawners
[D 2025-02-27 18:37:50.669 JupyterHub app:2506] Loaded users:

[I 2025-02-27 18:37:50.669 JupyterHub app:2786] Initialized 0 spawners in 0.002 seconds
[W 2025-02-27 18:37:50.671 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2025-02-27 18:37:50.671 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2025-02-27 18:37:50.671 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2025-02-27 18:37:50.672 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
18:37:50.929 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:37:50.931 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2025-02-27 18:37:51.244 JupyterHub proxy:728] Proxy started and appears to be up
[D 2025-02-27 18:37:51.247 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:37:51.255 [ConfigProxy] info: 200 GET /api/routes
[I 2025-02-27 18:37:51.255 JupyterHub app:3035] Hub API listening on http://127.0.0.1:8081/hub/
[D 2025-02-27 18:37:51.255 JupyterHub proxy:343] Fetching routes to check
[D 2025-02-27 18:37:51.255 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:37:51.257 [ConfigProxy] info: 200 GET /api/routes
[D 2025-02-27 18:37:51.257 JupyterHub proxy:346] Checking routes
[I 2025-02-27 18:37:51.257 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
[D 2025-02-27 18:37:51.257 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
18:37:51.259 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
18:37:51.259 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
18:37:51.260 [ConfigProxy] info: 201 POST /api/routes/
[I 2025-02-27 18:37:51.260 JupyterHub app:3101] JupyterHub is now running at http://:8000
[D 2025-02-27 18:37:51.260 JupyterHub app:2710] It took 0.775 seconds for the Hub to start
8

Your JupyterHub is very old, so, the first thing you can try is to use more recent version. And the logs you shared do not have any errors!! Share more logs that show errors!

1 Like
9

I update jupyterhub and jupyterlab, now i can login, but if launch a kernel i get this error in the browser:

WebSocket connection to 'jlab_core.3e79afb39b563f309a5d.js?v=3e79afb39b563f309a5d:1 WebSocket connection to 'wss://myhostname:7443/user/ubuntu/terminals/websocket/1'

the ssl certificate is valid.

Thanks,
Igor

10

and can you share the logs surrounding the error? From both jupyterhub and the singleuser server?

11

In jupyterhub logs:

Mar 03 18:07:14 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:14.231 ServerApp] Skipping XSRF check for insecure request GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2/channels
Mar 03 18:07:14 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:14.233 ServerApp] Replacing stale connection: c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2:ae220289-dd09-4ea8-9197-ac1fd498814d
Mar 03 18:07:14 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:14.234 ServerApp] 400 GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2/channels?session_id=ae220289-dd09-4ea8-9197-ac1fd498814d (ubuntu@193.205.82.17) 3.73ms
Mar 03 18:07:14 biorobotica jupyterhub[841]: [I 2025-03-03 18:07:14.367 ServerApp] 200 GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2?1741021633182 (ubuntu@193.205.82.17) 1.13ms
Mar 03 18:07:20 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:20.653 ServerApp] Skipping XSRF check for insecure request GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2/channels
Mar 03 18:07:20 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:20.656 ServerApp] Replacing stale connection: c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2:e4320c2f-705f-4959-a870-353870b95c56
Mar 03 18:07:20 biorobotica jupyterhub[841]: [W 2025-03-03 18:07:20.657 ServerApp] 400 GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2/channels?session_id=e4320c2f-705f-4959-a870-353870b95c56 (ubuntu@193.205.82.17) 5.05ms
Mar 03 18:07:20 biorobotica jupyterhub[841]: [I 2025-03-03 18:07:20.786 ServerApp] 200 GET /user/ubuntu/api/kernels/c96f7bbf-9baa-49c3-80c5-b5b42dd8c2f2?1741021639604 (ubuntu@193.205.82.17) 1.13ms
Mar 03 18:07:43 biorobotica jupyterhub[841]: [I 2025-03-03 18:07:43.497 ServerApp] 200 GET /user/ubuntu/api/contents/Untitled1.ipynb?content=0&hash=1&1741021662351 (ubuntu@193.205.82.17) 2.47ms

repeated

where can I find the singleuser server log ?

Igor

12

that’s the singleuser logs (identified by ServerApp), while jupyterhub will contain JupyterHub.

The 400 error connecting to kernels is peculiar, and suggests to me that you are hitting this error, where the Upgrade header is not proxied properly. But it looks from your nginx like this shouldn’t be the case. I’m not 100% sure what to try next.

13

Ah, I see the problem. It’s the variable name $http_upgrade2. This should be $http_upgrade. The variable $http_upgrade2 is not defined, so it will always be empty.

I made this little tool recently to test websocket proxying.

1 Like
14

Finally i got it works, with this nginx configuration:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 7443 ssl http2;

    server_name      myserver;
    ssl_certificate     /etc/nginx/ssl/gpu1.crt;
    ssl_certificate_key /etc/nginx/ssl/gpu1.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_stapling on;
    ssl_stapling_verify on;
    add_header Strict-Transport-Security max-age=15768000;

    location / {
        proxy_pass http://192.168.171.20:8000;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Scheme $scheme;

        proxy_buffering off;
    }
}

thanks to everyone for the help !

Igor

1 Like