Jupyerhub with ssh portforwarding wss fails with different port

ihatethecloud · June 3, 2024, 5:56pm

Hello,

I’m trying to access Jupyterhub by using a ssh jump server.
Jupyterhub runs behind an nginx and the reverse proxy configuration is the one in the documentation.

Using this command I can access https://127.0.0.1 just fine and everything works as expected.

ssh -4 -L 127.0.0.1:443:jupyterhub-host:443 jump-host

Using a different port prevents the browser to connect to the web socket

 ssh -4 -L 127.0.0.1:9443:jupyterhub-host:443 jump-host

In the browser console I see:

WebSocket connection to '
default.js: 73
wSS://127.0.0.1:9443/jupyterhub/user/admin/api/kernels/aa41f8c2-4cea-4561-8490-429aec727aff/c
• failed

Did anybody encounter this or know what’s the reason?

Thank you!

minrk · June 4, 2024, 6:05am

Can you share logs from the singleuser server you are trying to connect to, and perhaps the apache configuration? And your jupyterhub version?

ihatethecloud · June 4, 2024, 11:01am

Sure thing. Sorry about apache… I meant nginx. I’ll edit if possible.
jupyterhub version: 2.3.1

nginx config

#jupyterhub
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
.....
   location /jupyterhub/ {
        proxy_pass "http://127.0.0.1:8000";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Scheme $scheme;

        proxy_buffering off;
    }
}

jupyterhub_config.py

c.JupyterHub.bind_url = 'http://:8000/jupyterhub/'

Log:

[I 2024-06-04 10:57:19.877 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernelspecs?1717498637595 (admin@192.168.1.10) 4.08ms
[I 2024-06-04 10:57:19.927 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings?1717498637596 (admin@192.168.1.10) 48.81ms
[I 2024-06-04 10:57:19.931 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498637598 (admin@192.168.1.10) 3.64ms
[I 2024-06-04 10:57:19.932 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498637598 (admin@192.168.1.10) 2.61ms
[I 2024-06-04 10:57:19.932 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498637598 (admin@192.168.1.10) 2.50ms
[I 2024-06-04 10:57:20.088 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernelspecs?1717498637807 (admin@192.168.1.10) 4.13ms
[I 2024-06-04 10:57:21.261 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/translations/en?1717498638939 (admin@192.168.1.10) 40.39ms
[I 2024-06-04 10:57:21.492 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/listings/@jupyterlab/extensionmanager-extension/listings.json?1717498639213 (admin@192.168.1.10) 3.00ms
[I 2024-06-04 10:57:21.495 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498639215 (admin@192.168.1.10) 2.53ms
[I 2024-06-04 10:57:21.532 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings/@jupyterlab/mainmenu-extension:plugin?1717498639240 (admin@192.168.1.10) 13.40ms
[I 2024-06-04 10:57:21.535 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1717498639240 (admin@192.168.1.10) 11.78ms
[I 2024-06-04 10:57:21.542 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1717498639240 (admin@192.168.1.10) 6.79ms
[I 2024-06-04 10:57:21.775 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/translations/?1717498639478 (admin@192.168.1.10) 23.18ms
[I 2024-06-04 10:57:21.826 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/workspaces/default?1717498639541 (admin@192.168.1.10) 5.81ms
[I 2024-06-04 10:57:21.916 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings/@jupyterlab/application-extension:context-menu?1717498639550 (admin@192.168.1.10) 14.46ms
[I 2024-06-04 10:57:21.993 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1717498639550 (admin@192.168.1.10) 14.08ms
[I 2024-06-04 10:57:22.237 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/settings?1717498639778 (admin@192.168.1.10) 181.02ms
[I 2024-06-04 10:57:22.528 LabApp] Build is up to date
[I 2024-06-04 10:57:22.530 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/build?1717498639200 (admin@192.168.1.10) 1054.34ms
[I 2024-06-04 10:57:22.568 SingleUserLabApp log:189] 204 PUT /jupyterhub/user/admin/lab/api/workspaces/default?1717498640287 (admin@192.168.1.10) 3.82ms
[I 2024-06-04 10:57:22.821 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498640538 (admin@192.168.1.10) 9.19ms
[I 2024-06-04 10:57:23.107 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/nbconvert?1717498639493 (admin@192.168.1.10) 1331.16ms
[I 2024-06-04 10:57:23.125 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/nbconvert?1717498639492 (admin@192.168.1.10) 1347.83ms
[I 2024-06-04 10:57:23.323 SingleUserLabApp log:189] 204 PUT /jupyterhub/user/admin/lab/api/workspaces/default?1717498641045 (admin@192.168.1.10) 1.64ms
[I 2024-06-04 10:57:30.213 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498647927 (admin@192.168.1.10) 2.82ms
[I 2024-06-04 10:57:30.218 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498647926 (admin@192.168.1.10) 3.52ms
[I 2024-06-04 10:57:30.218 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498647927 (admin@192.168.1.10) 2.76ms
[I 2024-06-04 10:57:33.033 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498650750 (admin@192.168.1.10) 3.87ms
[I 2024-06-04 10:57:40.430 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498658142 (admin@192.168.1.10) 2.36ms
[I 2024-06-04 10:57:40.434 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498658143 (admin@192.168.1.10) 2.93ms
[I 2024-06-04 10:57:40.435 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498658143 (admin@192.168.1.10) 2.32ms
[I 2024-06-04 10:57:43.242 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498660960 (admin@192.168.1.10) 1.81ms
[I 2024-06-04 10:57:50.642 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498668358 (admin@192.168.1.10) 2.09ms
[I 2024-06-04 10:57:50.643 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498668358 (admin@192.168.1.10) 1.79ms
[I 2024-06-04 10:57:50.644 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498668359 (admin@192.168.1.10) 1.14ms
[I 2024-06-04 10:57:52.463 JupyterHub log:189] 200 POST /jupyterhub/hub/api/users/admin/activity (admin@127.0.0.1) 7.84ms
[I 2024-06-04 10:57:53.448 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498671169 (admin@192.168.1.10) 1.81ms
[I 2024-06-04 10:58:00.852 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498678572 (admin@192.168.1.10) 1.35ms
[I 2024-06-04 10:58:00.859 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498678571 (admin@192.168.1.10) 2.04ms
[I 2024-06-04 10:58:00.860 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498678574 (admin@192.168.1.10) 1.73ms
[I 2024-06-04 10:58:03.661 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/contents?content=1&1717498681376 (admin@192.168.1.10) 2.02ms
[I 2024-06-04 10:58:11.064 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/kernels?1717498688780 (admin@192.168.1.10) 2.58ms
[I 2024-06-04 10:58:11.076 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/terminals?1717498688796 (admin@192.168.1.10) 2.17ms
[I 2024-06-04 10:58:11.080 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/api/sessions?1717498688795 (admin@192.168.1.10) 2.42ms

mahendrapaipuri · June 4, 2024, 11:36am

I suspect it must be a CORS issue that you are facing. Try changing the following in your nginx config:

proxy_set_header Host $http_host;

which will set port in the Host header always.

ihatethecloud · June 4, 2024, 12:18pm

Oh man… it worked.
@mahendrapaipuri could you please tell me why it worked?

minrk · June 4, 2024, 1:15pm

old versions of JupyterHub check the Host and Origin headers to verify same-origin requests. If there’s a port mismatch, it’s not allowed.

This is no longer the case in JupyterHub 4, I think.

Topic		Replies	Views
"Server unavailable or unreachable" and websocket connection failure (Docker, Nginx) JupyterHub	14	349	September 6, 2024
Web Socket Connection Failed JupyterHub community , how-to , help-wanted	4	1865	November 6, 2024
Containers with Apache Reverse Proxy + JupyterHub JupyterHub jupyterhub , help-wanted	4	166	July 30, 2024
JupyterHub Apache Proxy Reverse not working JupyterHub help-wanted	2	1068	May 3, 2021
Technical Reference :Using a reverse proxy JupyterHub jupyterhub , how-to	4	827	July 13, 2021

Jupyerhub with ssh portforwarding wss fails with different port

Related topics