Running jupyter lab behind an apache proxy

JupyterLab
1

I’m trying to run jupyter lab behind a proxy server, so the code is running on an internal server, but it’s then accessed via a second server which is more widely visible. I’ve almost got it to work, but something is causing all of the websocket requests to fail (they give a 500 error), and I’m struggling to debug this.

An example of the code I’m using to launch the server is:

jupyter lab --debug --no-browser --ip=0.0.0.0 --port={port} --ServerApp.base_url=jupyterlab/{random_id} --ServerApp.token={token}

This seems to work OK. I’m then using these rules to proxy it to our public facing server:

ProxyPass /jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/kernels/ wss://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/kernels/
ProxyPassReverse /jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/kernels/ wss://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/kernels/
ProxyPass /jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/events/ wss://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/events/
ProxyPassReverse /jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/events/ wss://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/api/events/
ProxyPass /jupyterlab/ZGQCMLPSMHNQNWAISHRT/ http://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/
ProxyPassReverse /jupyterlab/ZGQCMLPSMHNQNWAISHRT/ http://compute-1-0:24063/jupyterlab/ZGQCMLPSMHNQNWAISHRT/

I can see the web interface, but calls to the websockets all fail with a 500 error. There are no relevant errors in the error log - I can see the web requests succeeding but the websocket requests don’t add anything to the log. I can see the kernel being started:

[D 2024-09-25 09:52:53.963 ServerApp] Starting kernel: ['/bi/apps/jupyterlab/latest/bin/python3', '-m', 'ipykernel_launcher', '-f', '/bi/home/andrewss/.local/share/jupyter/runtime/kernel-52
34417e-3c1e-4a46-b9b2-b0a7df33dcd7.json']
[D 2024-09-25 09:52:53.969 ServerApp] Connecting to: tcp://127.0.0.1:41619
[D 2024-09-25 09:52:53.985 ServerApp] Connecting to: tcp://127.0.0.1:36209
[I 2024-09-25 09:52:53.992 ServerApp] Kernel started: 5234417e-3c1e-4a46-b9b2-b0a7df33dcd7
[D 2024-09-25 09:52:53.993 ServerApp] Kernel args (excluding env): {'kernel_name': 'python3', 'cwd': '/bi/home/andrewss'}

…but nothing more.

Any suggestions for how to debug this? I presume the 500 error is generating an error message somewhere, but I can’t find where. Any previous experience of getting this to work?

Thanks

Simon.

2

Can you share your full Apache configuration, and your browser console logs? It sounds like your websockets configuration may be incomplete.

For comparison this is the suggested config for JupyterHub:

3

Thank you! This was helpful. I’d done some more debugging and identified the source(s) of my original problems. I found a solution which works which uses mod_proxy rather than rewrite but seems to work fine in the tests I’ve done.

The two issues which conspired were:

  1. I was forwarding to wss:// but actually the back end was just using ws:// I misunderstood the error I was getting and thought the 400 error was because I used the wrong protocol but it was actually because…
  2. I wasn’t preserving the headers I needed from the client so the websocket was rejecting the connection request.

The working config I now have relating to this connection is just:

ProxyPreserveHost On
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/ http://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/ http://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/