Use jupyter-server-proxy with hub in kubernetes

[seb31]

Hi all,
I’m struggling since a couple of days with jupyter-server-proxy setup:
Basically, I use a kubernetes cluster and I deployed z2jh helm chart on this cluster (I didn’t customize anything in the helm chart, except that I enabled “debug” in the values.yaml file)
In the k8s namespace, I can see a pod for the hub, a pod for the proxy, and once I spawned a single-user notebook, I can see an additional pod for user’s jupyter. Fine

The tricky point now
In my single-user’s notebook, from a terminal window, I install (pip install doccano) & start a webserver (doccano webserver --port 7860) that listens on a given port (7860 in my case) on the user’s pod.

I would like to reach out this web server from a browser but couldn’t figure out how to achieve this throught the different involved components (From a kubernetes perspective, I didn’t declare any additional service to point out to this pod/port):
My request basically goes to the proxy-public service that points to the proxy pod, and then cannot figure out what’s going on…
What should be the correct way for my request after reaching the proxy component ? Should it go to the hub or directly to the single user pod ?

After reading many posts & documentation, it is still unclear to me how to use/configure the “jupyter-server-proxy” component when used in combination with jupyterhub in kubernetes:

From what I see, the proxy pod (from z2jh helm chart) runs “configurable-http-proxy”. How it deals with “jupyter-server-proxy” ?
Do I have to install/configure something in the hub or the single-user (that both run in separate pods) ? the singleuser image already includes the jupyter-server-proxy python package from what I understand.

I can reach jupyterhub with this URL : “localhost:80”
I can reach user’s notebook with “localhost/user/MYUSER/lab/”
I failed to reach user’s doccano web server with following URLs:

• localhost/hub/user-redirect/proxy/7860/ => 404
• localhost/user/MYUSER/proxy/7860/ => 404

In the logs of my jupyter-MYUSER pod, I can see:

... 
[W 2023-05-05 08:29:49.974 SingleUserLabApp log:186] 404 GET /user/MYUSER/proxy/7860 
...

In the logs of the proxy pod, I can see:

...
08:29:49.969 [ConfigProxy] debug: PROXY WEB /user/MYUSER/proxy/7860 to http://<jupyter-MYUSER_IP>:8888
08:29:49.974 [ConfigProxy] debug: Not recording activity for status 404 on /user/MYUSER
...

=> It seems that the proxy redirects to port 8888 of my single user pod, which is the port used by jupyter notebook.
How to make the jupyter notebook redirecting to the 7860 port in the same pod ?

Any insight would be greatly appreciated…
Thanks in advance

[manics]

I think it’ll be easiest to build and debug your container image locally- everything is running inside your container.

A few things you can check:

• Is jupyter-server-proxy definitely working and configured?
• Check the logs for your doccano server, compare them with the requests you’re making. Is the 404 error coming from doccano or jupyter-server?

[seb31]

Hello manics,
Thanks for your help. I’m not sure I understand your suggestion:
Following your advice, I just tried out to run a jupyter-lab locally (i.e directly in a python environment on my computer, outside of any kubernetes environment).

I first tried to create a minimal env:

conda create --name jupyter-proxy-test
conda activate jupyter-proxy-test
conda install jupyterlab
pip install jupyter-server-proxy

and started my notebook with “jupyter-lab” command, but faced warning messages: “Debugger warning: It seems that frozen modules are being used…”

I struggled a bit but couldn’t understand exactly if jupyter-server-proxy is the right component to use in my case. Where/How should I install it ?
Additionally, is it compliant with jupyter-lab (or only with legacy “jupyter notebook”) ?

I also gave a try to jupyterlab “extensions” mechanism:

conda create --name jupyter-proxy-test
conda activate jupyter-proxy-test
conda install jupyterlab
jupyter labextension install @jupyterlab/server-proxy

And then I was able to start a notebook without previous warning about frozen modules. I could reach it with http://localhost:8888/lab URL
I installed doccano from a terminal in my jupyterlab:

pip install doccano

… and started it:

doccano webserver --port 9876 --workers 8

But couldn’t reach it out through the proxy as I was expecting:

• http://localhost:8888/lab/proxy/9876/
• http://localhost:8888/proxy/9876/

Did I miss something about the way all of this should work ?

[seb31]

A few additional details about the context (maybe I’ve not been very clear):
I want to give jupyter notebook’s users the ability to start a web server by themselves. I don’t want jupyter-server-proxy to start the web server itself, that’s why I didn’t provide any jupyter_notebook_config.py file as detailed in the documentation.

Anyway, I’m totally blind to understand what’s going on once I installed jupyter-server-proxy in the environment. Are there some logs files that could give me some clues ?

[manics]

jupyter-server-proxy has two components, a server side extension to jupyter-server, and an optional front-end extension that adds icons for installed applications but isn’t strictly necessary. Assuming you’re using latest version both parts are installed with the pip or conda packages.

The server side extension logs are part of the jupyter-server logs.

Can you try a simpler http server to test? Running

python -mhttp.server 12345

Will start a python webserver on port 12345. Can you access that via the proxy? Please share the logs from jupyter-server and from the python http server.

[bytesemantics]

So using doccano (which “runs through” its own web server process) is similar to how plotly dash, bokeh panel, atoti all work. They all expect a webserver to serve their front ends.

There are a number of examples of jupyter-server-proxy configurations detailed for both plotly dash and bokeh panel. Search for “jupyter-server-proxy configuration dash” (or panel) and specifically read the search results from the renku sites. They point to working git repo examples.

Hope this helps

[seb31]

Okay, I guess I was not really clear about what I want to do, and you already noted I’m new with jupyter concepts

Basic scenario:
We want to give the ability to our jupyter users to start their own web-server “from the jupyter environment” - i.e cells of their notebook or “Terminal” launcher they can use - and to be able to reach this server from another tab of their browser, going through the jupyterlab URL & port.

I had understood that jupyter was able to “proxy” processes that were started by its users, i.e making them accessible from the external world by using the same URL as the jupyter one, enriched with “/proxy/”.
Am I correct ?

I tried to narrow the scope of my tests to get rid of kubernetes and to understand how jupyter behaves by itself. Let’s try to sum up:

I created a brand new python environment on my laptop:

conda create --name my-proxy-test
conda activate my-proxy-test
conda install jupyterlab
pip install jupyter-server-proxy

At this point, I am able to start “jupyter lab”. In the console, it suggests a URL to use to display jupyterlab web UI in my browser:

...
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=xxxxxxxx

If I copy/paste this URL in a tab of my browser, I get in the jupyterlab UI (the “final” URL, displayed in the top bar of my tab, is “http://localhost:8888/lab”), great

If I select “File > New > Terminal”, it opens a new “terminal” tab in jupyterlab, and I can start a basic web server as you suggested:

python -mhttp.server 12345

Then I am able to open a new tab in my browser, using the URL “http://localhost:8888/proxy/12345/”, and it works fine

Now, my problem happens when I start using a “more sophisticated” stuff: let’s say I want to use “gradio” for instance. I write following code in my notebook cell:

import gradio as gr
def greet(name):
    return "Hello " + name + "!"
demo = gr.Interface(fn=greet, inputs="text", outputs="text")
demo.launch()   

When I execute the cell, I obtain the gradio UI as the “output” of my cell, in addition to following indication:

"Running on local URL:  http://127.0.0.1:7860"

But when I try to open another tab with http://localhost:8888/proxy/7860/, it fails to display gradio UI and I can see following in the logs of my jupyter:

[W 2023-05-11 14:24:06.640 ServerApp] 404 GET /theme.css (8a35b8ce150947ac906342bc60aee6bb@127.0.0.1) 2.73ms referer=http://localhost:8888/proxy/7860/

Though, if I try to access http://localhost:8888/proxy/7860/theme.css in another tab of my browser, the css file displays correctly…

Thus I guess it is a problem on gradio side, that does not properly manage the case where it runs behind a proxy…? Or did I miss something else ?

Thanks again for your great support

[manics]

It sounds like your webapp is expecting to be served from the root of the domain /, but since it’s being proxied it’s actually being served at /proxy/7860/. One thing you can try is /proxy/absolute/7860/ which sends the unmodified URL to the proxied application: Accessing Arbitrary Ports or Hosts — Jupyter Server Proxy documentation

If that doesn’t work then you need to tell your application (gradio, docano, etc) that it should prefix all URLs it generates with /proxy/7860 or similar.

[gyan_prakash]

@seb31 , did you managed to solve this , i am stuck in very similar problem. In my use case, user is statrting a app (service) at port 5151. And i am not finding a way to access it. I am using k8 and i also tried /proxy/5151/ but didn’t help

[seb31]

Hello ,
In the end, I created a kubernetes service+ingress to reach my application from outside of the cluster. No need to use jupyter-server-proxy in this case

[gyan_prakash]

Thanks @seb31 , how do you target user pod in ingress , specially if some user create this service , and some not

[seb31]

you’re right, that’s one of the drawback: I only permit that one (hardcoded) user will start the server. The k8s service uses a labelSelector based on a custom label (user ID) that I added to the single-user pod