Cant connect to kernel

guyq1997 · March 22, 2024, 11:25am

Hello guys,
I am using Apache as reverseproxy.

RequestHeader unset Accept-Encoding
ProxyPass         http://jupyterhub:8000/jupyterhub
ProxyPassReverse  http://jupyterhub:8000/jupyterhub

RewriteEngine On
ProxyPreserveHost on

AuthType shibboleth
ShibRequestSetting requireSession 1
Require valid-user
ShibUseHeaders On
RequestHeader set X-Remote-User %{REMOTE_USER}s

This error occurs in my single user server

And This is my Jupyterhub_config.py

c.Spawner.cmd = “jupyterhub-singleuser”
c.Spawner.debug = True
c.ConfigurableHTTPProxy.debug = True
c.DockerSpawmer.use_internal_ip = True
c.JupyterHub.spawner_class = DockerSpawner
c.DockerSpawner.image = “xxx/jupyterlab”

c.JupyterHub.hub_ip = “0.0.0.0”
c.JupyterHub.bind_url = ‘http://:8000/jupyterhub’

c.DockerSpawner.network_name = os.environ[“DOCKER_NETWORK_NAME”]
c.DockerSpawner.extra_host_config = { ‘network_mode’: os.environ[“DOCKER_NETWORK_NAME”] }
c.DockerSpawner.remove_containers = True
c.DockerSpawner.start_timeout = 120
c.JupyterHub.authenticator_class = ShibbolethAuthenticator

I can spawn and access Single user server, but i cant open a notebook and connect to kernel. The Eorror Code:

[W 2024-03-22 11:18:42.287 ServerApp] 400 GET /jupyterhub/user/yuqiang/api/events/subscribe?token=[secret] (yuqiang@::ffff:172.20.0.3) 2.13ms

[W 2024-03-22 11:18:49.038 ServerApp] 400 GET /jupyterhub/user/yuqiang/api/events/subscribe?token=[secret] (yuqiang@::ffff:172.20.0.3) 1.76ms

guyq1997 · March 22, 2024, 1:27pm

I found a not fancy solution in my case.
I manually redirect this failed request in Apache proxy like this:

        <Location "/jupyterhub/***[username]***/api/kernels">
            ProxyPass ws://localhost:8000/jupyterhub/***[username]***/api/kernels
            ProxyPassReverse ws://localhost:8000/jupyterhub/***[username]***/api/kernels
        </Location>

But it is really dumm, that, i have to add this code for all 30 users. (for every username)
Is there any better way to automatically redirect this request, like Regular expression?

mahendrapaipuri · March 22, 2024, 1:39pm

Here is a reference Apache proxy config that should work with JupyterHub → Using a reverse proxy — JupyterHub documentation

I guess you are missing webscokets connection upgrade directives in your config

guyq1997 · March 22, 2024, 3:21pm

Hi,

Thanks. You are right! I just missed the websockets upgrade part!
I can now acess to my content. But It starts to restart my single user server all the time.

And in Logs i found this:

15:22:57.392 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.393 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.395 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.397 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.399 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang

guyq1997 · March 26, 2024, 4:20pm

Another Problem:
And strange things happen, if I activate the Jupyter-collaboration extension.
I can normally open a notebook in my home directory (/home/joyvan)
But can not connect to Kernel and open a notebook in a folder (/home/joyvan/[some newfolder]).

The Cause according my guess:
This Problem doesn´t exsit if i only use Jupyterhub internal Proxy (CHP). So I guess the problem is the compatibility of Jupyter-collaboration extension and Apache2?

Logs seem good:

[I 2024-03-26 16:26:47.448 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/contents/Untitled%20Folder?content=1&1711470407345 (yuqiang@172.22.0.4) 4.89ms
[I 2024-03-26 16:26:50.123 ServerApp] 200 GET /jupyterhub/user/yuqiang/api (@172.22.0.5) 0.78ms
[I 2024-03-26 16:26:52.162 YDocExtension] Processed 366 Y patches in one minute
[I 2024-03-26 16:26:52.162 YDocExtension] Connected Y users: 8
[I 2024-03-26 16:26:52.563 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/terminals?1711470412349 (yuqiang@172.22.0.4) 2.14ms
[I 2024-03-26 16:26:52.749 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/me?1711470412533 (yuqiang@172.22.0.4) 1.96ms
[I 2024-03-26 16:26:52.771 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/kernelspecs?1711470412535 (yuqiang@172.22.0.4) 2.58ms
[I 2024-03-26 16:26:55.327 ServerApp] 200 GET /jupyterhub/user/yuqiang/api (@172.22.0.5) 0.78ms

manics · March 26, 2024, 4:40pm

What do your browser console logs (https://help.zapier.com/hc/en-us/articles/8496199434765-View-and-save-your-browser-console-logs) and Apache logs show?

guyq1997 · March 26, 2024, 4:57pm

There is Error in Browser Logs!:

There is nothing strange accrodding to Apache Logs:

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/terminals?1711471394311 HTTP/1.1” 200 823 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb” "

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/contents/Untitled%20Folder?content=1&1711471394574 HTTP/1.1” 200 1093 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb”

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/kernels?1711471394421 HTTP/1.1” 200 3475 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb” “Mozilla/5.0 (iPhone; CPU iPhone OS 17_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1”

guyq1997 · March 27, 2024, 12:07pm

The problematic error Log is likely this: Browser fails to find the URL of directory (https://jupyterhub//user/yuqiang/api/collaboration/session/Untitled%20Folder%2FUntitled1.ipynb?1711541015975)

And again, this problem doesn´t exist if a disable the jupyter-collaboration extention.

guyq1997 · March 28, 2024, 3:46pm

A new version of Jupyterlab==4.1.5 also doesn´t help.

I found that the single user server and my browser are opening different URL. Single user server found the right url but My Browser doesn´t.

Browser says it can´t open the Note book because following URL doesn´t exist

/user/yuqiang/api/collaboration/session/Assignment%2FVisualisierung_CO2_Konzentration.ipynb 404 (Not Found)

Single user Server says that, it can successfully access to it:

[I 2024-03-28 15:41:38.678 YDocExtension] Processed 53 Y patches in one minute
[I 2024-03-28 15:41:38.678 YDocExtension] Connected Y users: 3
[I 2024-03-28 15:41:40.648 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/contents/Assignment/Visualisierung_CO2_Konzentration.ipynb?type=notebook&content=0 (yuqiang@::ffff:172.31.0.3) 3.34ms

guyq1997 · April 8, 2024, 9:16pm

I found the problematic Request is sent by Apache Webserver. The Request sent by Apche seems to be blocked. Other Requests, which are sent by tornado, are no problem.

Any suggestions to let tornado send the request instead of Apache?

minrk · April 10, 2024, 9:01am

I’m not entirely sure what’s going on here, but if Apache is responding, this means that some routing rule in Apache is not forwarding the request to JupyterHub. That should all be in the apache config, since it means JupyterHub hasn’t had a chance to doesn’t get involved.

Since it’s a subset of URLs, I wonder if some other config is catching URLs with e.g. session in them.

Another possible option that I’m not sure about is a version mismatch in the jupyter-collaboration javascript and Python versions. Making sure both are up to date and restarting the server might help.

If you request a URL that definitely 404s and will be served by JupyterLab: /jupyterhub/user/yuqiang/api/nosuchthing/should/404

do you see it served by Apache or Tornado?

guyq1997 · April 12, 2024, 1:30pm

Hello @minrk ,

I am using python ==3.11.6 and jupyter-collaboration==2.0.11.

If i request URL eample.org/jupyterhub/user/yuqiang/api/nosuchthing/should/404, which is 404. It is served by Tornado.

guyq1997 · April 15, 2024, 8:16am

Tornado will serve the following URL
/jupyterhub/user/udo.heuser/api/collaboration/session/Assignment
But not
/jupyterhub/user/udo.heuser/api/collaboration/session/Assignment%2F…
I got a Apache error with this URL

Where Assignment is a directory in my workspace

So I think the %2F or % is the problem, that makes Apache confused, and Tornado can´t recive the request.

guyq1997 · April 15, 2024, 8:29am

Problem solved!
Add this to Apache confiiguration. If you want to use jupyter-collaboration with Apache as reverse proxy!

AllowEncodedSlashes On

Topic		Replies	Views
JupyterHub Apache Proxy Reverse not working JupyterHub help-wanted	2	902	May 3, 2021
Debugging Jupyterhub Authentication JupyterHub jupyterlab , jupyterhub , help-wanted	2	722	August 17, 2022
Jupyterhub does not start kernels neither terminals JupyterHub help-wanted	2	167	January 12, 2024
Jupyter notebook via jupyterhub not working on public IP address JupyterHub jupyterlab	3	666	March 2, 2022
Stuck with redirect Loop when spawning with Kubespawner JupyterHub	3	123	March 14, 2024

Cant connect to kernel

Related Topics