Cant connect to kernel

JupyterHub
1

Hello guys,
I am using Apache as reverseproxy.

<Location /jupyterhub>

RequestHeader unset Accept-Encoding
ProxyPass         http://jupyterhub:8000/jupyterhub
ProxyPassReverse  http://jupyterhub:8000/jupyterhub

RewriteEngine On
ProxyPreserveHost on

AuthType shibboleth
ShibRequestSetting requireSession 1
Require valid-user
ShibUseHeaders On
RequestHeader set X-Remote-User %{REMOTE_USER}s

This error occurs in my single user server

And This is my Jupyterhub_config.py

c.Spawner.cmd = “jupyterhub-singleuser”
c.Spawner.debug = True
c.ConfigurableHTTPProxy.debug = True
c.DockerSpawmer.use_internal_ip = True
c.JupyterHub.spawner_class = DockerSpawner
c.DockerSpawner.image = “xxx/jupyterlab”

c.JupyterHub.hub_ip = “0.0.0.0”
c.JupyterHub.bind_url = ‘http://:8000/jupyterhub’

c.DockerSpawner.network_name = os.environ[“DOCKER_NETWORK_NAME”]
c.DockerSpawner.extra_host_config = { ‘network_mode’: os.environ[“DOCKER_NETWORK_NAME”] }
c.DockerSpawner.remove_containers = True
c.DockerSpawner.start_timeout = 120
c.JupyterHub.authenticator_class = ShibbolethAuthenticator

I can spawn and access Single user server, but i cant open a notebook and connect to kernel. The Eorror Code:

[W 2024-03-22 11:18:42.287 ServerApp] 400 GET /jupyterhub/user/yuqiang/api/events/subscribe?token=[secret] (yuqiang@::ffff:172.20.0.3) 2.13ms

[W 2024-03-22 11:18:49.038 ServerApp] 400 GET /jupyterhub/user/yuqiang/api/events/subscribe?token=[secret] (yuqiang@::ffff:172.20.0.3) 1.76ms

2

I found a not fancy solution in my case.
I manually redirect this failed request in Apache proxy like this:

        <Location "/jupyterhub/***[username]***/api/kernels">
            ProxyPass ws://localhost:8000/jupyterhub/***[username]***/api/kernels
            ProxyPassReverse ws://localhost:8000/jupyterhub/***[username]***/api/kernels
        </Location>

But it is really dumm, that, i have to add this code for all 30 users. (for every username)
Is there any better way to automatically redirect this request, like Regular expression?

3

Here is a reference Apache proxy config that should work with JupyterHub → Using a reverse proxy — JupyterHub documentation

I guess you are missing webscokets connection upgrade directives in your config

3 Likes
4

Hi,

Thanks. You are right! I just missed the websockets upgrade part!
I can now acess to my content. But It starts to restart my single user server all the time.
image

And in Logs i found this:

15:22:57.392 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.393 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.395 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.397 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang
15:22:57.399 [ConfigProxy] debug: Not recording activity for status 304 on /jupyterhub/user/yuqiang

1 Like
6

Another Problem:
And strange things happen, if I activate the Jupyter-collaboration extension.
I can normally open a notebook in my home directory (/home/joyvan)
But can not connect to Kernel and open a notebook in a folder (/home/joyvan/[some newfolder]).

image
image1570×756 18.2 KB

The Cause according my guess:
This Problem doesn´t exsit if i only use Jupyterhub internal Proxy (CHP). So I guess the problem is the compatibility of Jupyter-collaboration extension and Apache2?

Logs seem good:

[I 2024-03-26 16:26:47.448 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/contents/Untitled%20Folder?content=1&1711470407345 (yuqiang@172.22.0.4) 4.89ms
[I 2024-03-26 16:26:50.123 ServerApp] 200 GET /jupyterhub/user/yuqiang/api (@172.22.0.5) 0.78ms
[I 2024-03-26 16:26:52.162 YDocExtension] Processed 366 Y patches in one minute
[I 2024-03-26 16:26:52.162 YDocExtension] Connected Y users: 8
[I 2024-03-26 16:26:52.563 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/terminals?1711470412349 (yuqiang@172.22.0.4) 2.14ms
[I 2024-03-26 16:26:52.749 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/me?1711470412533 (yuqiang@172.22.0.4) 1.96ms
[I 2024-03-26 16:26:52.771 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/kernelspecs?1711470412535 (yuqiang@172.22.0.4) 2.58ms
[I 2024-03-26 16:26:55.327 ServerApp] 200 GET /jupyterhub/user/yuqiang/api (@172.22.0.5) 0.78ms

7

What do your browser console logs (https://help.zapier.com/hc/en-us/articles/8496199434765-View-and-save-your-browser-console-logs) and Apache logs show?

1 Like
8

There is Error in Browser Logs!:

image

image

image

There is nothing strange accrodding to Apache Logs:

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/terminals?1711471394311 HTTP/1.1” 200 823 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb” "

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/contents/Untitled%20Folder?content=1&1711471394574 HTTP/1.1” 200 1093 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb”

[26/Mar/2024:16:43:14 +0000] “GET /jupyterhub/user/yuqiang/api/kernels?1711471394421 HTTP/1.1” 200 3475 “https:///jupyterhub/user/yuqiang/lab/tree/RTC%3AUntitled%20Folder/Untitled.ipynb” “Mozilla/5.0 (iPhone; CPU iPhone OS 17_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1”

9

The problematic error Log is likely this: Browser fails to find the URL of directory (https://jupyterhub//user/yuqiang/api/collaboration/session/Untitled%20Folder%2FUntitled1.ipynb?1711541015975)

image

And again, this problem doesn´t exist if a disable the jupyter-collaboration extention.

10

A new version of Jupyterlab==4.1.5 also doesn´t help.

I found that the single user server and my browser are opening different URL. Single user server found the right url but My Browser doesn´t.

Browser says it can´t open the Note book because following URL doesn´t exist

/user/yuqiang/api/collaboration/session/Assignment%2FVisualisierung_CO2_Konzentration.ipynb 404 (Not Found)

Single user Server says that, it can successfully access to it:

[I 2024-03-28 15:41:38.678 YDocExtension] Processed 53 Y patches in one minute
[I 2024-03-28 15:41:38.678 YDocExtension] Connected Y users: 3
[I 2024-03-28 15:41:40.648 ServerApp] 200 GET /jupyterhub/user/yuqiang/api/contents/Assignment/Visualisierung_CO2_Konzentration.ipynb?type=notebook&content=0 (yuqiang@::ffff:172.31.0.3) 3.34ms

11

I found the problematic Request is sent by Apache Webserver. The Request sent by Apche seems to be blocked. Other Requests, which are sent by tornado, are no problem.

Any suggestions to let tornado send the request instead of Apache?

image
image845×513 40.9 KB

12

I’m not entirely sure what’s going on here, but if Apache is responding, this means that some routing rule in Apache is not forwarding the request to JupyterHub. That should all be in the apache config, since it means JupyterHub hasn’t had a chance to doesn’t get involved.

Since it’s a subset of URLs, I wonder if some other config is catching URLs with e.g. session in them.

Another possible option that I’m not sure about is a version mismatch in the jupyter-collaboration javascript and Python versions. Making sure both are up to date and restarting the server might help.

If you request a URL that definitely 404s and will be served by JupyterLab: /jupyterhub/user/yuqiang/api/nosuchthing/should/404

do you see it served by Apache or Tornado?

13

Hello @minrk ,

I am using python ==3.11.6 and jupyter-collaboration==2.0.11.

If i request URL eample.org/jupyterhub/user/yuqiang/api/nosuchthing/should/404, which is 404. It is served by Tornado.

14

Tornado will serve the following URL
/jupyterhub/user/udo.heuser/api/collaboration/session/Assignment
But not
/jupyterhub/user/udo.heuser/api/collaboration/session/Assignment%2F…
I got a Apache error with this URL
image

Where Assignment is a directory in my workspace

So I think the %2F or % is the problem, that makes Apache confused, and Tornado can´t recive the request.

15

Problem solved!
Add this to Apache confiiguration. If you want to use jupyter-collaboration with Apache as reverse proxy!

AllowEncodedSlashes On
1 Like