JupyterHub network failed to start

I’m in a country where access to the US network is blocked, but I can access the US network with using a VPN.

I have a JupyterHub server. When I opened the VPN, the single-user jupyterlab can start, but when I closed the VPN, the single-user jupyterlab failed to start. I guess on the start of running a jupyterlab, the server need to get some data from US network, as the information below show:

19:21:18.284 [ConfigProxy] error: 503 GET /user/adfac/static/favicons/favicon.ico socket hang up
19:21:36.951 [ConfigProxy] error: 503 GET /hub/static/favicon.ico socket hang up

But I don’t need to get these data from the Internet, so how can I run the single-user jupyterlab without accessing the US network?

If you join and leave a VPN your network config changes. This can break any existing network connections, especially running servers even if they’re local. I think you best option is to download and save the data, disconnect from your VPN, and restart your server.

1 Like

Thanks for your help. But how can I download the data and where should I save the data? The links above ( /hub/static/favicon.ico) are not like web links.

I guess there maybe a process like this when I start a single-user jupyterlab:
The server is ready to start a jupyterlab, it needs to get some icons located locally first.If the server cannot find the icons, it will download the icons from the internet. If it cannot download the icons, the jupyterlab will not start.
So if I can manually download the icons, and put them in the right place, the server would work fine. But how can I download the icons, and where should I put them? Or is there a configurable setting that I can use to let the server completely ignore the icons?

I think I misunderstood what you’re asking. Is the problem that JupyterHub is running on your local network, but you need the VPN to access resources outside your network, from your local laptop?

If so that’s something you should configure in your VPN. Assuming you’ve got the right privileges you can setup different routes, so some traffic is directed over the VPN, and some is routed to your local network.

If I’m still misunderstanding you please could you explain your setup in more detail?

1 Like

I unplugged the network cable so that my computer can only access the local network(like 127.0.0.1:8888), not the public network (like www.google.com). Then I started the jupyterhub server, but the single-user jupyterlab failed to run the first cell. So I guess that in order to start a jupyterlab, I need to be able to access the public network. Am I right?

AFAIK, you do not need any internet connectivity to use jupyter-lab or jupyter-notebook. If you look into browser console, all the files will be served from the local server you started including static assets like imgs, favicons, etc. The same goes with JupyterHub.

Could you share your Jupyterhub, single user server logs and your browser logs to understand why it wouldnt work for you? I did a rapid test on my laptop by disconnecting WiFi and started a JupyterHub server and I could use notebook.

2 Likes

No, not in typical circumstances, at least. My guess is this is all related to the VPN configuration, which is introducing a requirement to connect to the internet, or changing routes when you enable or disable it. It could also be potentially related to local DNS, which usually resolves instantly, but can timeout instead if local names are not configured quite right. These names may change when your VPN switches on and off, too, depending on how that’s set up.

If JupyterLab loads but the execution never happens, this would likely be in the websocket. There will likely be a message in the javascript console, singleuser server log or both about the failure.

1 Like

Thanks for your help.
I tried the same process in another computer, and without internet connection, it also works fine. So I guess there must be some complicated internet configuration in my computer that is causing the problem above.
This is the browser logs when the cell execution never happens:

Thanks for your help.
I checked the Javascript console logs in the browser when the JuptyterLab loads but the execution never happens. And this is the output:


It shows that the Firefox can’t make the connection to the JupyterLab kernel. But I can’t figure out what is causing this connection failure.

There should also be logs from the proxy and/or single-user server, if you could include those. The thing to track down is where the communication breakdown is, because it could be:

  • browser->vpn
  • browser->jupyterhub proxy
  • jupyterhub proxy → singleuser-server

VPNs shouldn’t ever touch localhost traffic, but they sometimes do and ~always make a mess when they do. Some VPNs still do not support websocket traffic. The fact that regular requests work and websocket requests do not suggests to me that it is the VPN capturing traffic that it should not be, and then not supporting that traffic.

Some cases to test:

  • VPN off when JupyterHub starts, stays off
  • VPN on when JupyterHub starts, stays on
  • VPN on when JupyterHub starts, off when starting jupyter server
  • VPN off when JupyterHub starts, on when starting jupyter server

Knowing which of these work and which don’t may help.

2 Likes

This is the logs from single-user server:

[W 2024-02-21 19:24:10.046 ServerApp] jupyter_server_terminals | error adding extension (enabled: True): The module 'jupyter_server_terminals' could not be found (No module named 'terminado'). Are you sure the extension is installed?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyter_server/extension/manager.py", line 322, in add_extension
        extpkg = ExtensionPackage(name=extension_name, enabled=enabled)
      File "/usr/local/lib/python3.10/dist-packages/jupyter_server/extension/manager.py", line 186, in __init__
        self._load_metadata()
      File "/usr/local/lib/python3.10/dist-packages/jupyter_server/extension/manager.py", line 201, in _load_metadata
        raise ExtensionModuleNotFound(msg) from None
    jupyter_server.extension.utils.ExtensionModuleNotFound: The module 'jupyter_server_terminals' could not be found (No module named 'terminado'). Are you sure the extension is installed?
[W 2024-02-21 19:24:10.362 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-02-21 19:24:10.362 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-02-21 19:24:10.362 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2024-02-21 19:24:10.362 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /user/baiguo/lab/tree/example.ipynb
[I 2024-02-21 19:24:10.365 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-02-21 19:24:10.366 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-02-21 19:24:10.368 ServerApp] jupyterlab | extension was successfully linked.
[W 2024-02-21 19:24:10.369 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-02-21 19:24:10.370 ServerApp] notebook | extension was successfully linked.
[I 2024-02-21 19:24:10.371 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2024-02-21 19:24:10.498 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-02-21 19:24:10.507 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-02-21 19:24:10.508 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-02-21 19:24:10.512 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-02-21 19:24:10.512 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-02-21 19:24:10.519 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/dist-packages/jupyterlab
[I 2024-02-21 19:24:10.519 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-02-21 19:24:10.520 LabApp] Extension Manager is 'pypi'.
[I 2024-02-21 19:24:10.553 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-02-21 19:24:10.555 ServerApp] notebook | extension was successfully loaded.
[I 2024-02-21 19:24:10.555 ServerApp] Serving notebooks from local directory: /root/work/notebook
[I 2024-02-21 19:24:10.556 ServerApp] Jupyter Server 2.12.5 is running at:
[I 2024-02-21 19:24:10.556 ServerApp] http://2df03ecbcb38:8888/user/baiguo/lab/tree/example.ipynb?token=...
[I 2024-02-21 19:24:10.556 ServerApp]     http://127.0.0.1:8888/user/baiguo/lab/tree/example.ipynb?token=...
[I 2024-02-21 19:24:10.556 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-02-21 19:24:10.702 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-02-21 19:24:10.892 ServerApp] 302 GET /user/baiguo/ -> /user/baiguo/lab/tree/example.ipynb? (@172.17.0.1) 0.40ms
[I 2024-02-21 19:24:11.071 ServerApp] 302 GET /user/baiguo/ -> /user/baiguo/lab/tree/example.ipynb? (@127.0.0.1) 0.73ms
[I 2024-02-21 19:24:11.160 ServerApp] 302 GET /user/baiguo/lab/tree/example.ipynb? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-baiguo&redirect_uri=%2Fuser%2Fbaiguo%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 1.40ms
[I 2024-02-21 19:24:11.399 ServerApp] Logged-in user {'kind': 'user', 'name': 'baiguo', 'admin': False, 'groups': [], 'session_id': '7c8fd2d277454929bb6ac622915dbaa8', 'scopes': ['access:servers!server=baiguo/', 'read:users:groups!user=baiguo', 'read:users:name!user=baiguo']}
[I 2024-02-21 19:24:11.400 ServerApp] 302 GET /user/baiguo/oauth_callback?code=[secret]&state=[secret] -> /user/baiguo/lab/tree/example.ipynb? (@127.0.0.1) 40.43ms
[I 2024-02-21 19:24:11.500 ServerApp] 200 GET /user/baiguo/lab/tree/example.ipynb? (baiguo@127.0.0.1) 9.54ms
[W 2024-02-21 19:24:22.763 ServerApp] Native kernel (python3) is not available
[I 2024-02-21 19:24:22.769 ServerApp] 200 GET /user/baiguo/api/kernelspecs?1708514662716 (baiguo@127.0.0.1) 8.13ms
[I 2024-02-21 19:24:22.770 ServerApp] 200 GET /user/baiguo/api/me?1708514662717 (baiguo@127.0.0.1) 0.48ms
[I 2024-02-21 19:24:22.807 ServerApp] 200 GET /user/baiguo/lab/api/settings?1708514662720 (baiguo@127.0.0.1) 37.33ms
[I 2024-02-21 19:24:23.337 ServerApp] 200 GET /user/baiguo/lab/api/translations/default?1708514663272 (baiguo@127.0.0.1) 7.94ms
[I 2024-02-21 19:24:23.432 ServerApp] 200 GET /user/baiguo/api/config/jupyterlabapputilsextensionannouncements?1708514663389 (baiguo@127.0.0.1) 1.53ms
[I 2024-02-21 19:24:23.486 ServerApp] 200 GET /user/baiguo/lab/api/translations?1708514663431 (baiguo@127.0.0.1) 11.79ms
[I 2024-02-21 19:24:23.792 ServerApp] 200 GET /user/baiguo/api/kernels?1708514663734 (baiguo@127.0.0.1) 2.64ms
[I 2024-02-21 19:24:23.793 ServerApp] 200 GET /user/baiguo/api/sessions?1708514663734 (baiguo@127.0.0.1) 0.74ms
[I 2024-02-21 19:24:23.843 ServerApp] 101 GET /user/baiguo/api/events/subscribe?token=[secret] (baiguo@127.0.0.1) 1.27ms
[W 2024-02-21 19:24:24.790 ServerApp] Native kernel (python3) is not available
[I 2024-02-21 19:24:24.791 ServerApp] 200 GET /user/baiguo/api/kernelspecs?1708514664744 (baiguo@127.0.0.1) 1.94ms
[I 2024-02-21 19:24:24.795 ServerApp] 200 GET /user/baiguo/api/me?1708514664746 (baiguo@127.0.0.1) 0.67ms
[I 2024-02-21 19:24:25.820 ServerApp] 200 GET /user/baiguo/lsp/status?1708514665777 (baiguo@127.0.0.1) 1.67ms
[I 2024-02-21 19:24:25.838 ServerApp] 200 GET /user/baiguo/api/contents?content=1&hash=0&1708514665785 (baiguo@127.0.0.1) 7.46ms
[W 2024-02-21 19:24:25.843 LabApp] Could not determine jupyterlab build status without nodejs
[I 2024-02-21 19:24:25.844 ServerApp] 200 GET /user/baiguo/lab/api/build?1708514665753 (baiguo@127.0.0.1) 45.99ms
[I 2024-02-21 19:24:25.952 ServerApp] 200 GET /user/baiguo/lab/api/workspaces/default?1708514665909 (baiguo@127.0.0.1) 0.90ms
[I 2024-02-21 19:24:26.056 ServerApp] 200 GET /user/baiguo/lab/api/settings?ids_only=true&1708514666010 (baiguo@127.0.0.1) 3.85ms
[I 2024-02-21 19:24:26.882 ServerApp] 200 GET /user/baiguo/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1708514666829 (baiguo@127.0.0.1) 4.98ms
[I 2024-02-21 19:24:26.886 ServerApp] 200 GET /user/baiguo/lab/api/settings/@jupyterlab/notebook-extension:panel?1708514666831 (baiguo@127.0.0.1) 3.05ms
[I 2024-02-21 19:24:27.943 ServerApp] 200 GET /user/baiguo/api/contents?content=1&hash=0&1708514667887 (baiguo@127.0.0.1) 10.10ms
[I 2024-02-21 19:24:28.965 ServerApp] 204 PUT /user/baiguo/lab/api/workspaces/default?1708514668911 (baiguo@127.0.0.1) 2.00ms
[I 2024-02-21 19:24:29.065 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb?content=0&hash=0&1708514669016 (baiguo@127.0.0.1) 3.50ms
[I 2024-02-21 19:24:29.163 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb?content=0&hash=0&1708514669109 (baiguo@127.0.0.1) 3.09ms
[I 2024-02-21 19:24:29.270 ServerApp] 200 GET /user/baiguo/api/contents?content=1&hash=0&1708514669208 (baiguo@127.0.0.1) 10.43ms
[I 2024-02-21 19:24:29.363 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb?content=0&hash=0&1708514669313 (baiguo@127.0.0.1) 2.98ms
[I 2024-02-21 19:24:29.585 ServerApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[W 2024-02-21 19:24:29.611 ServerApp] Notebook example.ipynb is not trusted
[I 2024-02-21 19:24:29.613 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb?type=notebook&content=1&hash=1&1708514669426 (baiguo@127.0.0.1) 145.18ms
[I 2024-02-21 19:24:29.709 ServerApp] 200 GET /user/baiguo/api/nbconvert?1708514669427 (baiguo@127.0.0.1) 239.29ms
[I 2024-02-21 19:24:30.006 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb/checkpoints?1708514669959 (baiguo@127.0.0.1) 1.33ms
[I 2024-02-21 19:24:30.007 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb/checkpoints?1708514669959 (baiguo@127.0.0.1) 0.67ms
[I 2024-02-21 19:24:32.362 ServerApp] 200 GET /user/baiguo/api/contents/example.ipynb/checkpoints?1708514669963 (baiguo@127.0.0.1) 0.76ms
[I 2024-02-21 19:24:32.379 ServerApp] 201 POST /user/baiguo/api/contents/example.ipynb/checkpoints?1708514670237 (baiguo@127.0.0.1) 2.03ms
[I 2024-02-21 19:24:33.659 ServerApp] Kernel started: ac4b184d-e862-4093-b973-1b279a391a96
[I 2024-02-21 19:24:33.660 ServerApp] 201 POST /user/baiguo/api/sessions?1708514673280 (baiguo@127.0.0.1) 17.17ms
[I 2024-02-21 19:24:33.740 ServerApp] 200 GET /user/baiguo/api/sessions?1708514673698 (baiguo@127.0.0.1) 0.71ms
[I 2024-02-21 19:24:33.781 ServerApp] 204 PUT /user/baiguo/lab/api/workspaces/default?1708514673716 (baiguo@127.0.0.1) 0.82ms
[I 2024-02-21 19:24:33.841 ServerApp] 200 PATCH /user/baiguo/api/sessions/980719e0-d946-478b-b281-16468271818f?1708514673794 (baiguo@127.0.0.1) 0.99ms
[I 2024-02-21 19:24:33.842 ServerApp] 200 GET /user/baiguo/api/kernels?1708514673794 (baiguo@127.0.0.1) 0.61ms
[I 2024-02-21 19:24:33.923 ServerApp] 200 PATCH /user/baiguo/api/sessions/980719e0-d946-478b-b281-16468271818f?1708514673881 (baiguo@127.0.0.1) 0.95ms
[I 2024-02-21 19:24:33.925 ServerApp] 200 GET /user/baiguo/api/sessions?1708514673882 (baiguo@127.0.0.1) 0.53ms
[W 2024-02-21 19:24:34.004 ServerApp] Native kernel (python3) is not available
[I 2024-02-21 19:24:34.004 ServerApp] 200 GET /user/baiguo/api/kernelspecs?1708514673963 (baiguo@127.0.0.1) 1.24ms
[I 2024-02-21 19:24:35.352 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514675300 (baiguo@127.0.0.1) 8.63ms
[I 2024-02-21 19:24:35.958 ServerApp] 204 PUT /user/baiguo/lab/api/workspaces/default?1708514675908 (baiguo@127.0.0.1) 2.51ms
[W 2024-02-21 19:24:36.418 ServerApp] Notebook rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb is not trusted
[I 2024-02-21 19:24:36.419 ServerApp] 200 GET /user/baiguo/api/contents/rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb?type=notebook&content=1&hash=1&1708514676346 (baiguo@127.0.0.1) 17.79ms
[I 2024-02-21 19:24:36.424 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514676366 (baiguo@127.0.0.1) 7.81ms
[I 2024-02-21 19:24:36.521 ServerApp] 200 GET /user/baiguo/api/contents/rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb/checkpoints?1708514676479 (baiguo@127.0.0.1) 1.23ms
[I 2024-02-21 19:24:36.522 ServerApp] 200 GET /user/baiguo/api/contents/rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb/checkpoints?1708514676479 (baiguo@127.0.0.1) 0.64ms
[I 2024-02-21 19:24:36.531 ServerApp] 200 GET /user/baiguo/api/contents/rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb/checkpoints?1708514676490 (baiguo@127.0.0.1) 0.71ms
[I 2024-02-21 19:24:36.712 ServerApp] 201 POST /user/baiguo/api/contents/rust_base/Rust_Crash_Course_02_Functions_and_Control_Flow.ipynb/checkpoints?1708514676663 (baiguo@127.0.0.1) 1.83ms
[I 2024-02-21 19:24:36.815 ServerApp] 200 GET /user/baiguo/api/sessions?1708514676757 (baiguo@127.0.0.1) 0.70ms
[I 2024-02-21 19:24:36.924 ServerApp] 204 PUT /user/baiguo/lab/api/workspaces/default?1708514676877 (baiguo@127.0.0.1) 0.69ms
[I 2024-02-21 19:24:36.936 ServerApp] Kernel started: 548752bd-9665-426b-80aa-600c51c79c7d
[I 2024-02-21 19:24:36.937 ServerApp] 201 POST /user/baiguo/api/sessions?1708514676857 (baiguo@127.0.0.1) 18.08ms
[I 2024-02-21 19:24:37.025 ServerApp] 200 GET /user/baiguo/api/sessions?1708514676984 (baiguo@127.0.0.1) 0.71ms
[I 2024-02-21 19:24:37.110 ServerApp] 200 PATCH /user/baiguo/api/sessions/818b4754-ce30-424e-83e5-2387d3448062?1708514677069 (baiguo@127.0.0.1) 0.99ms
[I 2024-02-21 19:24:37.111 ServerApp] 200 GET /user/baiguo/api/kernels?1708514677069 (baiguo@127.0.0.1) 0.50ms
[I 2024-02-21 19:24:37.194 ServerApp] 200 PATCH /user/baiguo/api/sessions/818b4754-ce30-424e-83e5-2387d3448062?1708514677150 (baiguo@127.0.0.1) 0.91ms
[I 2024-02-21 19:24:37.195 ServerApp] 200 GET /user/baiguo/api/sessions?1708514677151 (baiguo@127.0.0.1) 0.50ms
[W 2024-02-21 19:24:37.275 ServerApp] Native kernel (python3) is not available
[I 2024-02-21 19:24:37.276 ServerApp] 200 GET /user/baiguo/api/kernelspecs?1708514677234 (baiguo@127.0.0.1) 1.25ms
[I 2024-02-21 19:24:37.810 ServerApp] 204 PUT /user/baiguo/lab/api/workspaces/default?1708514677766 (baiguo@127.0.0.1) 1.10ms
[I 2024-02-21 19:24:38.140 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514678091 (baiguo@127.0.0.1) 6.53ms
[I 2024-02-21 19:24:47.349 ServerApp] 200 GET /user/baiguo/api/kernels?1708514687254 (baiguo@127.0.0.1) 1.08ms
[I 2024-02-21 19:24:48.310 ServerApp] 200 GET /user/baiguo/api/sessions?1708514688258 (baiguo@127.0.0.1) 2.16ms
[I 2024-02-21 19:24:48.367 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514688259 (baiguo@127.0.0.1) 6.58ms
[I 2024-02-21 19:24:53.934 ServerApp] 200 GET /user/baiguo/api/kernels/ac4b184d-e862-4093-b973-1b279a391a96?1708514693801 (baiguo@127.0.0.1) 0.87ms
[I 2024-02-21 19:25:14.332 ServerApp] 200 GET /user/baiguo/api/kernels/ac4b184d-e862-4093-b973-1b279a391a96?1708514714230 (baiguo@127.0.0.1) 1.91ms
[I 2024-02-21 19:25:27.621 ServerApp] 200 GET /user/baiguo/api/kernels?1708514727501 (baiguo@127.0.0.1) 0.86ms
[I 2024-02-21 19:25:28.557 ServerApp] 200 GET /user/baiguo/api/sessions?1708514728510 (baiguo@127.0.0.1) 1.04ms
[I 2024-02-21 19:25:28.605 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514728511 (baiguo@127.0.0.1) 7.55ms
[W 2024-02-21 19:25:33.892 ServerApp] Timeout waiting for kernel_info reply from ac4b184d-e862-4093-b973-1b279a391a96
[I 2024-02-21 19:25:33.895 ServerApp] 101 GET /user/baiguo/api/kernels/ac4b184d-e862-4093-b973-1b279a391a96/channels?session_id=042b194b-6afc-4df4-85a0-44c07f3cf204 (baiguo@127.0.0.1) 60005.78ms
[I 2024-02-21 19:25:33.897 ServerApp] Connecting to kernel ac4b184d-e862-4093-b973-1b279a391a96.
[I 2024-02-21 19:25:35.287 ServerApp] 200 GET /user/baiguo/api/kernels/ac4b184d-e862-4093-b973-1b279a391a96?1708514735171 (baiguo@127.0.0.1) 0.78ms
[W 2024-02-21 19:25:38.420 ServerApp] Nudge: attempt 10 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[I 2024-02-21 19:25:38.448 ServerApp] 200 GET /user/baiguo/api/kernels?1708514738403 (baiguo@127.0.0.1) 0.67ms
[I 2024-02-21 19:25:39.562 ServerApp] 200 GET /user/baiguo/api/sessions?1708514739506 (baiguo@127.0.0.1) 2.17ms
[I 2024-02-21 19:25:39.609 ServerApp] 200 GET /user/baiguo/api/contents/rust_base?content=1&hash=0&1708514739508 (baiguo@127.0.0.1) 10.50ms
[W 2024-02-21 19:25:43.445 ServerApp] Nudge: attempt 20 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:25:48.471 ServerApp] Nudge: attempt 30 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:25:53.492 ServerApp] Nudge: attempt 40 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:25:54.303 ServerApp] Timeout waiting for kernel_info reply from ac4b184d-e862-4093-b973-1b279a391a96
[I 2024-02-21 19:25:54.306 ServerApp] 101 GET /user/baiguo/api/kernels/ac4b184d-e862-4093-b973-1b279a391a96/channels?session_id=f1b5975f-ebd3-48bb-a87c-b27ea9bc4719 (baiguo@127.0.0.1) 60004.58ms
[I 2024-02-21 19:25:54.307 ServerApp] Connecting to kernel ac4b184d-e862-4093-b973-1b279a391a96.
[I 2024-02-21 19:25:55.284 ServerApp] 200 GET /user/baiguo/api/kernels/548752bd-9665-426b-80aa-600c51c79c7d?1708514755185 (baiguo@127.0.0.1) 0.86ms
[W 2024-02-21 19:25:58.517 ServerApp] Nudge: attempt 50 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:25:58.857 ServerApp] Nudge: attempt 10 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:26:03.537 ServerApp] Nudge: attempt 60 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:26:03.879 ServerApp] Nudge: attempt 20 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[E 2024-02-21 19:26:03.897 ServerApp] Exception in callback <bound method WebSocketMixin.send_ping of <jupyter_server.services.kernels.websocket.KernelWebsocketHandler object at 0x7facd514a140>>
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 937, in _run
        val = self.callback()
      File "/usr/local/lib/python3.10/dist-packages/jupyter_server/base/websocket.py", line 123, in send_ping
        self.ping(b"")
      File "/usr/local/lib/python3.10/dist-packages/tornado/websocket.py", line 440, in ping
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError
[W 2024-02-21 19:26:08.563 ServerApp] Nudge: attempt 70 on kernel ac4b184d-e862-4093-b973-1b279a391a96
[W 2024-02-21 19:26:08.903 ServerApp] Nudge: attempt 30 on kernel ac4b184d-e862-4093-b973-1b279a391a96

And I tested the four cases you mentioned above, the case 2 and case 4 can finish the execution, but case 1 and case 3 can’t.

VPN off and stays off not working is very surprising. This suggests to me something very peculiar in the local network configuration that the VPN is bypassing which actually makes it work.

Is there a chance there is any other firewall software running on the machine?

1 Like

I turned off all firewalls both in the host mathine (Windows11) and in wsl2 (The Jupyterhub server was running in wsl2 and it starts a container when starting a single-user).

I’m afraid I don’t know what to do next. So far, the evidence points to an intermediary proxy layer between the browser and the jupyterhub proxy that is not implementing or not allowing websocket connections. Some folks have reported this to be run by firewalls or antivirus software.

It’s also possible it’s related to the WSL networking layer, but that would be pretty odd, since websockets operate at a much higher level.

1 Like