Huge latency when connecting to remote Lab server

JupyterLab
,
1

I am running a Jupyter Lab server on our HPC cluster. I create an SHH tunnel (over our university VPN) to it in order to work using notebooks. In this setup, whenever I open the Jupyter interface, I get a huge increase in latency on my laptop (by pinging google.com) - from ~50-100 ms to many seconds. Even if I am not doing anything on it. Here is a sample of ping trace when I opened Jupyter Lab:

64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13044 ttl=112 time=56.3 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13045 ttl=112 time=65.2 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13046 ttl=112 time=64.3 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13047 ttl=112 time=430 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13048 ttl=112 time=451 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13049 ttl=112 time=506 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13050 ttl=112 time=555 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13051 ttl=112 time=650 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13052 ttl=112 time=869 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13053 ttl=112 time=1197 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13054 ttl=112 time=3195 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13055 ttl=112 time=3259 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13056 ttl=112 time=2453 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13057 ttl=112 time=1429 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13058 ttl=112 time=4829 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13059 ttl=112 time=3828 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13060 ttl=112 time=3788 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13061 ttl=112 time=2764 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13062 ttl=112 time=1746 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13063 ttl=112 time=5620 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13064 ttl=112 time=6595 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13065 ttl=112 time=5913 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13066 ttl=112 time=6412 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13067 ttl=112 time=6304 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13068 ttl=112 time=8935 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13069 ttl=112 time=8397 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13070 ttl=112 time=9390 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13071 ttl=112 time=9675 ms
64 bytes from lhr48s28-in-f14.1e100.net (172.217.16.238): icmp_seq=13072 ttl=112 time=10865 ms

This also affects other devices on the same netowork, e.g. my andorid phone often even decides to switch to the mobile network instead of using WiFi.

This doesn’t happen if I use a single Jupyter Notebook instead of Jupyter Lab.

Has this ever been reported or discussed? What could be causing this? Is there any other info I can provide to help narrow it down?

1 Like
2

Do you have any extensions installed? What version of JupyterLab are you running?

3

Maybe looking at the network tab of the developer tools in your browser could revel what is going on with the network? In addition to simple HTTP requests a lot is being sent through websockets, this is sometimes more difficult to get but as useful if not more.

4

I have Lab 3.0.14. I have just a couple extensions installed, judging from the Extension Manager, but I just disabled them all with no improvement.

5

Could you help me, what I should be looking for in the console? I have no experience with this. Here is what I see there.

image
image986×769 169 KB

6

Small correction: I think occasionally using just a notebook instead of a Lab, also increases ping to a few seconds, but only for a short time (less than a minute), and I can’t really be sure it’s caused by jupyter and not something else.