Creating or opening a notebook remotely hangs with kernel freeze

General
1

Hello Jupyter community,

I have recently experienced issues with remote access to jupyter notebooks. I can launch the jupyter kernel on the server, then connect to it locally by ssh port forwarding. However when I try to open or create any notebooks, the browser hangs and the kernel freezes. The issue started when the remote server updated from Debian 8 to 9 and finally 10.

Steps to reproduce:
Launch the jupyter-lab kernel
jupyter-lab --no-browser --port=8080
SSH port forward from a new console
ssh -L 8080:localhost:8080 user@[...].ac.uk
In a browser, visit the jupyter-lab custom URL given by the kernel
http://localhost:8080/?token=[...]
Create a new jupyter notebook
File -> New -> Notebook
Expected outcome: notebook creation hangs. Kernel freezes.

When running jupyter notebook instead of jupyter-lab:
→ Creating new text files works.
→ Creating new notebooks fails (hangs with no error message).
→ Opening existing notebooks fails with the following error:
Loading [MathJax]/extensions/Safe.js] File failed to load: http://127.0.0.1:8080/static/components/MathJax/extensions/Safe.js

I tried running jupyter notebook --debug mode, but no extra messages appeared at the point of error.
I have also tried uninstalling and reinstalling conda to the latest version. But the issue persists. The IT team that maintain the server have confirmed the issue is reproducible but have not been able to troubleshoot it.

EDIT 2020-11-06:
I tried to (re)start kernel in jupyter-lab browser, monitoring both jupyter-lab --debug and Chrome javascript console output.

I run jupyter-lab --no-browser --port=8080 --debug on remote.
Then from browser, File>New>Console. The browser hangs at “Kernel starting” and CLI content is as follows:

[D 11:20:54.796 LabApp] Starting kernel: [‘/home/USSR/jb2094/miniconda3/bin/python’, ‘-m’, ‘ipykernel_launcher’, ‘-f’, ‘/home/USSR/jb2094/.local/share/jupyter/runtime/kernel-2dfd23c1-2270-4ae4-aec7-af4880d88da4.json’]
[D 11:20:54.803 LabApp] Connecting to: tcp://127.0.0.1:59369
[D 11:20:54.818 LabApp] Connecting to: tcp://127.0.0.1:59191
[I 11:20:54.820 LabApp] Kernel started: 2dfd23c1-2270-4ae4-aec7-af4880d88da4, name: python3
[D 11:20:54.820 LabApp] Kernel args: {‘kernel_name’: ‘python3’, ‘cwd’: ‘/home/USSR/jb2094/py/sc-cleanpy/scripts’}
[D 11:20:54.822 LabApp] 201 POST /api/sessions?1604661654681 (::1) 124.72ms
[D 11:20:54.898 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:54.900 LabApp] 200 GET /api/sessions?1604661654879 (::1) 2.27ms
[D 11:20:54.985 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:54.986 LabApp] 200 PATCH /api/sessions/af08e3c3-6aa3-437b-8b7f-823010429f64?1604661654968 (::1) 1.92ms
[D 11:20:54.997 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:54.997 LabApp] 200 GET /api/kernels?1604661654977 (::1) 1.06ms
[D 11:20:55.011 LabApp] Initializing websocket connection /api/kernels/2dfd23c1-2270-4ae4-aec7-af4880d88da4/channels
[D 11:20:55.013 LabApp] Requesting kernel info from 2dfd23c1-2270-4ae4-aec7-af4880d88da4
[D 11:20:55.013 LabApp] Connecting to: tcp://127.0.0.1:40131
[D 11:20:55.050 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:55.051 LabApp] 200 PATCH /api/sessions/af08e3c3-6aa3-437b-8b7f-823010429f64?1604661655029 (::1) 1.78ms
[D 11:20:55.091 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:55.092 LabApp] 200 GET /api/sessions?1604661655045 (::1) 1.20ms
[D 11:20:58.427 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:58.428 LabApp] 200 GET /api/terminals?1604661658405 (::1) 1.34ms
[D 11:20:59.673 LabApp] Accepting token-authenticated connection from ::1
[D 11:20:59.685 LabApp] 200 GET /metrics?1604661659653 (::1) 12.84ms
[D 11:21:02.481 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:02.483 LabApp] 200 GET /api/contents/py/sc-cleanpy/scripts?content=1&1604661662456 (::1) 2.70ms
[D 11:21:05.079 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:05.079 LabApp] 200 GET /api/kernels?1604661665057 (::1) 1.06ms
[D 11:21:05.210 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:05.211 LabApp] 200 GET /api/sessions?1604661665189 (::1) 1.18ms
[D 11:21:08.496 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:08.496 LabApp] 200 GET /api/terminals?1604661668474 (::1) 0.91ms
[D 11:21:10.396 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:10.398 LabApp] Found kernel python3 in /home/USSR/jb2094/miniconda3/share/jupyter/kernels
[D 11:21:10.399 LabApp] 200 GET /api/kernelspecs?1604661670376 (::1) 3.51ms
[D 11:21:12.578 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:12.580 LabApp] 200 GET /api/contents/py/sc-cleanpy/scripts?content=1&1604661672559 (::1) 2.47ms
[D 11:21:15.142 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:15.143 LabApp] 200 GET /api/kernels?1604661675122 (::1) 1.04ms
[D 11:21:15.274 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:15.275 LabApp] 200 GET /api/sessions?1604661675253 (::1) 1.16ms
[D 11:21:18.559 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:18.559 LabApp] 200 GET /api/terminals?1604661678541 (::1) 0.89ms
[D 11:21:22.668 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:22.671 LabApp] 200 GET /api/contents/py/sc-cleanpy/scripts?content=1&1604661682654 (::1) 3.28ms
[D 11:21:25.208 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:25.209 LabApp] 200 GET /api/kernels?1604661685186 (::1) 1.02ms
[D 11:21:25.336 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:25.337 LabApp] 200 GET /api/sessions?1604661685317 (::1) 1.16ms
[D 11:21:32.768 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:32.770 LabApp] 200 GET /api/contents/py/sc-cleanpy/scripts?content=1&1604661692745 (::1) 2.57ms
[D 11:21:35.273 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:35.274 LabApp] 200 GET /api/kernels?1604661695251 (::1) 1.03ms
[D 11:21:35.401 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:35.402 LabApp] 200 GET /api/sessions?1604661695382 (::1) 1.14ms
[D 11:21:35.608 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:35.619 LabApp] 200 GET /metrics?1604661695590 (::1) 11.91ms
[D 11:21:39.610 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:39.611 LabApp] 200 GET /api/terminals?1604661699589 (::1) 0.96ms
[D 11:21:39.967 LabApp] Accepting token-authenticated connection from ::1
[D 11:21:44.975 LabApp] Kernel is taking too long to finish, killing

Then, Kernel>Restart Kernel.

jupyter-lab --debug output:

followed by (and repeating…)

Chrome Javascript console output:

Any help appreciated,
James.

2

Were you able to make any additional progress? I’m having a similar issue that seems to have come out of nowhere.

3

Hi Adam,

No progress yet. Will keep you posted if I have any updates.

4

The issue was fixed by our IT team. Here is the message that I got from them:

basically in the /etc/exports file there were two lines, the config file originally was created on Debian 8, when upgraded to 10 I didn’t touch the file for obvious reasons so it looked like this:
/export 192.168.1.0/24 (rw,fsid=0,insecure,no_subtree_check,async)
/export/homedirs 192.168.1.0/24(rw,nohide,sync,insecure,crossmnt,root_squash,no_subtree_check,fsid=1)

/export/data 192.168.1.0/24(rw,nohide,sync,insecure,crossmnt,root_squash,no_subtree_check,fsid=1)

if I comment out the first line /export, people can login just fine(ssh or gui), but nobody can map /export/data using autofs package. so it will have to be mount manually
If I leave the first line, people can login via ssh just fine and /data can be mapped with autofs, but nobody can login via any GUI, stock ubuntu gui (GNOME or any others), jupyter gui login etc.

1 Like