I have a JupyterHub running on Google Cloud Platform (setup following the z2jh tutorial).
It’s been working very nicely for a long time, but recently I (and other users) are finding that the notebook/browser tab is regularly losing its connection to the remote server (e.g. every few minutes).
The kernel is still alive and code continues to run, but notebooks stop displaying output in their cells making it impossible to see what’s happening. Refreshing the browser tab brings JupyterLab back to life, but notebooks that have lost their connections will not show any output again until the kernel is restarted and the code run once more.
It’s now happening so regularly that I’ve had to modify some of my notebooks to write progress and outputs directly to files (rather than displaying them inline in the notebook). This loses many of the advantages of the notebook interface, but at least I can see that my code is running after I’ve refreshed the browser tab.
I’m not sure how to start debugging this. Does anyone have any suggestions, please? JupyterLab itself is working fine and I haven’t noticed the problem when running the same container locally via Docker, so presumably it’s something to do with the connection between my browser (Chrome) and my notebook server on Google Cloud/Kubernetes.
This sounds like keepalive in a proxy somewhere. One thing to check is the logs of the notebook container and the javascript console in the browser to see if there’s an indication of connections being lost.
In the (Chrome) browser console, I see that connections are being lost. There are lots of errors like:
default.jss:73 WebSocket connection to 'wss://myjupyterhuburl/user/myusername/api/kernels/de7b428b-1bc6-47fd-a72c-69c3b760ca62/channels?session_id=dd539f94-0625-4f53-911c-d624bf0f2c75' failed:
_createSocket @ default.jss:73
and also some like this (but not so many):
default.jss:517 Uncaught (in promise) Error: Kernel connection disconnected
at b.t (default.jss:517:31)
at m (index.es6.jss:461:18)
at Object.l [as emit] (index.es6.jss:416:17)
at e.emit (index.es6.jss:99:17)
at b._updateConnectionStatus (default.jss:1144:39)
at b._reconnect (default.jss:1250:18)
at WebSocket._onWSClose (default.jss:128:22)
Similarly, in the notebook container logs I see errors like
So it appears to be a problem with WebSockets, which unfortunately are a complete mystery to me! The issue appeared quite recently, so presumably either related to a browser update, some update I’ve made to the Hub, or my organisation’s IT Department fiddling with our network/domain.
Any further tips gratefully received, although I appreciate this is all very vague and could presumably be caused by all kinds of things. I guess some reading on WebSockets is in order…
I still haven’t found the cause of this issue, but the problem seems to go away if users sign in to the Hub from an “Incognito” or “Private browsing” session.
I have no idea why this should make a difference, but it’s a welcome temporary fix while I dig deeper.
This often points to a caching or similar problem. Can you try clearing the JupyterLab state (ViewActivate Control PaletteReset Application State), and/or clearing your cookies and local storage (Forget about this site in Firefox), and cache, if that’s not too disruptive?
