Better user experience when jupyter kernel dies/restarts

General
1

On kubernetes with a strict memory limit, it’s common for a user’s notebook kernel to get shut down or restarted because it tries to exceed the allocated resources. Most of the time this results in a very puzzled user - cells are stuck in waiting forever mode, until the user realizes that it was actually the kernel that died (you need to see the upper-right of the jupyterlab window that says “no kernel”). Sometimes the user might get a pop up that says “kernel restarting” or similar - I guess this depends on the JupyterLab version.

Question: how do others handle the UX around this? Is there some way to signal to the user “hey, you are using 80% of your allocated memory, be careful!”) Could there be a sidecar container monitoring the resource consumption and it would fire off a notification to the user via an API call? Is there any mechanism that exists already that could be hacked together to achieve something like this?

Apologies if this should go in a more specific category… seems to sort of span various parts of the discourse board hence it landed in Q&A.

1 Like
2

There is a resource usage monitor extension, but it is for the jupyter server, so if the kernels are remote it won’t do anything for you. However, perhaps monitoring resources from the jupyter server is enough for common cases?

1 Like
3

Hi @jasongrout thanks for the suggestion - the setup is quite simplistic and the kernels are running in the same pod with the server, so this would provide enough info (last time I looked it wasn’t really supported in JupyterLab but it seems to be now so :tada:!) Still, the kernel might be terminated externally for other reasons (disk quota etc.) and telling the user something would be nice.

4

JupyterLab should tell the users these days that the kernel was restarted, though JupyterLab has no idea why so it can’t give users a reason.

1 Like