On kubernetes with a strict memory limit, it’s common for a user’s notebook kernel to get shut down or restarted because it tries to exceed the allocated resources. Most of the time this results in a very puzzled user - cells are stuck in waiting forever mode, until the user realizes that it was actually the kernel that died (you need to see the upper-right of the jupyterlab window that says “no kernel”). Sometimes the user might get a pop up that says “kernel restarting” or similar - I guess this depends on the JupyterLab version.
Question: how do others handle the UX around this? Is there some way to signal to the user “hey, you are using 80% of your allocated memory, be careful!”) Could there be a sidecar container monitoring the resource consumption and it would fire off a notification to the user via an API call? Is there any mechanism that exists already that could be hacked together to achieve something like this?
Apologies if this should go in a more specific category… seems to sort of span various parts of the discourse board hence it landed in Q&A.