Hi all,
I am trying to configure Jupyter to free up resources so users don’t leave expensive machines running idle but we’re having an issue where notebooks are being culled when they have a very long running cell.
I’ve been trying to use the jupyter-idle-culler but it was still killing kernels which have long running cells.
After some digging it seems that the culler uses the “idle” status along with the “last_activity” field of the kernel, but when a single cell is executing for a long amount of time the kernel thinks it is idle and the last_activity field doesn’t get updated until it gets some output again.
I wrote a small script that would detect the long running notebook and try to hit the rest API to update the last_activity field but it seems /interrupt is the only one that would update that field, which I obviously can’t use. I also tried using the web socket api to execute pass hoping that would update the last_activity field but that call gets queued to run after the current cell finishes.
Our current workaround was just to disable the culler but that’s not viable long term.
Would greatly appreciate any input or similar situations, thanks