Kill Slurm Job when browser tab closes

JupyterHub
1

Hello,

I’m deploying an HPC-enabled Jupyterhub which will use a SLURM spawner. I would like to communicate to SLURM to cancel the scheduler jobs when the browser tab is closed. Is it possible to send signals to the system on a browser-tab-close action? (sorry if that phrasing is confusing)

Any hints would be appreciated! Thanks!
Paul

2

The closest is probably the GitHub - jupyterhub/jupyterhub-idle-culler: JupyterHub service to cull idle servers and users which will automatically stop a user’s server if it detects it’s idle. Not the same as automatically stopping jobs when a user closes their browser, but potentially within a few minutes depending on your idle timeout. It’d still need some customisation to kill the extra jobs.

I’ve added the hpc tag to your post, so hopefully it’ll be seen by people who know a bit more about HPC.

1 Like
3

As @manics nicely summarized, this is what we do on our HPC platform. What you are suggesting is to fire a callback on tab close event that sends an API request to stop a server? In SLURM context, stopping a server is cancelling SLURM job if you are using batchspawner.

I guess this might be possible but I am not sure if it has any potential security issues. I am not a frontend expert. Also this can have undesired consequences as well if user closes the tab by error. JupyterHub Idle culler lets you configure a timeout for inactivity, and after this timeout, the server will be stopped (killing your SLURM job) even if the tab is open.

1 Like
4

I’ve already communicated the potential “accident” risks to the rest of our HPC team, and there are a few considerations here that I am personally not really OK with. Killing off your notebook when you close a browser tab is, in my option, not really a good solution. I might, for example, go eat lunch and close my browser. Killing off my Jupyter notebook which might have done some expensive calculations in the process. I could now go and tell users to save their intermediate results in some sensible file, but…na ja, not ideal.

In my view, there is no elegant solution here beyond clearly informing users how long their session will exist. Maybe there are other admins who can give me some stories here?

5

There is in general no distinction between closing a tab and reloading it, so I don’t think it makes a lot of sense to use that as a signal for much.

Culling has a lot of tuning parameters, especially so-called internal culling, which is a feature of jupyter-server, not JupyterHub. This has the option of separately deciding on multiple things:

  1. stop kernels one at a time, not just servers
  2. consider idle ‘connections’ (almost open tabs, but not precisely, because a tab could be open without a notebook being open in the tab) activity or not
  3. stop the server when it hasn’t had any activity in X time

Whenever a deployment has aggressive culling behavior (e.g. mybinder.org), we often get the opposite question from users: “how do I keep my server alive while it’s working on something?” So clearly communicating when things will be shutdown is perhaps the best one can do, and balance in your configuration:

  1. the cost of premature culling (lost data, lost in-process computation)
  2. the cost of idle servers (idle compute, wasted quota)

And perhaps even surface one or two options directly to users to set e.g. a max lifetime of their server (this requires some custom culling that the existing cullers don’t yet support, but is doable).

6

When you submit your slurm job you set the max wall time, so as long as you can customize that in slurmspawner you won’t need anything extra.