I use the batchspawner (GitHub - jupyterhub/batchspawner: Custom Spawner for Jupyterhub to start servers in batch scheduled systems)with slurm and want to use the preemption function.
If a job is now suspend because a higher prioritized job has been executed, I have problems resuming this job.
When the JupyterHub GUI of the stopped job is updated, you will be redirected to the start page to select a new profile.
As a result, the stopped job cannot be executed again, but a new job has to be started.
When resources are released again, the paused job is started again, but cannot be accessed from the GUI.
Is there a way to resume the old job without having to start a new one?
Am I correct in interpreting your situation as follows?
- Slurm automatically resumes the job (user server)
- JupyterHub is unaware of the resumed job
It seems like we’d need an api option to join externally spawned servers to the hub to do what you want. I’m not sure whether this currently exists, but if not it would be good to add. (It would be especially useful on HPC systems where you can’t guarantee that resources will be available to spawn a user’s job in a reasonable/short amount of time.)
I think this is probably a feature request for BatchSpawner to handle jobs in a suspended state. I don’t believe it exists yet. “resume” would usually mean checking for state in .start() and resuming if it is suspended, instead of starting anew. DockerSpawner has similar functionality for supporting resuming stopped containers, which may be useful as inspiration.