Run individual cells through queueing system/scheduler

Special Topics
1

Does anyone have ideas on how to set up a system, where interactive notebook sessions would be run through a queueing system at the individual cell level?

The scenario is, that some typical commands/cells use plenty of memory for a short while. So when many users are using the system at the same time, there is a high probability for running out of memory, if everything is just executed instantly in an interactive session. By having the execution of a cell wait for resources, this would be avoided.

It will not work to schedule the whole notebook through a queue system, as most of the time, the notebook is idle. It is only when running the cells that the start of the execution should wait for resources to become available (if necessary).

Any and all ideas on how to implement this is welcome and highly appreciated!

2

Jupyter kernels have to maintain their state between cells, e.g. If you generate a Python object in one cell, then use it in another, it has to remain in memory in the kernel (Python interpreter).

If you can take care of saving and reloading variables on disk between each cell you could perhaps split each cell into it’s own notebook, and manage them with something like Welcome to papermill — papermill 2.3.1 documentation ?

Alternatively you might be able to write you own kernel that takes care of saving, loading, and transferring state between other kernels, see for example SoS Notebook: one notebook, multiple languages | vatlab but that’s going to be a lot of work.