I recently collected a list of “flow control” / batch processing / etc types of infrastructure, in particular in the notebook ecosystem. Just sharing them below in case they are useful! Please feel free to suggest something new if I missed something!
- rstudio launcher
- elyra pipelines
- mlflow on jupyterhub
- prefectio
- paperboy
- notebooker
- volcano
- facebook submitit
- papermillhub (papermill on jupyterhub)
(disclaimer: I’m the author). Builds on top of papermill for developing multi-stage/batch pipelines.