Hi all,
I’m a big fan of papermill and have been using it for a while! Two things that came up recently when talking to other papermill users was debugging and profiling, so I decided to tackle this and build some custom papermill engines to solve these issues.
debugging: when notebooks crash, the partially executed notebook is helpful, but I missed having post-mortem debugging. The new engine serializes the traceback so you can start a debugging session at any moment. This is especially useful when the notebooks are scheduled and/or executed remotely.
profiling: since papermill’s default engine uses nbclient to execute notebooks, it only sends messages to the kernel and captures responses, so monitoring resource usage isn’t possible. The new engine executes the notebook in the same process, allowing monitoring resources. (I was having some issues when implementing this, but lucky me, I found this project from fast.ai, which helped me figure things out!)
Feedback is much appreciated!