# Auto-save periodically during nbconvert

**URL:** <https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361>\
**Category:** nbconvert\
**Tags:** nbconvert\
**Created:** [December 27, 2022, 12:29pm UTC](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361 "2022-12-27T12:29:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaya](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/jaya/32/8786_2.png) [@jaya](https://discourse.jupyter.org/u/jaya)\
**Post date:** [December 27, 2022, 12:29pm UTC](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361/1 "2022-12-27T12:29:01Z")

</div>

I’m currently executing a notebook with a command like `jupyter-nbconvert --to notebook --execute work/notebook.ipynb`

There is a cell in the notebook that takes about 16 hours to run.

Is there a command line option to save the `notebook.nbconvert.ipynb` periodically, like every 5 minutes? I want to be able to check the cell’s output while it’s running.

---

<div class="post-metadata">

**Author:** ![fomightez](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/fomightez/32/495_2.png) [@fomightez](https://discourse.jupyter.org/u/fomightez)\
**Post date:** [December 27, 2022, 8:01pm UTC](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361/2 "2022-12-27T20:01:06Z")

</div>

My understanding is that `jupyter nbconvert` doesn’t even save on a per cell basis, let alone as the cell progresses. See the last paragraph of [my reply here](https://discourse.jupyter.org/t/how-to-run-all-cells-at-once-programmatically/16193/2) where I suggest papermill if you want to look in as the run progress **on a per cell basis**. I don’t know how papermill handles outputting mid-cell execution.

There are some places where you have to do above and beyond the standard notebook cell model to accomplish what you need. Traditionally, jupyter notebooks aren’t ideal for long-running compute tasks (and there are efforts to address that such as [this way to attach a sub-console to visualize intermediary results of a long-running computation](https://twitter.com/davidbrochart/status/1611368054772105217)); however, you are already partly addressing that shortcoming by executing the notebook on the command line. With some additional different approaches it usually can be adapted to still accomplish what you need. For example, it may be better to set up what is taking 16 hours to have logging steps where the progress you need to monitor is mirrored in an external log/text file (or other output forms) at times you control. You can also add in clean-up of that mirrored content at the end of the cell run so you aren’t necessarily permanently duplicating stdout or output. Or still using that approach of adding logging & mirroring, package things up into a script you can run separate from the notebook and then make output files that you then integrate into the notebook as documentation of that code run. Ideally, you can run that script behind a screen (or similar tech) and reattach to it and monitor it to your heart’s content using `ps` or other monitoring you’ve built in. Even if you don’t run the script separate packaging it up as a script you can run from inside the notebook with `%run` or `%run -i`, while adding the external logging or mirroring of output you want to monitor may suffice.

---

<div class="post-metadata">

**Author:** ![jaya](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/jaya/32/8786_2.png) [@jaya](https://discourse.jupyter.org/u/jaya)\
**Post date:** [December 27, 2022, 11:03pm UTC](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361/3 "2022-12-27T23:03:34Z")

</div>

> [@fomightez](#):
>
> package things up into a script

thanks. this is almost what i ended up doing.

```auto
ipython work/notebook.ipynb

```

i’m already running it in a `screen` with `-L` so just changing from `jupyter` to `ipython` gives me the desired result

---

<div class="post-metadata">

**Author:** ![fomightez](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/fomightez/32/495_2.png) [@fomightez](https://discourse.jupyter.org/u/fomightez)\
**Post date:** [February 2, 2023, 9:38pm UTC](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361/4 "2023-02-02T21:38:10Z")

</div>

[This approach to answer 'Get partial output from nbconvert.preprocessors.ExecutePreprocessor](https://stackoverflow.com/a/75316929/8508004) seems related and so I’m noting here. It saves the notebook every second. I think the OP was seeking something much less often, and so it could be edited.