I haven’t much more to add here - I’m mainly using this thread as a journal of ideas.
Cells as Transforms
In generalising the notebook schema, I think a basic cell definition would have
-
data- source of cell -
outputs- existing code-cell outputs -
mimetype- mimetype of cell
I was thinking about the multiple-view feature, and I actually think that’s an orthogonal feature of selecting which mimetype should be rendered in an output.
Maybe what we want is to consider a cell undergoing a transformation from data to outputs. We then have a distinction between a notebook consumer (a read-only viewer of the notebook, e.g. nbconvert without the execution), and a notebook transformer (a read-write viewer of the notebook, e.g. JupyterLab).
I imagine a Markdown cell is then just a transform on the data. But, unlike the present situation where this transformation is special-cased in JupyterLab’s cell handling, we have a text/markdown;flavor=GFM transformer that runs markdown-it. It’s up to the frontend to do this, whether that’s nbconvert, jupyterlab etc. The frontend can make some per-mimetype decisions to improve the UX, e.g. special shortcuts for Markdown ctrl+b to insert emphasis during editing, or defaulting Markdown cells to collapse the source when opening a notebook, like ObservableJS which separates input and output:
The significant benefit of this in the Markdown case is that your transformer can choose to embed both the original text/markdown data and rendered HTML text/html into the outputs bundle, so that HTML frontends are left with no ambiguity as to how to render the markup. In fact, the best approach here would be to always consider the cell itself alongside its outputs, e.g.
mimebundles = cell.outputs or {cell.mimetype: cell.data}
The really interesting side effect of this is that the problems discussed in Inline variable insertion in markdown become less important - we don’t have to enshrine a “perfect solution” because anyone can write a markdown transformer and associate it with the Markdown cell type.
Writing jupyterlab-imarkdown was fiddly, because (as I recall) JupyterLab’s notebook rendering is slightly “all or nothing”. This new system would make it easy to modify a cell’s rendering using this transform system.
Drawbacks
Right now, we can get fairly far using jupytext to convert between notebook representations. This works because ultimately notebooks are code-focused, and we can settle on a code representation by dropping any Markdown / Raw cells. However, by extending to arbitrary MIME types, jupytext et al. would need to become smarter to support this system. This could actually be a good thing - it would be a new avenue for developers to be able to extend jupytext with per-cell transformations etc.