Inline variable insertion in markdown

Notebook
, ,
95

I’ve had some thoughts about this! I feel that this entire discussion is the tip of a much larger question about the Notebook (which I’ve tried unsuccessfully to reason about here). Hopefully, you’ll forgive me for talking to that as we go along? I don’t think we can solve this problem without at least discussing the other problems (as I see them). Also, apologies if I’ve touched on this already; I think this is a slow descent into madness.

TL;DR — (imo) there are some big problems that need to be solved before we can look at making this an official part of the Jupyter Notebook spec, so let’s not make any drastic changes like new cell types just yet.

Rendering expressions is a specific case of injecting content

The prototype for Jupyterlab separates the kernel query (that generates the attachments) from the rendering stage (that retrieves the cell attachments). It does this by generating a mapping from a templated name `${ATTACHMENT_PREFIX}-${index}` to the DOM node that marks the location of the rendered attachment.

I could see a future extension exposing attachments in a JupyterLab UI, and allowing them to be directly embedded via emarkdown, e.g.

Let's embed a video here: {{#video}}

In an ideal world, the XMarkdown class that we currently have would only be responsible for transforming rendered Markdown (HTML) to inject the rendered attachments; the kernel processing would be managed by another extension that understands the kernel and populates the attachments. This is almost what we have now, except that currently we assume that the markup embeds an expression, and generate the attachment name in TS, rather than embedding the attachment name in the HTML. I might open a discussion on this idea, actually, as I think it’s important to get right.

So, because of this, I think cell.attachments is the right place to store the MIME bundles, rather than a new cell property. Maybe need a new cell type at the same time, though (see next point)!

We have introduced a new Markdown flavour

The Notebook markdown is “standardised” on marked.js, but even this standard is loose. Whatever we use to implement inline-expressions, the resulting Markdown will not behave properly when rendered in a default marked.js viewer; we’re creating our own flavour.

I like your use of emarkdown. Given the idea to generalise this to “embed attachments”, maybe EMarkdown is a good name (embedded-markdown rather than expression-markdown) for our flavour.

For this reason, I think a new cell type might help enforce this distinction. However, see my final point as to why we might also not do this.

But, elsewhere there is more than one Markdown flavour

If we only consider the context of emarkdown, then we can probably create a new cell type, and be done with it. However, notebooks containing ill-defined Markdown flavours is already a problem elsewhere:

  • jupterlab-markup re-uses the Markdown cell with a different (custom) flavour (in JLab)
  • Jupyter Book re-uses the Markdown cell with MyST syntax (rendered + JLab)

This is the Big Issue™ that I’m worried about. Things are chaotic already; introducing a new flavour of Markdown is just going to make it worse if we can’t concretely define it for frontends.

I don’t think that these use cases can be solved by choosing one particular flavour; users should be able to add “extra” features through extensions, just like we do for other areas of JupyterLab (e.g. rendermime). Equally, at present we have “no” guarantees about what the frontend can render, or an indication about what the notebook needs. A user would have to have existing knowledge of various MyST markup to realise that they needed to install extensions.

In my opinion, any solution to this problem would need to accomplish the following:

  1. Establish/choose a system of identifying Markdown syntaxes
  2. Store this metadata in the notebook (at notebook or cell level)

The scope of this change may vary

I’d love it if the various stakeholders could get together to discuss this. @bollwyvl has spoken on this before, but I don’t know who else needs to be involved.

Conclusion

Maybe the “short-term” solution to this is to boot the problem down the road; there is nothing in the notebook schema that stops us from using different flavours — the original Markdown cell doesn’t have a hard-defined flavour, and unless we can introduce one here, we should avoid making a new cell type that isn’t any more well-defined than the original one!

Let’s just keep the existing Markdown cell + attachments, and wait to implement this in core (e.g. nbclient) until we’re ready to solve these problems.

1 Like