Inline variable insertion in markdown

[choldgraf]

On the topic of syntax, just pulling down some ideas from earlier in the thread

I think these were the main ones other than what has just been mentioned:

{{. expression() }} OR   {{.python expression() }}
[]:`expression()`    OR   [python]:`expression()`
[]`expression()`     OR   [python]`expression()`

I actually kind-of like having brackets in there somewhere, since Jupyter already uses them in In/Out statements and this kind-of conceptually maps onto these executable markdown expressions as well. Also is a nice parallel to the IP[y]: logo

[agoose77]

I’ve updated the Binder so that you can play around with different start/stop tokens. I haven’t checked what predence we’re currently using over other inline markup tokenizers, so it is possible that some syntax won’t work because MarkdownIt will first try a different tokenizer.

[choldgraf]

@agoose77 it’d be helpful if you provided some minimal instructions for how people could try different syntax. The example/ notebook still only shows off one kind of syntax, and I couldn’t find anything about switching syntax in the README.

[agoose77]

Good point.

In the Advanced Settings viewer, there is a “Markdown Plugins” section,
which you can replace with the following JSON

{
    "plugin-options": {
        "markdown-it-expression": [
            {
                "openDelim": "<<",
                "closeDelim": ">>"
            }
        ]
    }
}

Due to a missing feature in jupyterlab-markup, you have to reload the page after making these changes (or close and re-open the notebook).

[fperez]

Thanks for pushing forward and making the binder available, @agoose77!

One use case that’s not quite obvious in how to make it work right now, but that I think would be extremely valuable, is rendering auto-generated LaTeX:

Sympy can create latex representations, and being able to insert rendered math from computations directly in the prose would be fantastic for certain kinds of writing. But right now the mathjax rendering collides with this…

It’s late and I’m tired, so I don’t have any suggestions to make right now, but figured I’d flag it as a use case for the design space…

[fperez]

ps - that example was inspired by this short post I came across that got me thinking about using JupyterLab + Book + sympy in authoring of heavily mathematical content…

[agoose77]

@fperez on the Binder I posted above I can directly embed the rich rendered result, which is where I picture Markdown-embedding being most useful:

If you try to embed the result of latex() by itself, it is display()-ed as text/plain, whereas if you were to wrap that in a IPython.display.Latex object, it would be rich rendered.

Here’s a direct Binder link

[fperez]

Excellent, thanks for that example. Even simpler than what I was attempting, glad to see that works!

Thinking along these lines, it would be cool to be able to compose hand-crafted LaTeX expressions with dollar-math that included sub-expressions pulled in from the kernel, but that sounds complicated enough to implement not to worry about it for now.

Excellent to have a live binder to experiment with this, thanks again!

[agoose77]

I'm now no longer iterating further on this concept, because I think it's reached a point where we have a good base implementation to use as a reference in discussions.

I’m convinced that this is a must-have for Jupyter, and (as outlined above) I think we need bigger discussions to address the next steps. I’d like to quickly say a big thank-you to all the contributors here for some interesting ideas and observations.

I don’t really know how best to take the next steps with this idea. I’m going to present it at the community call on Tuesday, so perhaps some ideas will come from that discussion.

[blois]

An alternative to consider is something more akin to the %%markdown cell magic:

1. When executed, parse the expressions out of the markdown string.
2. Evaluate all of the expressions
3. Return in an item with a _repr_mimebundle_ which includes:
   1. text/markdown which is the original markdown string with expressions replaced with the string representation of the value.
   2. text/plain which is the same as text/markdown.
   3. application/x-imarkdown which contains the original markdown string.
   4. A metadata object with application/x-imarkdown set to a dict with the keys being the expressions and the values being the result of shell.display_formatter.format(value).

There are upsides and downsides to this approach:

1. Cell being code maintains the ‘execution’ semantics- execution count works, UI around execution time works, clear all outputs works, etc.
2. Will ‘just work’ with tools like nbconvert, github doc viewing etc. It has graceful fallback behavior.
3. The support displaying the imarkdown output format with rich inline-editors may be something that other code libraries would generate as well.
4. Downside is that %%imarkdown would not feel as first-class as it does in other tools without additional client work.

I feel like there’s a bit of a balance between 1 & 4- how much of code cell semantics would an executable text cell need and how much ‘special handling’ would a code cell beginning with %%imarkdown need.

[agoose77]

This is definitely another way of doing things. I think the downside of moving this into the kernel (via magics) is that we then need to do this for each kernel rather than at the frontend.

This does touch on the idea that it would be nice to be able to provide a fallback, which relates to Notebook Cell-Type Generaliastion - #7 by agoose77. I’d like to be able to specify a text/markdown fallback which removes the injection markup, i.e. what is {{ x }}? → what is ⚠?

[chrisjsewell]

Hey gang, FYI I just added inline variable evaluation into myst-nb (the underlying package for jupyter-book) : Inline variable evaluation (eval)

Would love to see this get implemented on the Jupyter side, but I feel that’s not going to happen unless there is a clear advocate within the core Jupyter team.
Honestly, that’s my frustration with some of these discourse threads; I feel they tend to devolve into many people having many different (irreconcilable?) view points, and nothing ever actually getting done

[agoose77]

and nothing ever actually getting done

I think one could generalise this more broadly to a “problem” with forum discourse! But, that’s not necessarily a bad thing. My assessment of the present situation is that the abstract feature of “inline variable rendering” is a very open ended one, and simultaneously involves the notebook schema, and every notebook frontend at the same time! Because of this, I reckon it’s going to need someone in the core team to either feel passionately enough to guide through a project, or at the very least provide good feedback about how things align with the wider Jupyter ecosystem.

I also wonder whether the elephant in the room w.r.t Markdown specification makes people nervous about going anywhere near modifying the Markdown features.

Maybe half the battle is getting something into users hands. I think you’ve done something fantastic with this eval directive, and that’s a great first step.

Let me be clear though - if I want this feature, I should write a PR. I don’t expect the core team to do it for me, in case my post has given that impression.

[chrisjsewell]

Yeh exactly “nothing ever actually getting done” is obviously a bit harsh, but that is my honest user feedback, especially when it comes to all this Markdown stuff in Jupyter.
I (and I’m sure everyone else ) has way too much work, to dedicate time/effort to something that may likely go nowhere, and it just feels like way too deep of a “local minima” to overcome within the Jupyter world

[chrisjsewell]

@chrisjsewell

it just feels like way too deep of a “local minima” to overcome within the Jupyter world

happy to be proved wrong though

[agoose77]

I’ve just released 0.2.0 for jupyterlab-imarkdown. This release cleans up the implementation now that I want to try and rely on this package for jupyterlab-myst. It also removes the jupyterlab-markup parser that added the {{ 1 + 2 }} syntax. Out of the box, jupyterlab-imarkdown only supports <input type="hidden" class="eval-expr" value="1 + 2">.

There is a new, separate JupyterLab plugin jupyterlab-markup-expr that generates these input elements from {{ expr }}. You can install this alongside jupyterlab-imarkdown.

Evaluated expressions are now stored in the cell metadata under the user_expressions key. This is not to be relied upon - it may move in future!

[rowanc1]

As a follow up on this work @agoose77 last posted:

@agoose77 and some of the ExecutableBooks team have been working on jupyterlab-myst which brings these ideas to life with additional MyST Markdown. This supports the same inline execution as jupyterlab-imarkdown (in the same metadata as well) and also incorporates it into other MyST Markdown styles like admonitions, tabs, proofs, etc. Works with widgets, text, math, and inline matplotlib images as sparklines.

In this implementation we went with the {eval}`x.sum()` syntax, which is already present in MyST Markdown.

A blog post about our implementation is here:

Using MyST Markdown in JupyterLab

[stefanv]

Thank you for your work on this; it looks fantastic!