Hi all - just joined. I wrote Mathcad back in 1986 so dynamic documents are my thing ;-} Have used R/Rmd, PHP, JSP, ASP, Mathematica Notebooks - and now Jupyter. Love it so far, but stumped trying to do something I think of as obvious - substitution macros that reference kernel variables containing markdown inline in a markdown cell. I read that there was once a package that allowed this, but not supported in Jupyter-lab and I saw a hack to make a code cell look like a markdown cell. Two questions:
(1) is there a way ? and (2) does this somehow go against the grain of Jupyter’s intended use?
There have been some efforts to extend the spec in a useful way, one way or another, for dynamic markdown substitution based on various template syntaxes, but the sticking point has generally been: “Jupyter” can run, most likely, all of the linked languages, or at least their underlying runtimes, which introduces a high likelihood of syntax collision in the relatively under-specificed Markdown “grammar”.
It also has a lot of different clients and tools that use the various Jupyter protocols and file formats, some which don’t even look like the classic Notebook, or JupyterLab, and may not use markdown cells at all. Further, the spec doesn’t provide a compelling way to document, in a cross-language, cross-client manner, what a “named variable” even is (though the DAP integration goes a bit further), much less complex control flow which templates can often inject. Finally, as Markdown is never supposed to really “break” (unlike e.g. LaTeX, ReST), storing the cached state of a template expression would need to be solved as well.
Part of this stems from the somewhat arbitrary distinction between a “static” markdown cell, a “dynamic” code cell, and the “whatever” of a raw cell, and the lack of knowledge by a cell of its source content. Some ongoing work is exploring whether a longer-term solution would be to conflate all the cell types together, making source into a self-describing structure, keyed by mime types, much like how outputs are currently described. This would allow for all kinds of crazy features, such as multiple (uncoordinated) kernels in a notebook, rendering “narrative” cells in a kernel, and who knows what else.
3 Likes
Thanks for writing this, I have a much better idea of how this all works now (a little knowledge is a dangerous thing they say), and why it seems like a simple feature but it’s not. I’m intrigued by the long-term solutions you mention, if there are open discussions, please point me to them.
Maybe it was Knuth’s “literate programming” that first thought to apply both rendering and computing engines to marked up text, and here we are still at it!
Sorry for the delay in replying!
open discussions
There’s a slot on Tuesday called Rethinking Notebook Cells (GitHub issue) on the Community Calendar… though attendance has been pretty sparse, recently (mostly the host and myself). But more’s the merrier!
There are number of ideas in the Jupyter Enhancement Proposal pipeline that would be very useful to effecting real change, mostly around standards alignment… of note, one just got merged (with work not yet started) which would create a schemas.jupyter.org (or something) which would serve as both a long-term, resolvable home for things, and documentation. The “or something” is partially driven by a desire for it to be able to host a number of specs, including grammars and derived type stubs, so that some of the ideas can be realized more concretely without every implementation/language starting from scratch.
However, there are some other proposals that would move Jupyter further away from markdown “standards” to more platform-specific features, and introducing even more “special” schema features per cell type, etc.
“literate programming”
Anyhow: part of the novel challenge is evolving the classic tangle/weave phases of literate programming production to something which takes into account not only the many languages supported by Jupyter tools, but also the conflict between the “high energy” state of authoring a Jupyter document and the “low energy” state of viewing a static(-ish), archival-grade state… in a sense this even goes beyond the standard concept of “hypermedia,” as an author/viewer would likely expect to see per-keystroke/mouse nudge changes reflected in one, or multiple, places in a document… or even multiple documents. Of course, this immediately crashes into the The Hard Problems of naming things and cache invalidation.
long-term solutions
If those are “solved”, I’ve imagined repurposing the markdown link syntax, already overloaded at the protocol level for markdown cell attachments, e.g.
The value of `x` was .
Which would be naively rendered with a “broken” image (showing the alt text):
The value of
xwas.
By moving transclusion out of the kernel language, no particular language/format would be preferred, and no new syntax would be created. In its “low energy” state, the resulting HTML would be fully resolved as an output (rather than the “special” state of markdown). In the “high energy” state, DOM could be updated dynamically, with anything that knows how to work with rich displays (usually one of the first features implemented in a kernel). It gets fun when considering the ready-built ecosystem of Jupyter Widgets, some of which would even work without kernels (though, generally, with CDN, buh), such that was above could be replaced with is now.
By adopting “real” URL/URIs, transclusion could theoretically be extended to embedding cells, or whole other notebooks, inside other cells. The other knock-on effect of strong, standards-based IDs and references is the improved feasibility of oft-requested features like annotation and single- or multiple-document internationalization.
1 Like
Thanks much for this, I feel myself drawn in (and not up to speed - I need to read more and chat less)!
Avoiding adding to kernel languages is good, couldn’t DOM approach do that? Hi-Lo energy-wise, I would ask whether or not the final-form document is the top-level paradigm, regardless of dynamism. One makes changes and whenever things settle down, we have a new state of the document regardless of pace (energy). Linearity plays a role too or it would be a dashboard, not a document.
In Mathcad, equations/expressions had two semantics: they were run as calculations, and they were rendered as math markup. Plots and graphs were just operators with complex rendering and no calculation purpose. In notebooks, programming languages are rendered as usual in text, and its output that must be transcluded, no?
In any case, for the original need that brought me here, I wonder, what is that cell-like place below a code cell where its output is written? It’s not really a cell, is it? What if it was a markdown cell in which the author put named placeholders like {{foo}}, and when the code cell above (with any kernel) printed text, it could print to “foo”? That following md cell would then be rendered, including whatever was printed to “foo” rendered in place of {{foo}} inline? That way there are no value caching issues, it’s just a convenient way to format what a code cell prints, rather than just rendering it as text below the code cell.