Idea: `%%playground` cell magic for debugging

It would be amazing to have a %%playground cell magic that runs a Jupyter cell in “safe” mode:
After cell execution, all changes would be reverted and all new defined variables deleted.
That could be useful for debugging.
Here is a fictional code example:

In your example, it looks simple. However, I have serious doubts that this model could be further elaborated to all kinds of edge cases. As any kind of Python code could be part of the playground_cell, there could be also commands that work with system resources. For example, how is this supposed to work?


#%% Cell 1
f = open("test.txt")

#%% Cell 2

%% playground_cell
print(f.read())
f.close()

#%% Cell 3
print(f.read())

Without the playground, in cell 3 the f.read() would fail because the file is already closed. If the playground works as intended, the file should be un-closed (re-opened?) and f.read() should still/again be possible. As the file is opened and closed, Python interacts with the operating system (Windows, Linux, MacOS, etc.) and Python tells the operating system to provide the file. Also closing the file is not completly Python-internal but instead this is achieved by interacting with the operating system. So if a playground is established, we not only need to keep track of Python-internal variables, but also the operating system state. And if we write to a file in the playground, should it disappear if it did not work out? You would need to define an undo operation for every statement issued there in case of a naive implementation or you need to have two copies of the whole operating system including its resources (virtual machines?).

The same issue arises for e.g. socket connections (how do you un-do a web request?) or hardware-related communication (how do you un-do issuing a command?). So even if you created a playground that only works for pure Python objects, you still would need to treat this environment with great care because some things have side effects that cannot be undone by definition. With the second approach of copying the whole system, there are many other issues like how the running process is transferred between them? How are system resources treated in that scenario? You cannot go back to the initial state in case of e.g. an encrypted connection with an external process because external processes might not have the concept of playgrounds.

In general, Jupyter only forwards the code in its respective programming language to the kernel. What you describe should most likely reside in the kernel. There are different solution approaches and none of them tackle all issues as sketched out above. They are also very resource-consuming and complex.

You might want to have a look at Step-back while debugging with IntelliTrace - Visual Studio Blog to get an idea of how Microsoft solved the issue of going backwards while debugging. Taking the system snapshots it deeply rooted in the core of the debugger. At python - Is it possible to step backwards in pdb? - Stack Overflow, people discuss how stepping back in Python is (not) possible. I guess this would be one important requirement for your idea. And have you seen GitHub - jupyterlab/debugger: A visual debugger for Jupyter notebooks, consoles, and source files before (which, by the way, works with the xeus kernel for debugging)? Maybe some of your day-to-day issues in debugging could be solved that way?

3 Likes

I wrote something very similar to this for SageMath using the fork system call a decade ago… Here’s a little example: CoCalc -- fork.ipynb

2 Likes

Thank you two for your feedback!
I agree with you @1kastner, edge cases are not very usable in a playground environment.
But also a playground with pure Python objects could be really useful.
Thanks, @William_Stein for your code sample, I will try this out!

And now a completely different use case idea of playgroundcells:

Lately, I had the idea to run Jupyter in 3 cell different types: “Header” , “Normal” and “Dependent”.
I’ve also built a library that then turns these notebooks into a HTML gallery:
https://kolibril13.github.io/plywood/

With these three cell types, one can easily break example code down to small pieces and see variations of parameters quite nicely.

This is also possible without the playground magic, but as “Normal” cells should stay independent of each other, users who create these galleries can introduce “Normal” cell dependencies by mistake.

@William_Stein : The @ fork magic will be very useful for “Normal” cells that are followed by other “Normal” cells.

Is there a way to make the subprocess last, when a “Normal” cell is followed by a “Dependent” cell?
It should then stop again, when another “Normal” cell is coming.
What do you think about this?

That is a very interesting solution building on top of a readily-available concept! Could you explain what exactly is forked? In your example it says “forked subprocess” so I guess it is based on certain (unixoid) operating system features?

Do I understand this correctly? In the HTML library, there are the header cells, normal cells, and dependent cells. The header cells set up the ground for one row, i.e. all cells right to it should see the imported modules etc. This is not true for different header cells. The normal cell can have the same code like a header cell but it should have a visual output. Each normal cell should only depend on the header cell in its row. They should never depend on each other. The dependent cells access the information in the normal cell.

So your real issue is how to isolate the different cells from each other so that the different Jupyter Notebooks cells that are not meant to affect each other in fact technically even do not have the chance to do so. Is that correct? And currently you only allow a sequence, there are no two depedent cells that depend one the same normal or dependent cell (which would result in a tree structure).

Some interesting ideas of how to isolate Python in Python I got from the sandboxing discussion here:

Since security might be less an issue in your case, the approach of exec("my_code()", dict(locals())) might be a suitable option? Here, you could rely on Python features that are identical on all operating systems. Obviously, the sandboxing behavior would be different from the fork approach which is also very interesting!

1 Like