It would be amazing to have a %%playground cell magic that runs a Jupyter cell in “safe” mode:
After cell execution, all changes would be reverted and all new defined variables deleted.
That could be useful for debugging.
Here is a fictional code example:
In your example, it looks simple. However, I have serious doubts that this model could be further elaborated to all kinds of edge cases. As any kind of Python code could be part of the playground_cell, there could be also commands that work with system resources. For example, how is this supposed to work?
#%% Cell 1
f = open("test.txt")
#%% Cell 2
%% playground_cell
print(f.read())
f.close()
#%% Cell 3
print(f.read())
Without the playground, in cell 3 the f.read() would fail because the file is already closed. If the playground works as intended, the file should be un-closed (re-opened?) and f.read() should still/again be possible. As the file is opened and closed, Python interacts with the operating system (Windows, Linux, MacOS, etc.) and Python tells the operating system to provide the file. Also closing the file is not completly Python-internal but instead this is achieved by interacting with the operating system. So if a playground is established, we not only need to keep track of Python-internal variables, but also the operating system state. And if we write to a file in the playground, should it disappear if it did not work out? You would need to define an undo operation for every statement issued there in case of a naive implementation or you need to have two copies of the whole operating system including its resources (virtual machines?).
The same issue arises for e.g. socket connections (how do you un-do a web request?) or hardware-related communication (how do you un-do issuing a command?). So even if you created a playground that only works for pure Python objects, you still would need to treat this environment with great care because some things have side effects that cannot be undone by definition. With the second approach of copying the whole system, there are many other issues like how the running process is transferred between them? How are system resources treated in that scenario? You cannot go back to the initial state in case of e.g. an encrypted connection with an external process because external processes might not have the concept of playgrounds.
In general, Jupyter only forwards the code in its respective programming language to the kernel. What you describe should most likely reside in the kernel. There are different solution approaches and none of them tackle all issues as sketched out above. They are also very resource-consuming and complex.
You might want to have a look at Step-back while debugging with IntelliTrace - Visual Studio Blog to get an idea of how Microsoft solved the issue of going backwards while debugging. Taking the system snapshots it deeply rooted in the core of the debugger. At python - Is it possible to step backwards in pdb? - Stack Overflow, people discuss how stepping back in Python is (not) possible. I guess this would be one important requirement for your idea. And have you seen GitHub - jupyterlab/debugger: A visual debugger for Jupyter notebooks, consoles, and source files before (which, by the way, works with the xeus kernel for debugging)? Maybe some of your day-to-day issues in debugging could be solved that way?
3 Likes
I wrote something very similar to this for SageMath using the fork system call a decade ago… Here’s a little example: CoCalc -- fork.ipynb
2 Likes
Thank you two for your feedback!
I agree with you @1kastner, edge cases are not very usable in a playground environment.
But also a playground with pure Python objects could be really useful.
Thanks, @William_Stein for your code sample, I will try this out!
And now a completely different use case idea of playgroundcells:
Lately, I had the idea to run Jupyter in 3 cell different types: “Header” , “Normal” and “Dependent”.
I’ve also built a library that then turns these notebooks into a HTML gallery:
https://kolibril13.github.io/plywood/
With these three cell types, one can easily break example code down to small pieces and see variations of parameters quite nicely.
This is also possible without the playground magic, but as “Normal” cells should stay independent of each other, users who create these galleries can introduce “Normal” cell dependencies by mistake.
@William_Stein : The @ fork magic will be very useful for “Normal” cells that are followed by other “Normal” cells.
Is there a way to make the subprocess last, when a “Normal” cell is followed by a “Dependent” cell?
It should then stop again, when another “Normal” cell is coming.
What do you think about this?
That is a very interesting solution building on top of a readily-available concept! Could you explain what exactly is forked? In your example it says “forked subprocess” so I guess it is based on certain (unixoid) operating system features?
Do I understand this correctly? In the HTML library, there are the header cells, normal cells, and dependent cells. The header cells set up the ground for one row, i.e. all cells right to it should see the imported modules etc. This is not true for different header cells. The normal cell can have the same code like a header cell but it should have a visual output. Each normal cell should only depend on the header cell in its row. They should never depend on each other. The dependent cells access the information in the normal cell.
So your real issue is how to isolate the different cells from each other so that the different Jupyter Notebooks cells that are not meant to affect each other in fact technically even do not have the chance to do so. Is that correct? And currently you only allow a sequence, there are no two depedent cells that depend one the same normal or dependent cell (which would result in a tree structure).
Some interesting ideas of how to isolate Python in Python I got from the sandboxing discussion here:
Since security might be less an issue in your case, the approach of exec("my_code()", dict(locals())) might be a suitable option? Here, you could rely on Python features that are identical on all operating systems. Obviously, the sandboxing behavior would be different from the fork approach which is also very interesting!
1 Like
Better late than never, I’ve now implemented the sandbox cell magic (see below), and it was much easier than expected! @1kastner, thank you so much for the exec idea!
This will be very useful, I’ve just tested it while generating this Matplotlib functions gallery using this notebook, and it was possible to isolate the individual Jupyter cells while still having all defined variables and imports available.
Here is the cell magic:
from IPython import get_ipython
from IPython.core.magic import (Magics, cell_magic, magics_class)
@magics_class
class MyMagic(Magics):
@cell_magic
def sandbox(self, line, cell):
exec(cell)
ipy = get_ipython()
ipy.register_magics(MyMagic)
x = 3
%%sandbox
x = x + 7
print(x)
Out: 10
x
out: 3
2 Likes
That is so nice to see! While not a sandbox in the sense of security, it is a neat sandbox for playing around with variables without effectively changing them.
1 Like
I was thinking about publishing this magic as a pypi package.
What would be a good name for this magic?
%%playground or %%sandbox ?
In the context of security, the term sandbox promises some real isolation, e.g. suitable to run code that you do not fully trust. Your package does not provide that because one can break out of eval() as described in some of my posts on the top. Thus, I’d rather go for the term playground which does not have such a connotation.
3 Likes
Good point, I will go with playground then.
-
The proposed
%%playgroundmagic is almost functionally equivalent to the built-in%%pythonmagic, but the latter spawns a new process and is therefore more isolated; for example your proposed magic will still have side effects like:%%playground import os os.environ["SIDEEFFECT"] = "1"import os os.environ["SIDEEFFECT"] # 1but
%%python import os os.environ["SIDEEFFECT_2"] = "1"import os os.environ["SIDEEFFECT_2"] # KeyError: 'SIDEEFFECT_2' -
If we get lucky next version of IPython will include guarded evaluation which might be of interest here (Replace greedy completer with guarded evaluation, improve dict completions by krassowski · Pull Request #13852 · ipython/ipython · GitHub)
-
Naming-wise,
%%isolated_namespaceor%%do_not_pollute_my_namespace(yes, verbose but accurate). Note we already have jupyterlab-plauground-plugin so for SEO probably better to avoid confusion.
1 Like
Naming-wise,
%%isolated_namespaceor%%do_not_pollute_my_namespace
Good point, I will change the name! But is isolated the right word here? Variables from the other cells can still be accessed with the playground magic. I was thinking that %%branched_namespace might be a good name.
The proposed
%%playgroundmagic is almost functionally equivalent to the built-in%%python
Similar yes, but I want to access the variables, functions and imports from other cells.
That is not possible with the %%python magic:
x = 10
%%python
x
#NameError: name 'x' is not defined
Therefore, %%python won’t be too helpful in my goal to bring example branching into documentation of python packages with image output.
If we get lucky next version of IPython will include guarded evaluation which might be of interest here.
I am curious to learn more about how guarded_eval() could be used here. Just had a look at the PR, but I don’t have the background knowledge to understand what’s going on there.
One thought experiment: Would there be also an option to make even sub-branches of namespaces? A potential use case would be for this gallery to show variations of examples.
I just had the idea of the following workaround:
A package that allows to run an isolated cell-execution-cain from registered cells.
This package would be very similar to the %%python magic, but custom code can pre-run before the code of the cell with the magic itself is executed.
This needs two cell magics and works the following way:
- A code cell can be registered by
%%register_cell_chain_entry MyFistCell - The cell chain can be run in an isolated environment by
%%run_isolated_cell_chain --pre_run_code MyFistCell.
The code of the cell that contains this magic will be attached at the end of the code chain.
Here is an example:
z = 42
%%register_cell_chain_entry MyFistCell
x = 1
%%run_isolated_cell_chain MyOtherCell
y = x + 1
%%run_isolated_cell_chain --pre_run_code MyFistCell MyOtherCell
z = y + 1
print(z)
#Out : 3
print(z)
#Out:42