Lightweight semantic jupyter notebook screencasting with RTC

At the JupyterCon coding sprint today, @kolibril13 initiated today a brainstorm with
(? sorry, I am bad with names and should have written it down) and myself about
screencasting jupyterlab notebook sessions in a better way than with a plain video,
so that:

  • the screen cast file would be much more lightweight
  • the user could interrupt at any time the screen cast and start live coding from there; and get back to the screen cast.
  • the screen cast would (ideally) self adapt to the user’s device and configuration
  • accessibility! the user is facing a real running application, and can use screen readers & such.

@kolibril13 had gotten the inspiration from application XXX (@kolibril13, can you remind me?)

Ideas that were floated around:

  • Use RRWeb ; rrweb is about pixel-perfect replay, not self-adapting replay
  • Use Jupyter-graffiti: jupyter 6 only at this stage
  • Use Jupyter telemetry to record the user actions, and replay them

At the end, we realized that a very natural candidate would be to use JupyterLab’s Real Time Collaboration: after all, RTC is all about replaying on another device the user’s actions on a Jupyter notebook. Here the replay is just delayed in time.

Here is some feedback from @SylvainCorlay :

Indeed we already considered using RTC to replay a session in a Jupyter notebook, though we only had in mind the use case of debugging RTC. This should be feasible since there is a full history of the ystore in the backend. All that is missing is a user interface. @kolibril13: if you’d be interested in exploring this, please get in touch!

@SylvainCorlay : are there timestamps in the history of the ystore so that one would know at which rate to replay the events?

Caveats:

  • RTC is not tracking the user’s mouse; without additional work this form of screencasting would not be adequate for showing user interactions involving the mouse; and interactions outside of the notebook for that matter.
  • I am not sure how RTC handles interactions with the kernel: here we really need to replay these interactions to have a kernel in the proper state.

Cheers,
Nicolas

Thoughts anyone?

3 Likes

Hey one possibility could be to use the code generator from the integration test framework playwright

Pro:

  • record all actions
  • can generate scenario in python, node js, …
  • is cross browser
    Cons
  • cannot be executed on your classical browser. It requires installing custom browsers (see playwright install)
  • it is tricky to pass the authentication
3 Likes