"Custom Messages" suggests the front-end can communicate to the kernel. How is this done?

Widgets
,
1

According the the Messages documentation on custom messages, I quote,
" … messaging system for developers to add their own objects with Frontend and Kernel-side components, and allow them to communicate with each other.
".

It says " The Kernel listens for these messages on the Shell channel, and the Frontend listens for them on the IOPub channel.".

Now, how in Javascript one connects to IOPub channel ? Is there a library for that ? Or an examples somewhere ?

Thanks in advance for any pointers!

2

Yes, it’s possible to do all kinds of exciting things with kernel messages: the trick is getting kernels to support them, and handshaking between versions of kernels/clients.

For an approach less likely to conflict with existing messages, I usually recommend investigating (if not directly using) the comm “meta” messages, which provides a documented, namespaced framework for custom interaction. This system is implemented in ipykernel (and probably all wrapper kernels), IRKernel and IJulia, and likely other kernels. Among the nice features is built-in support for binary buffers, which can provide significant performance benefits over marshalling everything as text/JSON for e.g. typed numeric arrays, video, audio, etc.

The original useful implementation of the comm infrastructure is Jupyter Widgets, which use a few named comm targets in its messages. And indeed, I would recommend thinking about wrapping whatever this feature is going to be in a widget, as then it can play nicely with all manner of other interaction pieces, while providing a useful abstraction above raw message-banging.

This is discussed at greater length on this thread. Some excerpted links:

1 Like
3

hey @bollwyvl , thanks for following up on the question!

I think the kernel is the easy part in my case, since I’m writing it. So once you pointed me to “Custom Messages” in the Messaging docs in the other thread, in 10 minutes my kernel was “supporting” (that is, simply printing them out) the comm messages.

My question is what needs to happen in the client’s javascript side ?

What would be the minimal %js cell that could open the websocket and send a “comm_msg” message ? Or a “comm_info_request” ?

Notice, my view of the world is the following:

(1) Kernel <---- ZMQ ----> (2) JupyterServer(?) <—> WebSockets(??) <—> (3) Jupyter WebApp/FrontEnd (in Browser) <—> (4) User Javascript/Wasm code (Inside the output of a cell)

And I’m not yet sure if it’s possible to communicate from (4) (the output of a cell) to (1), or only from (3) (the Jupyter webapp, where I think(?) Jupyter Widgets extensions execute).

Thanks for the link on the Jupyter Widgets protocol, I could also try to use Juptyer Widgets to communicate. But then to instantiate the widgets I have reverse engineer (my kernel is in Go) the python versions (I noticed it uses a custom mime type in the “execute_result” or “display_data” messages).

Notice the Low Level Widget Explanation page doesn’t explain how to connect from the web app side (or cell output side).

cheers

Edit: I also found the Websocket Kernel Wire Protocols pages, that specifies what goes on the websocket that presumably connects the front-end to JupyterServer. But it doesn’t say the address I should open that websocket. Or, if I should reuse the one Jupyter already has opened, how do I do it from the javascript that is the output of a cell ?

4

Once it “leaves the nest,” the only thing a kernel ever does with an output is, potentially, to update it, replacing its data and metadata. It has no knowledge of where it’s going, as the other end of the ZMQ could be… anything.

comm provides a two-way pipe to talk about any thing, one of which could be an agreed-upon structure of something in a client that should change when the kernel’s thoughts of it changes, and conversely should report to the kernel that something has changed in the client. This is, in a nutshell, what “A Jupyter Widget” is, which has triggered other features such as saving a whole notebook’s worth of state in the .ipynb file.

And indeed, the HTML widget is exactly a single DOM node container, inside of which could be anything, without any particular reference to the surrounding client UI.

There isn’t a lot of docs about re-implementing some of these things, as comm is already implemented in the client, e.g. @jupyterlab/services, along with the references in the other thread to various kernels that implement a CommManager.

In that model:

  • the client knows in advance it might get these kinds of messages
    • a manager is created for each soon-to-be-kernel-owning thing (in this case a NotebookPanel)
  • when the kernel becomes available
    • it registers its named comm target
    • it asks for any existing client/kernel shared objects
      • if it finds any, it draws them on the page if their display object placeholders exist
    • it manages an internal state of the model, in the case of HTML just value (some raw HTML)
  • when a user executes code
    • a live object is created in the kernel
      • as part of its startup machinery, it sends a comm_open
    • if the user asks to display the object
      • it sends a display_data message
      • the client draws it in the right place
      • is set up to observe (and update) the underlying model

As for reverse engineering the existing implementation:

While there is a spec for the wrapper format, the actual content inside them that corresponds to different widget classes like HTML are currently tied up in the python implementation. This broader discussion is on-going: tl;dr: it would be nice if there was a JSON schema which could be used directly by downstreams, either to do static codegen or dynamic instantation.

From live code opjects: here’s an example of instantiating live python objects, then generating backend classes. Here, “backend” means “running in a WebWorker in JS,” but it’s basically the same idea: “not python”.

From the wire: off-the-shelf tools such as wireshark would work, but one can also load up jupyterlab-kernelspy or use the browsers’ built-in websocket message viewer, which have gotten much better of late.

image
image2304×1752 426 KB

5

Thanks again @bollwyvl, the help is very appreciated!

So while I understand the separation of the kernel to where its outputs are going to be displayed, I’ll limit myself for now to the concrete case where the display is a browser with Javascript support.

And my (still the same) question is how in Javascript to send a message back to the kernel.

The link to the comm.ts look similar to the messages I saw floating around when I connected to the JupyterServer path /api/kernels/<kernel id??>/channels using the standard Javascript Websocket, and printed out what was being sent there.

Is that the URL I need to connect a WebSocket to, in order to exchange messages with the IOPub ZeroMQ in the JupyterServer, and through it to the kernel ?

The links and description of the CommManager are interesting. I believe though once I get the “comm_*” messages flowing from/to Javascript, I can design my own CommManager to keep shared values in sync (between the Javascript widget and my Go kernel) – I’ve implemented similar protocols on distributed systems before, so I don’t foresee any difficulties (I hope at least :slight_smile: , I may be reaching out with more questions in the future).

ps.: Javascript that I’m using to debug (most of it):

(()  => {
    var conn;
    var msg = document.getElementById("{{.MsgId}}");
    var log = document.getElementById("{{.LogId}}");
    var form = document.getElementById("{{.FormId}}");

    form.onsubmit = function () {
        if (!conn) {
            return false;
        }
        if (!msg.value) {
            return false;
        }
        conn.send(msg.value);
        msg.value = "";
        return false;
    };

    if (window["WebSocket"]) {
        conn = new WebSocket("ws://" + document.location.host + "/api/kernels/{{.KernelId}}/channels");
        conn.onclose = function (evt) {
            var item = document.createElement("div");
            item.innerHTML = "<b>Connection closed.</b>";
            appendLog(item);
        };
        conn.onmessage = function (evt) {
            const data = JSON.parse(evt.data);
            console.log(JSON.stringify(data, null, 2));
        };
    }
})();

(With the “id” fields being generated dynamically by executing it as a Go template)