"Custom Messages" suggests the front-end can communicate to the kernel. How is this done?

[Jan_Pfeifer]

According the the Messages documentation on custom messages, I quote,
" … messaging system for developers to add their own objects with Frontend and Kernel-side components, and allow them to communicate with each other.
".

It says " The Kernel listens for these messages on the Shell channel, and the Frontend listens for them on the IOPub channel.".

Now, how in Javascript one connects to IOPub channel ? Is there a library for that ? Or an examples somewhere ?

Thanks in advance for any pointers!

[bollwyvl]

Yes, it’s possible to do all kinds of exciting things with kernel messages: the trick is getting kernels to support them, and handshaking between versions of kernels/clients.

For an approach less likely to conflict with existing messages, I usually recommend investigating (if not directly using) the comm “meta” messages, which provides a documented, namespaced framework for custom interaction. This system is implemented in ipykernel (and probably all wrapper kernels), IRKernel and IJulia, and likely other kernels. Among the nice features is built-in support for binary buffers, which can provide significant performance benefits over marshalling everything as text/JSON for e.g. typed numeric arrays, video, audio, etc.

The original useful implementation of the comm infrastructure is Jupyter Widgets, which use a few named comm targets in its messages. And indeed, I would recommend thinking about wrapping whatever this feature is going to be in a widget, as then it can play nicely with all manner of other interaction pieces, while providing a useful abstraction above raw message-banging.

This is discussed at greater length on this thread. Some excerpted links:

• Low Level Widget Explanation — Jupyter Widgets 8.1.0 documentation
• Messaging in Jupyter — jupyter_client 8.3.1 documentation

[Jan_Pfeifer]

hey @bollwyvl , thanks for following up on the question!

I think the kernel is the easy part in my case, since I’m writing it. So once you pointed me to “Custom Messages” in the Messaging docs in the other thread, in 10 minutes my kernel was “supporting” (that is, simply printing them out) the comm messages.

My question is what needs to happen in the client’s javascript side ?

What would be the minimal %js cell that could open the websocket and send a “comm_msg” message ? Or a “comm_info_request” ?

Notice, my view of the world is the following:

(1) Kernel <---- ZMQ ----> (2) JupyterServer(?) <—> WebSockets(??) <—> (3) Jupyter WebApp/FrontEnd (in Browser) <—> (4) User Javascript/Wasm code (Inside the output of a cell)

And I’m not yet sure if it’s possible to communicate from (4) (the output of a cell) to (1), or only from (3) (the Jupyter webapp, where I think(?) Jupyter Widgets extensions execute).

Thanks for the link on the Jupyter Widgets protocol, I could also try to use Juptyer Widgets to communicate. But then to instantiate the widgets I have reverse engineer (my kernel is in Go) the python versions (I noticed it uses a custom mime type in the “execute_result” or “display_data” messages).

Notice the Low Level Widget Explanation page doesn’t explain how to connect from the web app side (or cell output side).

cheers

---

Edit: I also found the Websocket Kernel Wire Protocols pages, that specifies what goes on the websocket that presumably connects the front-end to JupyterServer. But it doesn’t say the address I should open that websocket. Or, if I should reuse the one Jupyter already has opened, how do I do it from the javascript that is the output of a cell ?

[bollwyvl]

Once it “leaves the nest,” the only thing a kernel ever does with an output is, potentially, to update it, replacing its data and metadata. It has no knowledge of where it’s going, as the other end of the ZMQ could be… anything.

comm provides a two-way pipe to talk about any thing, one of which could be an agreed-upon structure of something in a client that should change when the kernel’s thoughts of it changes, and conversely should report to the kernel that something has changed in the client. This is, in a nutshell, what “A Jupyter Widget” is, which has triggered other features such as saving a whole notebook’s worth of state in the .ipynb file.

And indeed, the HTML widget is exactly a single DOM node container, inside of which could be anything, without any particular reference to the surrounding client UI.

There isn’t a lot of docs about re-implementing some of these things, as comm is already implemented in the client, e.g. @jupyterlab/services, along with the references in the other thread to various kernels that implement a CommManager.

In that model:

• the client knows in advance it might get these kinds of messages
  • a manager is created for each soon-to-be-kernel-owning thing (in this case a NotebookPanel)
• when the kernel becomes available
  • it registers its named comm target
  • it asks for any existing client/kernel shared objects
    • if it finds any, it draws them on the page if their display object placeholders exist
  • it manages an internal state of the model, in the case of HTML just value (some raw HTML)
• when a user executes code
  • a live object is created in the kernel
    • as part of its startup machinery, it sends a comm_open
  • if the user asks to display the object
    • it sends a display_data message
    • the client draws it in the right place
    • is set up to observe (and update) the underlying model

As for reverse engineering the existing implementation:

While there is a spec for the wrapper format, the actual content inside them that corresponds to different widget classes like HTML are currently tied up in the python implementation. This broader discussion is on-going: tl;dr: it would be nice if there was a JSON schema which could be used directly by downstreams, either to do static codegen or dynamic instantation.

From live code opjects: here’s an example of instantiating live python objects, then generating backend classes. Here, “backend” means “running in a WebWorker in JS,” but it’s basically the same idea: “not python”.

From the wire: off-the-shelf tools such as wireshark would work, but one can also load up jupyterlab-kernelspy or use the browsers’ built-in websocket message viewer, which have gotten much better of late.

[Jan_Pfeifer]

Thanks again @bollwyvl, the help is very appreciated!

So while I understand the separation of the kernel to where its outputs are going to be displayed, I’ll limit myself for now to the concrete case where the display is a browser with Javascript support.

And my (still the same) question is how in Javascript to send a message back to the kernel.

The link to the comm.ts look similar to the messages I saw floating around when I connected to the JupyterServer path /api/kernels/<kernel id??>/channels using the standard Javascript Websocket, and printed out what was being sent there.

Is that the URL I need to connect a WebSocket to, in order to exchange messages with the IOPub ZeroMQ in the JupyterServer, and through it to the kernel ?

The links and description of the CommManager are interesting. I believe though once I get the “comm_*” messages flowing from/to Javascript, I can design my own CommManager to keep shared values in sync (between the Javascript widget and my Go kernel) – I’ve implemented similar protocols on distributed systems before, so I don’t foresee any difficulties (I hope at least , I may be reaching out with more questions in the future).

ps.: Javascript that I’m using to debug (most of it):

(()  => {
    var conn;
    var msg = document.getElementById("{{.MsgId}}");
    var log = document.getElementById("{{.LogId}}");
    var form = document.getElementById("{{.FormId}}");

    form.onsubmit = function () {
        if (!conn) {
            return false;
        }
        if (!msg.value) {
            return false;
        }
        conn.send(msg.value);
        msg.value = "";
        return false;
    };

    if (window["WebSocket"]) {
        conn = new WebSocket("ws://" + document.location.host + "/api/kernels/{{.KernelId}}/channels");
        conn.onclose = function (evt) {
            var item = document.createElement("div");
            item.innerHTML = "<b>Connection closed.</b>";
            appendLog(item);
        };
        conn.onmessage = function (evt) {
            const data = JSON.parse(evt.data);
            console.log(JSON.stringify(data, null, 2));
        };
    }
})();

(With the “id” fields being generated dynamically by executing it as a Go template)

[Jan_Pfeifer]

Took some time, but I think I put the pieces together – I hope they stay together With that widgets are working in my Go kernel:

For others similarly looking for how to make front-end connect to kernel/users’s cell program:

• Detailed communication documentation for the GoNB kernel – one could apply the same ideas for other languages/kernels.
• Javascript code used to connect to the kernel, which could also be reused for other kernels – I’m a noob javascript coder, but it seems to work.

[jjvraw]

Hi @Jan_Pfeifer !

I stumbled upon this thread while searching for ways to establish a communication link between a custom kernel and the front-end in Jupyter.

I’ve been experimenting with using Jupyter’s comm_manager to create a comm from the front-end like so:

var comm = Jupyter.notebook.kernel.comm_manager.new_comm('my_custom_comm');

This seems to work well for initiating the communication channel. However, when I try to send a message through this comm using comm.send({}), my kernel logs the following warnings:

[IPKernelApp] WARNING | Unknown message type: 'comm_open'
[IPKernelApp] WARNING | Unknown message type: 'comm_msg'

It seems like the kernel is not recognizing the comm_open and comm_msg message types, even though these are standard parts of the Jupyter messaging protocol.

In your exploration with custom kernels and front-end communication, have you encountered this issue or found a workaround? Any insights or guidance would be greatly appreciated.

I’m using ipykernel version 6.22.0. I noticed in the release notes for this version that there’s a deprecation warning for the Comm class, and I’m wondering if this might be related to the issue I’m experiencing.

Thank you in advance for any help!

[Jan_Pfeifer]

Odd … I never found this Jupyter object you are referring to – I cannot see it if I open the Chrome console with my notebook opened, for instance. Where do you find it ?

I ended up recreating something similar, per my earlier messages. A full description here, with many details that may interest you.

In any case, the comm_open and comm_msg are part of the protocol in between JupyterServer and the kernel, and are described as “Custom Messages”, and indeed the kernel you are using must implement it, otherwise the messages are ignored.

Also comm_open has a “target_name” associated, which I would expect is an identifier for the “sub-protocol” your kernel is using to talk front-end widgets. If you use a different one, the kernel may ignore (that’s what I did in GoNB. comm_open establishes a “connection”, which in this case is just an agreed UUID to be used in the future comm_msg messages (in the comm_id field).

I dont’ know how your kernel is using comm_open, but you probably need to investigate that to see what can be done with it. If you are willing to change the kernel, then it’s more straight forward to establish open communications with the Front-End. (It’s never “straight forward” in distributed systems … but at least in principle it’s straight forward)

G’luck!

[jjvraw]

Hi,

Apologies, I don’t think the Jupyter object is typically directly accessible from the browser’s console in a standard Jupyter Notebook setup. It is part of the internal API used by Jupyter extensions and widgets to communicate with the kernel.

You would generally need to create a Jupyter Notebook extension or a custom widget.

For example:

define([
    'base/js/namespace'
], function(
    Jupyter
) {
    function load_ipython_extension() {
        var comm_manager = Jupyter.notebook.kernel.comm_manager;
        
        // Create comm
        var comm = comm_manager.new_comm('my_custom_comm');
        
        // Set up a message handler
        comm.on_msg(function(msg) {
            console.log('Received message:', msg);
        });
        
        // Send message
        comm.send({foo: 'bar'});
    }
    
    return {
        load_ipython_extension: load_ipython_extension
    };
});

I believe loading into a notebook with such an extension will give access in console.

---

Thank you for the response! I’ll have a look into it.

[krassowski]

To be more accurate, the Jupyter object you are pointint to was a part of Jupyter Notebook in up to v6 but I think it is no longer present in Jupyter Notebook v7+ nor in other frontends. I am not sure it it was ever meant to be a public API.

[Jan_Pfeifer]

It’s still not clear to me what Jupyter makes available to the cell JS output, maybe nothing, but still I managed to create a small global object that connects back to JupyterServer → Kernel, and implement widgets without having to install something extra (an extension), which I find convenient.

Btw, what was the idea of having the user install one more “thing” (an extension) to allow widgets ? Why not just do it like I did, establish some communication with kernel API in the browser, and let it work with no extensions needed ?

cheers

[krassowski]

Btw, what was the idea of having the user install one more “thing” (an extension) to allow widgets ? Why not just do it like I did, establish some communication with kernel API in the browser, and let it work with no extensions needed ?

I would love to see this streamlined to remove any barriers. I do not know what are the arguments for status quo. Maybe no one proposed changes before?

[bollwyvl]

It’s mostly about user choice.

Users (and maintainers) of IPython, ipykernel, ipywidgets,notebook, jupyterlab, web browser, and JavaScript-in-a-browser form a complex Venn diagram. By not shipping any widget concept in the kernel, widgets were allowed to mature at their own pace after being split off from the pre-Jupyter ipython/ipython.

Indeed, this was a large motivator of The Big Split: before then, nbformat, notebook, ipywidgets, ipython, ipykernel, traitlets, etc. etc. were all contained within a single repo.

[Jan_Pfeifer]

Thanks for the context @bollwyvl , that’s interesting to know, and it makes lots of sense to split those projects into different packages.