An experimental graph plotting library, optimized for speed

General Show and Tell
1

Hi all,

Hope this is the right forum. :slight_smile: I recently built a small graph plotting library, which is still very early stage but highly optimized for speed, so I thought people here might like it.

I normally use plotly for drawing graphs, which is a very nice library with tons of features, but its performance could be… better. I didn’t like how the browser would slow down with large data. So, I thought, how far can I push it if I build it myself?

In the end, I could comfortably plot 10K+ lines with 100M+ points in total, fully interactive. You can zoom in, pan, highlight the line under the cursor, or select a subset of lines by label names (regex is supported) and instantly update the graph.

The library is here: GitHub - yongjik/croquis: Python plotting library for Jupyter Notebook

image
image924Ă—473 126 KB

I named it “croquis” because it’s all about drawing stuff really, really fast. :slight_smile: Please tell me what you think, and especially, what features you’d want to see before you would find it useful!

2

Thanks for sharing! This seems really impressive!

I took a look at your code. It looks like you send your image data over comm messages, but as JSON? Have you tried sending the binary parts like the png data using comm buffers, which triggers sending the comm message as a binary websocket message rather than a text websocket message? We’ve seen big speedups in transferring plotting data when we switched to using binary comm messages (i.e., sending comm messages with binary buffers).

3

Hi @jasongrout, thanks for taking a look!

I think I am using binary transfer for images. In CommManager.send (in src/croquis/comm.py), we call the underlying ipykernel.comm using buffers argument, which contains the image data as a binary buffer. So, the resulting message will contain metadata in JSON format plus actual image data in binary format.

Or at least that’s how I understand it… I’m not an expert on Jupyter messaging internals. :slight_smile:

What I found was that the bottleneck is the browser trying to update the “canvas” with new images every time the mouse moves. (When the mouse moves, croquis looks at the mouse coordinate, decides which line is under the cursor, and then asks the backend to create an image of just that line, so that we can use it as “highlight” on top of the existing image. Admittedly, that’s a lot of processing, when the mouse keeps moving.) I think I could have better performance with larger tiles (because then the browser will need to update fewer number of images), but I didn’t have time to try it out yet.