The big change since last time: pyomq switched from the Compio backend (io_uring, mostly Linux-only) to Tokio, which is much more mature and should allow for better cross-platform support. By default it uses Tokio’s current_thread runtime flavor, which avoids thread contention within the OMQ machinery. If io_threads>1 is specified during Context creation, it’ll use the multi_thread flavor (when the IO perf is really needed).
zstd+tcp:// transport removed. lz4+tcp:// performs amazingly well. Simpler. Will probably remove it from the open ZMQ RFC PR.
CURVE + BLAKE3ZMQ (experimental) authentication with per-socket allow-lists or Python callback authenticators (no ZAP).
Here the perf chart with the defaults (PUSH/PULL throughput + REQ/REP latency, 2-process over TCP loopback):
@MatthieuDartiailh is working on Windows compatibility (PRs #87, #94, …). Windows pipes: Now with Tokio those should be a possibility soon. Matthieu can handle that hopefully.
I’ve also tried running Jupyter Client’s test suite against pyomq. 239/252 pass. 1 fail (no ifconfig on my Linux VM), 2 skips, 10 hangs (kernel restart lifecycle tests). I’m currently not looking further into it.
Regarding CURVE: So far I’ve decided against ZAP because IMHO that’s mostly useful in C. Both on the Rust side and on the pyomq side you can either pass in a list of authorized client pubkeys or a lambda authenticator.
I’ve been publishing 4 Linux wheel flavors using GitHub’s maturin-action: manylinux and musllinux, each for x86_64 and aarch64. Haven’t looked into cibuildwheel but I’m keeping it in mind.
If you’re interested and have a use case for it, give it a try. ZMQ is amazing.
Tbh, I don’t know what exactly is open for that PR. Could you link to something specific?
CURVE support is implemented. No libsodium. Instead, it uses crypto_box and crypto_secretbox. No ZAP because there are nicer ways to authenticate clients imho. From the CURVE example in the pyomq README:
pyomq.has("curve") # True
server_pub, server_sec = zmq.curve_keypair()
client_pub, client_sec = zmq.curve_keypair()
pull = ctx.socket(zmq.PULL)
pull.curve_server = 1
pull.curve_publickey = server_pub
pull.curve_secretkey = server_sec
# Option 1: allow specific client keys (checked in Rust, no GIL overhead)
pull.set_curve_auth([client_pub])
# Option 2: custom callback receiving a PeerInfo with a .public_key (Z85 bytes)
pull.set_curve_auth(lambda peer: peer.public_key in allowed_keys)
# Option 3: accept any valid CURVE client (the default)
pull.set_curve_auth(None)
I think your proposal includes key exchange over the kernel’s registration socket?
If you’re looking for better throughput (almost 2x), you may consider BLAKE3ZMQ. But it should probably not be used for anything that matters, as it has not been independently audited. Same API:
No, nothing specific I was just curious if there are any feature parity considerations to be aware of. It sounds like everything could be swapped one to one.
If you’re looking for better throughput (almost 2x), you may consider BLAKE3ZMQ.
Oh, that’s cool, thank you for sharing!
But it should probably not be used for anything that matters, as it has not been independently audited
I see. I think once it matures, we could add it in jupyter-client as a known, faster alternative to curve.
