Yes, it’s possible to do all kinds of exciting things with kernel messages: the trick is getting kernels to support them, and handshaking between versions of kernels/clients.
For an approach less likely to conflict with existing messages, I usually recommend investigating (if not directly using) the comm “meta” messages, which provides a documented, namespaced framework for custom interaction. This system is implemented in ipykernel (and probably all wrapper kernels), IRKernel and IJulia, and likely other kernels. Among the nice features is built-in support for binary buffers, which can provide significant performance benefits over marshalling everything as text/JSON for e.g. typed numeric arrays, video, audio, etc.
The original useful implementation of the comm infrastructure is Jupyter Widgets, which use a few named comm targets in its messages. And indeed, I would recommend thinking about wrapping whatever this feature is going to be in a widget, as then it can play nicely with all manner of other interaction pieces, while providing a useful abstraction above raw message-banging.
This is discussed at greater length on this thread. Some excerpted links: