Errors with asyncio/io_loop in custom IPython kernel

Kernels
1

Hi all, I’ve been scratching my head on this for a while and thought I’d try this forum in the hopes of finding somebody else who has experience developing around the core of Jupyter.

Long story short, I’m implementing a kind of “proxy” Jupyter kernel, which extends from IPythonKernel. I have need to override the “execute_request” function and I’m doing something like this:

async def execute_request(self, stream, ident, parent):
    if not use_new_behavior:
        await super(MyKernel, self).execute_request(stream, ident, parent)
        return

    try:
        do_some_thing()
    except Exception:
        self.log.error("Caught an error")
        return

    # more logic...

What I am finding is that, if an exception is raised in do_some_thing(), the handler runs, but then the kernel dies with this error:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/ext/hydra_kernel/__main__.py", line 5, in <module>
    IPKernelApp.launch_instance(kernel_class=HydraKernel)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start
    self.io_loop.start()
  File "/opt/conda/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1875, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque

Now, I recently updated everything to the newest version of ipython, ipykernel etc, which had significant changes into how the async stuff worked (used to be Tornado coroutines, now native). So, it could very well be that I’m doing something bad as I’m not used to asyncio.

As an aside, I also was getting the same error when doing a wait loop like this inside execute_request:

while some_condition:
    await asyncio.sleep(0.1)

The errors go away if I switch to a synchronous time.sleep implementation. I tried passing in various values for the loop keyword argument in case this is a nested event loop thing but didn’t have any luck; I expected that awaiting asyncio.sleep should work here.

Can anybody advise as to why I’m seeing this behavior and what I am missing?

2

In case anybody else comes here, I believe I’ve resolved my problems.

The issue (I think) stems from using the new kernel provisioning code in combination with KernelManager and KernelClient implementations that do not extend from the Async* base classes. In my case I had been using older ThreadedIO* base classes, which use Tornado’s ioloop under the hood. I don’t know the codebase well enough to know what trickery was happening here, but cleaning up my code to only use native coroutines basically anywhere I could find seems to have made the situation more tractable.

Thanks @kevin-bates btw for the great improvement in the kernel provisioning abstraction :slight_smile:

2 Likes