[Spawner / ProgressAPI] only flushing events once the server is ready

Hi all !

I have been deploying hubs with custom spawners for years (3 at least) and I never got the event log / progress bar to work properly : events are flushed to the browser in one go when the single user server is ready.
Our custom spawner does not implement progress() method yet, so only the 2 default event messages from the main Spawner class are sent.

Versions used:
jupyterhub==2.3.1 (same behavior with previous versions and also with 3.0.0)
tornado==6.2 (same behavior with previous versions)

I found other mentions of similar behavior… but none is providing anything that looks like a working solution.

I opened an issue on github (Spawner / ProgressAPI only flushing events once the server is ready · Issue #4043 · jupyterhub/jupyterhub · GitHub) ; but it looks like I am the only one with this issue - so probably something I’m doing wrong.

Can anyone point me to the right documentation on implementing progress() in our custom spawner ?
I checked Starting servers with the JupyterHub API — JupyterHub 3.0.0 documentation but I did not find what I was looking for.

Thanks !

Is your custom Spawner available somewhere to look at? The issue may be in start (or any other call) blocking somewhere instead of letting async things proceed. The progress events will only be delivered if all pending operations are in an await step, because asyncio is ‘cooperative’.

Hi @minrk, thank you for this info, it looks promising.

If I understand correctly : our custom start() (or some other method) is not being cooperative and does not let other asynchronous jobs (including progress events) the chance to run.

I’m going to dig into that. Our spawner is not (yet) available in a public repo - still not sure if it would make sense since it is very specific to our deployment. Thanks again for pointing me into that direction.

Hi again!

I checked and our start() method is basically defined like this:

    async def start(self):
        # build ssh command and send it to the remote host
        for i in range(self.start_timeout):
            is_up = await self.poll()
            if is_up is None:
                return (self.ip, self.port)
            await asyncio.sleep(1)
        return None

… so it definitively makes use of await.

I also checked the poll() method, and it’s a non blocking method returning almost immediately.
Still, our poll() method does not follow the rules from (jupyterhub/spawner.py at 38afbcc0d067863d9690328dc41a24cd7487e62b · jupyterhub/jupyterhub · GitHub) - it never returns 0.
I’ll make it comply and see it that changes something.