Need some help with spawners

I need some help with a couple of things that I’ve been struggling with sorting out.

The first thing should be pretty easy. I have two docker images one that we’re using SystemUserSpawner to relate it to the user home directories on the system where everything is being run. The other is self-contained and doesn’t require any volume mounts to the host system. I think that I would need to implement a custom spawner that can use either SystemUserSpawner or DockerSpawner depending on which image is selected in the image picker. I haven’t been able to find a good example of how to go about implementing this though. I would be grateful for any pointers or examples that anyone knows of that does a similar thing.

This second thing is less straight forward and I’m not even sure where it might make the most sense to handle the problem. As I mentioned above, we have an image that uses SystemUserSpawner. The system itself has an automated account request process so a user that doesn’t already have access can request an account and after some amount of time they’ll be ready to work. The problem is that if a user attempts to use our Hub without an account on the host system they get stuck where they’re unable to successfully spawn a container or stop any of the phantom containers they attempted to spin up. As an admin, I can go in and see their account and these “containers” that they attempted to start but cannot stop them[1] or delete the user[2]. I made a new API token and tried to work around the Admin UI to purge the containers and users with no luck. I also tried to drop the rows of the users from the sqlite table[3] which didn’t let the user log in, but did remove their row from the Admin user server view.

The only resolution we’ve been able to figure out so far is to restart the Hub which clears up the problematic entries in the user table and drops the phantom containers from the hub/proxy. It would be great if there were some way to reset things for those users without requiring a restart. Even more ideal would be if there is some logic that could be added to the spawning process that would prevent the bad state ever getting reached to begin with.

Our production instance is running Hub version 1.4.2, but I’ve gotten 2.3.0 staged and should be deploying this week if that matters any.

Thank you so much for reading and if you have any questions please let me know.

These aren’t the exact same problem, but the errors I get in my scenario match these two:
[1] - Deleting a spawning server · Issue #2975 · jupyterhub/jupyterhub (github.com)
[2] - Deleting user who has a running server is creating a warning message · Issue #647 · jupyterhub/jupyterhub (github.com)

Sorry I could only include 2 links in my initial post :-/

[3] - As detailed in the “how to fix” section from jstaf:
Deleting OS users causes Jupyterhub server to fail to start · Issue #1060 · jupyterhub/jupyterhub (github.com)

Have a look at GitHub - jupyterhub/wrapspawner: Mechanism for runtime configuration of spawners for JupyterHub, it may work for you.

Would you mind expanding on the exact problem you need to solve? I think if you modify your spawner to return an error if the user doesn’t exist on the host system you won’t have any stale servers since they’re never started.

Sorry if I was meandering to the point. Yes exactly, I believe that would resolve the issue since as far as I’ve been able to tell there is no other way to interrupt or clear the bad state a user gets into unless you restart the Hub. I think I remember seeing a spawner function that resolves just before spawning happens, would that be the most reasonable place to check for a user on the host system?

Thanks,
-Dustin

That would be Spawner.pre_spawn_hook.

1 Like

If I return False if the check for the user account isn’t successful, would the spawn process be interrupted and everything else that might be pending start get stopped on its own, or would I also need to call stop() to terminate spawning? I think if I’m understanding the Spawner class code correctly the call to run_pre_spawn_hook function it just returns whatever is returned from pre_spawn_hook so returning False should stop the rest?

You’ll need to raise an exception in your hook. The server should be stopped: