Hi,
I’m using the PAM authenticator and I notice when I get quite a number of users on our JupyterHub, the authentication dispatch becomes unreliably slow for whatever reason. It’s seemingly inside of the JupyterHub application where it slows way down it seems. Is there some advice on tracing those calls down to figure out why it gets bogged down? I use the same PAM stack (specifically, one that implements radius) for SSH and have tested via Pamela and pamtester on the command line and each of those return near instant where as when trying to log on via JupyterHub, I don’t get my two-factor dispatch for upwards of 10-20 minutes at which point the time-based codes are invalid.
py-spy may be helpful in tracking down where JupyterHub is spending its time. This is how we’ve diagnosed many JupyterHub performance issues in the past.
How many is ‘quite a number’?
I’ll have a look at py-spy. Thanks. It’s not unusual for us to have a couple hundred sessions at any point, but yesterday was one of the first times I had this happen at around 20 users attempting to log in in roughly the same time period. Seems that at some point JupyterHub might fall behind and has trouble catching back up and ultimately a series of restarts is the only real mechanism to hopefully get back into production.