Is it possible to configure what executor the PAMAuthenticator uses? I’ve tried the obvious adding c.PAMAuthenticator.executor=ThreadPoolExecutor(10) to my jupyterhub_config.py with no luck.
Context:
I’ve been debugging an issue we’ve been running into with our JupyterHub install (~100 active users at any time) where if too many users (>5, so not that many) attempt to login at the same time authentication hangs forever (or at least 5 min, I tend to get bored waiting for it after that long) the only resolution we’ve found thus far is restarting the hub.
Our pam authentication flow uses radius, and can take 30+ seconds per user, so my best guess for whats happening is these long requests are stacking up, causing a timeout somewhere to be hit, breaking authentication on our hub.
So it seems there is a bug in the code somewhere, that causes this infinite hang to never resolve, but being able to configure some level of parallelism will likely solve our problem, or at least mean I’ll have to stop restarting the hub so often.
I’d have expected c.PAMAuthenticator.executor=ThreadPoolExecutor(10) to do what you want. Can you run JupyterHub with debug logging, and show us the logs when multiple people attempt to login?
Good point! I can’t think why it couldn’t be made configurable, or at least the number of threads could be configurable.
In the meantime you could try creating an in-line subclass in your jupyterhub_config.py file, something like:
from jupyterhub.auth import PAMAuthenticator
class CustomPAMAuthenticator(PAMAuthenticator):
executor = Any() # I'm not sure if this is necessary, of if overriding `_default_executor` is enough without defining `executor` again
@default('executor')
def _default_executor(self):
return ThreadPoolExecutor(10)
c.JupyterHub.authenticator_class = CustomPAMAuthenticator
from concurrent.futures import ThreadPoolExecutor
from jupyterhub.auth import PAMAuthenticator
from traitlets import default
class CustomPAMAuthenticator(PAMAuthenticator):
@default('executor')
def _default_executor(self):
return ThreadPoolExecutor(10)
c.JupyterHub.authenticator_class = CustomPAMAuthenticator
It might just be my lack of understanding how the configuration magic works, but it looks like making executor a config option might be tricky, but adding a new executor_thread_pool_size config option to initialize it with should be trivial. Is that something that would be accepted as a PR? I’d be happy to submit it if so. I have to imagine this would be useful to more than just our site, and would have been a very helpful clue for our debugging if it had existed already.
Thank you for all the help! We’ve been working on this for a while so its frustrating to see how simple the fix was, but great to have finally found it.