Error while logging in to jupyter hub via SAML auth

JupyterHub Zero to JupyterHub on Kubernetes
1

Recently some of our users started getting " 500 : Internal Server Error" while logging in to hub via SAML. It’s odd that some users are able to login just fine while some are facing this issue. Could someone please explain the below events which i’m seeing from jupyterhub logs and what possible reason could be behind this:

[W 2024-06-26 15:51:26.010 JupyterHub samlauthenticator:485] Bad timing condition
[W 2024-06-26 15:51:26.010 JupyterHub samlauthenticator:487] Sent SAML Response before it was permitted
[W 2024-06-26 15:51:26.010 JupyterHub samlauthenticator:511] The SAML Assertion did not match the physical constraints
[E 2024-06-26 15:51:26.010 JupyterHub samlauthenticator:597] Error validating SAML response
[W 2024-06-26 15:51:26.010 JupyterHub base:714] Failed login for unknown user
[E 2024-06-26 15:51:26.011 JupyterHub web:1792] Uncaught exception POST /hub/login ()
HTTPServerRequest(protocol=‘https’, host='‘, method=‘POST’, uri=’/hub/login’, version=‘HTTP/1.1’, remote_ip=‘***’)
Traceback (most recent call last):
File “/usr/local/lib64/python3.6/site-packages/tornado/web.py”, line 1703, in _execute
result = await result
File “/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/login.py”, line 153, in post
login_error=‘Invalid username or password’, username=data[‘username’]
KeyError: ‘username’

[D 2024-06-26 15:51:26.011 JupyterHub base:1197] No template for 500

2

Seems like there is a clock skew between your IDP and JupyterHub. The error says that the SAML assertion is being consumed before it was allowed to consume. To my guess only when there is clock skew between your SAML IDP and JupyterHub SP.

1 Like
3

You bring an interesting point and thank you for you response. However, if that was the case wouldn’t all of the users be impacted? As I mentioned, the impact is limited to some users while others can login and spawn their nb just fine.

1 Like
4

Well, the thing is the clock on your JupyterHub server is running behind the one of IDP and the network latencies can sometimes be bigger than clock skew which will nullify this issue. Does it make sense?

1 Like
5

You were right our host machines were not running ntpd or were running with stale config. Fixing that fixed the issue. Appreciate your comments!! Thank you!!

1 Like