I have Nextcloud configured as an Oauth2 identity provider, allowing my JupyterHub users to authenticate against their Nextcloud accounts using the GenericOauthenticator using a configuration like this one. This system works smoothly for a number of users as they log in, but then inexplicably I start receiving this error on the JupyterHub login page:
500 : Internal Server Error
This can even happen for the same Nextcloud accounts that authenticated properly just minutes before. The Nextcloud log only says something like:
{"reqId":"C3D8JoWxk3Lw6atnul7Y","level":2,"time":"2021-07-04T01:01:03+00:00","remoteAddr":"141.142.216.48","user":"--","app":"core","method":"POST","url":"/apps/oauth2/api/v1/token","message":"Login failed: 'C5UWHxTlbk7RwvDhsPoL8oZ640mORcJq5po6H4s9zyMy9xyOgoLOL0e66VZqUDKp' (Remote IP: '1.2.3.4')","userAgent":"JupyterHub","version":"21.0.2.1"}
The JupyterHub log shows this:
result = await result
File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 224, in get
user = await self.login_user()
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
authenticated = await self.authenticate(data)
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
authenticated = await maybe_future(self.authenticate(handler, data))
File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 155, in authenticate
token_resp_json = await self._get_token(http_client, headers, params)
File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 97, in _get_token
resp = await http_client.fetch(req)
tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20000 milliseconds with 0 bytes received
[E 2021-07-04 01:00:58.488 JupyterHub log:173] {
"X-Real-Ip": "2.3.4.5",
"X-Forwarded-Server": "traefik-5684766988-sj49c",
"X-Forwarded-Proto": "https,http",
"X-Forwarded-Port": "443,80",
"X-Forwarded-Host": "jupyter.example.com",
"X-Forwarded-For": "****",
"Upgrade-Insecure-Requests": "1",
"Te": "trailers",
"Sec-Gpc": "1",
"Dnt": "1",
"Cookie": "_fbp=[secret]; OptanonConsent=[secret]; OptanonAlertBoxClosed=[secret]; _ga_F6CG3GSQTQ=[secret]; _ga=[secret]; _xsrf=[secret]; oauthenticator-state=[secret]",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
"Host": "jupyter.example.com",
"Connection": "close"
}
I have not changed any configuration between when the auth works and when it fails, making this problem particularly challenging to diagnose. Any ideas from the JupyterHub community would be appreciated.
I’m clutching at straws here…