Failing to authenticate against Auth0 with 'stream closed' exception

JupyterHub
1

Hello, I am hoping someone will be able to help. Today our JupyterHub server began responding with 500’s to all login attempts using Auth0 OAuth. This was working as of a few days ago and no changes have been made to the server. Looking at the longs we are getting the trace as

Traceback (most recent call last):
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:       File "/opt/jupyterhub/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:         result = await result
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:       File "/opt/jupyterhub/lib/python3.6/site-packages/oauthenticator/oauth2.py", line 213, in get
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:         user = await self.login_user()
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:       File "/opt/jupyterhub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 699, in login_user
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:         authenticated = await self.authenticate(data)
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:       File "/opt/jupyterhub/lib/python3.6/site-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:         authenticated = await maybe_future(self.authenticate(handler, data))
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:       File "/opt/jupyterhub/lib/python3.6/site-packages/oauthenticator/auth0.py", line 90, in authenticate
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:         resp = await http_client.fetch(req)
Nov 12 22:33:50 ip-172-31-30-184 jupyterhub[3343]:     tornado.simple_httpclient.HTTPStreamClosedError: Stream closed

I have tried upgrading JupyterHub to latest. I have also tried forcing different version of Tornado, including version 5.1.1 which I found many had luck with when encountering this problem in the past. However, I have not had any luck with any of this.

Auth0 logs are showing that the authentication was successful. This issue seems to be in Tornado, but I am not 100% sure of that.

Thank you in advance.
Josh Rogers

2

I was able to get this working by switching to the curl_httpclient configuration for Tornado’s AsyncHTTPClient. Steps I took were as follows.

  1. Modify /opt/jupyterhub/lib/python3.6/site-packages/oauthenticator/auth0.py
    to replace line 71 with
    AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
  2. From the CLI ran
    python3 -m venv /opt/jupyterhub/bin/activate
  3. From the CLI ran
    pip3 install pycurl
  4. From the CLI ran
    sudo service jupyterhub restart

These steps allowed me to begin fully authenticating with Auth0 again and users are now able to work in the system.

3

After further discovery here: Implement the curl_httpclient · Issue #385 · jupyterhub/oauthenticator · GitHub

It was determined that simply installing pycurl causes the switch to the Curl HTTP client and the code change was unnecessary.

1 Like