Problem logging in to Azure AD with MFA enabled

Hi, I posted this on Github first but was asked to move it here:

We have installed Jupyterhub into our on-prem Kubernetes cluster using the Helm chart (version 1.2.0). The values file has been modified to enable HTTPS and Authenticate with Azure AD. Our AD configuration also required MFA. For some reason this triggers a server side error in the hub when a user logs in.

The config file:

hub:
  config:
    AzureAdOAuthenticator:
      client_id: "REDACTED"
      client_secret: "REDACTED"
      oauth_callback_url: "https://jupyterhub.myhost.com/hub/oauth_callback"
      tenant_id: "REDACTED"
    JupyterHub:
      authenticator_class: azuread
proxy:
  https:
    enabled: true
    hosts:
      - https://jupyterhub.myhost.com
    type: secret
    secret:
      name: jupyterhub-tls

The “sign in with Azure AD” button appears in Jupyterhub, but after logging in to AD the following error appears in the logs (and “500 : Internal Server Error” appears on the page):

[I 2022-01-06 20:25:01.505 JupyterHub oauth2:111] OAuth redirect: 'https://jupyterhub.myhost.com/hub/oauth_callback'
[I 2022-01-06 20:25:01.506 JupyterHub log:189] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://login.microsoftonline.com/$TENANT_ID/oauth2/authorize?response_type=code&redirect_uri=https%3A%2F%2Fjupyterhub.myhost.com
[E 2022-01-06 20:25:16.581 JupyterHub oauth2:389] Error fetching 400 POST https://login.microsoftonline.com/$TENAND_ID/oauth2/token: {
     "claims": "{\"access_token\":{\"capolids\":{\"essential\":true,\"values\":[\"REDACTED\"]}}}",
     "correlation_id": "REDACTED",
     "error": "interaction_required",
     "error_codes": [
      50076
     ],
     "error_description": "AADSTS50076: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '00000002-0000-0000-c000-000000000000'.\r\nTrace ID: 11ecc9d4-34b
     "error_uri": "https://login.microsoftonline.com/error?code=50076",
     "suberror": "basic_action",
     "timestamp": "2022-01-06 20:25:16Z",
     "trace_id": "REDACTED"
    }
[E 2022-01-06 20:25:16.581 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?code=REDACTED
    HTTPServerRequest(protocol='https', host='jupyterhub.myhost.com', method='GET', uri='/hub/oauth_callback?code=REDACTED
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 231, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/azuread.py", line 76, in authenticate
        resp_json = await self.fetch(req)
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 390, in fetch
        raise e
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 369, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
    tornado.httpclient.HTTPClientError: HTTP 400: Bad Request

[E 2022-01-06 20:25:16.582 JupyterHub log:181] {                                                                                                                                                                                                                    "X-Forwarded-Host": "jupyterhub.myhost.com",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Forwarded-For": "::ffff:10.17.132.1",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Site": "cross-site",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "oauthenticator-state=[secret]",
      "Connection": "close",
      "Dnt": "1",
      "Referer": "https://login.microsoftonline.com/",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0",
      "Host": "jupyterhub.myhost.com"
    }
[E 2022-01-06 20:25:16.582 JupyterHub log:189] 500 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] (@::ffff:10.17.132.1) 233.58ms

Hopefully someone in the Jupyter community here has experience with Azure OAuth and can help!

According to the “Coditional Access” logs in Azure Portal the sign-in is “successful”, so i’m not sure where the AADSTS50076 is coming from. As far as I can tell the App Registration (client id) is configured in the same way as another application for which AAD authentication works fine (ArgoCD).

I tried Googling for azure AADSTS50076 and the results suggest changing the Azure application settings, e.g.

1 Like

I had a very similar problem, and in the end it was somehow related to requesting the right OAuth scopes. I found if you leave some off, it won’t work with MFA. Here is my quote from my other “solution message”:

c.AzureAdOAuthenticator.scope = ['openid', 'profile', 'email']
('openid' is the important one, then the others as needed. By default
it used no scopes).

I’m not sure how to translate this to the helm YAML configuration.

I hope this helps. I had been meaning to post this to the issue tracker there or otherwise add it to the documentation… maybe I will do that now…

1 Like

Thanks @manics and @rkdarst.

@manics: I saw similar results in my searching. Unfortunately i’m not an AD admin so I cannot check the “security defaults” settings, and I don’t think the admins would be willing to change those settings for this use case, so if that is the issue then it’s a dead end.

But i’m hoping that there’s something more to it, since another app is working fine. The other app we have does not have the additional scopes you mention, @rkdarst .

Is there perhaps something in the OAuthenticator package that isn’t working correctly? Because having to use MFA shouldn’t be a problem in itself, as long as the app follows the flow, right?

I wouldn’t assume anything when it comes to authentication across multiple systems :smiley:

Unfortunately it’s very difficult to provide advice without access to Azure AD, and I don’t have any experience with it. I do know that some other Authentication providers have multiple layers of rules, so a login may succeed from the point of view of a user but backend permissions can result in the client being blocked.

I think @rkdarst’s suggestion is worth trying if possible- it could be that you other application has different requirements from OAuthenticator. Or maybe you could ask for a developer account from your Azure admin to play around?

I’ve got in touch with an AAD admin to help us try and debug the issue, so hopefully i’ll have an answer sometime next week :+1:

1 Like

Good luck! If you figure it out let us know, and hopefully it’ll help someone else!