Problem logging in to Azure AD with MFA enabled

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi, I posted this on Github first but was asked to move it here:

We have installed Jupyterhub into our on-prem Kubernetes cluster using the Helm chart (version 1.2.0). The values file has been modified to enable HTTPS and Authenticate with Azure AD. Our AD configuration also required MFA. For some reason this triggers a server side error in the hub when a user logs in.

The config file:

hub:
  config:
    AzureAdOAuthenticator:
      client_id: "REDACTED"
      client_secret: "REDACTED"
      oauth_callback_url: "https://jupyterhub.myhost.com/hub/oauth_callback"
      tenant_id: "REDACTED"
    JupyterHub:
      authenticator_class: azuread
proxy:
  https:
    enabled: true
    hosts:
      - https://jupyterhub.myhost.com
    type: secret
    secret:
      name: jupyterhub-tls

The “sign in with Azure AD” button appears in Jupyterhub, but after logging in to AD the following error appears in the logs (and “500 : Internal Server Error” appears on the page):

[I 2022-01-06 20:25:01.505 JupyterHub oauth2:111] OAuth redirect: 'https://jupyterhub.myhost.com/hub/oauth_callback'
[I 2022-01-06 20:25:01.506 JupyterHub log:189] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://login.microsoftonline.com/$TENANT_ID/oauth2/authorize?response_type=code&redirect_uri=https%3A%2F%2Fjupyterhub.myhost.com
[E 2022-01-06 20:25:16.581 JupyterHub oauth2:389] Error fetching 400 POST https://login.microsoftonline.com/$TENAND_ID/oauth2/token: {
     "claims": "{\"access_token\":{\"capolids\":{\"essential\":true,\"values\":[\"REDACTED\"]}}}",
     "correlation_id": "REDACTED",
     "error": "interaction_required",
     "error_codes": [
      50076
     ],
     "error_description": "AADSTS50076: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '00000002-0000-0000-c000-000000000000'.\r\nTrace ID: 11ecc9d4-34b
     "error_uri": "https://login.microsoftonline.com/error?code=50076",
     "suberror": "basic_action",
     "timestamp": "2022-01-06 20:25:16Z",
     "trace_id": "REDACTED"
    }
[E 2022-01-06 20:25:16.581 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?code=REDACTED
    HTTPServerRequest(protocol='https', host='jupyterhub.myhost.com', method='GET', uri='/hub/oauth_callback?code=REDACTED
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 231, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/azuread.py", line 76, in authenticate
        resp_json = await self.fetch(req)
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 390, in fetch
        raise e
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 369, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
    tornado.httpclient.HTTPClientError: HTTP 400: Bad Request

[E 2022-01-06 20:25:16.582 JupyterHub log:181] {                                                                                                                                                                                                                    "X-Forwarded-Host": "jupyterhub.myhost.com",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Forwarded-For": "::ffff:10.17.132.1",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Site": "cross-site",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "oauthenticator-state=[secret]",
      "Connection": "close",
      "Dnt": "1",
      "Referer": "https://login.microsoftonline.com/",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0",
      "Host": "jupyterhub.myhost.com"
    }
[E 2022-01-06 20:25:16.582 JupyterHub log:189] 500 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] (@::ffff:10.17.132.1) 233.58ms
2

Hopefully someone in the Jupyter community here has experience with Azure OAuth and can help!

3

According to the “Coditional Access” logs in Azure Portal the sign-in is “successful”, so i’m not sure where the AADSTS50076 is coming from. As far as I can tell the App Registration (client id) is configured in the same way as another application for which AAD authentication works fine (ArgoCD).

4

I tried Googling for azure AADSTS50076 and the results suggest changing the Azure application settings, e.g.

1 Like
5

I had a very similar problem, and in the end it was somehow related to requesting the right OAuth scopes. I found if you leave some off, it won’t work with MFA. Here is my quote from my other “solution message”:

c.AzureAdOAuthenticator.scope = ['openid', 'profile', 'email']
('openid' is the important one, then the others as needed. By default
it used no scopes).

I’m not sure how to translate this to the helm YAML configuration.

I hope this helps. I had been meaning to post this to the issue tracker there or otherwise add it to the documentation… maybe I will do that now…

1 Like
6

Thanks @manics and @rkdarst.

@manics: I saw similar results in my searching. Unfortunately i’m not an AD admin so I cannot check the “security defaults” settings, and I don’t think the admins would be willing to change those settings for this use case, so if that is the issue then it’s a dead end.

But i’m hoping that there’s something more to it, since another app is working fine. The other app we have does not have the additional scopes you mention, @rkdarst .

Is there perhaps something in the OAuthenticator package that isn’t working correctly? Because having to use MFA shouldn’t be a problem in itself, as long as the app follows the flow, right?

7

I wouldn’t assume anything when it comes to authentication across multiple systems :smiley:

Unfortunately it’s very difficult to provide advice without access to Azure AD, and I don’t have any experience with it. I do know that some other Authentication providers have multiple layers of rules, so a login may succeed from the point of view of a user but backend permissions can result in the client being blocked.

I think @rkdarst’s suggestion is worth trying if possible- it could be that you other application has different requirements from OAuthenticator. Or maybe you could ask for a developer account from your Azure admin to play around?

8

I’ve got in touch with an AAD admin to help us try and debug the issue, so hopefully i’ll have an answer sometime next week :+1:

1 Like
9

Good luck! If you figure it out let us know, and hopefully it’ll help someone else!

10

Just to rule out the obvious first: log line 2 and 3 have different values $TENANT_ID and $TENAND_ID. That’s just your editing right? The real log entries contains the value of the $TENANT_ID variable right?

We have been running AAD auth with MFA for a while now, but we use full vms in Azure instead for containers. (we want to have shared servers which are joined into the AD domain)
Here is the ansible block we use for configuration it seems to be more or less the same info as you are providing apart from the username_claim config line. I do not know if this matters or not, but in our setup we needed it to be able to match the local user id we get from AD to the info coming from AAD.

- name: Setup jupyterhub AAD authentication
      blockinfile:
        path: /etc/jupyterhub/jupyterhub_config.py
        marker: "#{mark} AAD LOGIN (ANSIBLE MANAGED BLOCK)"
        block: |
          import os
          from oauthenticator.azuread import AzureAdOAuthenticator
          c.JupyterHub.authenticator_class = AzureAdOAuthenticator
          c.Application.log_level = 'DEBUG'
          c.AzureAdOAuthenticator.tenant_id = os.environ.get('AAD_TENANT_ID')
          c.AzureAdOAuthenticator.oauth_callback_url = '{{ aad_oauth_callback_url }}'
          c.AzureAdOAuthenticator.client_id = '{{ aad_client_id }}'
          c.AzureAdOAuthenticator.client_secret = '{{ aad_client_secret }}'
          c.AzureAdOAuthenticator.username_claim="preferred_username"
1 Like
11

Whoops, yes, TENANT vs TENAND was just a typo during my editing.

Thanks for sharing your experience. I’ve spoken to the AAD admin who says that they have disabled the MFA requirement for this app, but the same error is still appearing!

I wonder if it has something to do with the fact that we currently have a self-signed certificate? But that doesn’t make sense.

12

Wowwww ok, problem solved! Turns out the problem was simply because my browser was not trusted, because it’s a bit of a PITA to get my usual browser (Chrome) to trust a self-signed certificate. So i was using Firefox, for which i had never logged into AAD before. So THAT is what the message was trying to tell me. Once I opened portal.office.com and signed in there then AAD trusted Firefox, and then I was able to log in to Jupyterhub. It’s a bit annoying that that message wasn’t more clearly visible in the logs i reviewed in Azure portal.

I have now also jumped through the hooks to get Chrome to trust my self-signed cert on this Mac, and there it works fine. So if i had done that in the first place then I wouldn’t have noticed!

Hope this helps someone in the future!

2 Likes
13

Thanks for sharing… the error message really wasn’t helpful!

14

OK, happy to see you got it working. We do ssl termination in the web application firewall (WAF) which runs in the front of the cluster so I didn’t think of that.

15

Chiming in to agree with @rkdarst , requesting OpenID scopes fixed the MFA issue for us. We just added the following values (in bold) to our Helm values.yaml:
jupyterhub:
hub:
config:
JupyterHub:
authenticator_class: azuread
AzureAdOAuthenticator:
tenant_id:
scope:
- openid
- email