Jupyterhub on Kubernetes, LDAPAuthenticator used to connect via Active Directory, Error with minimal example

Hello Jupyter Community,

I want to connect Jupyterhub running on kubernetes (at the moment of course just on a minikube) to the Active Directory Server. I am using the newest helm chart (helm version 3.5.4) to deploy Jupyterhub (version 0.11.1).
The tutorial on Authentication and authorization - Zero to JupyterHub with Kubernetes documentation, under LDAP and Active Directory gives a nice looking minimal example I tried.
But the Result is an “Invalid Password Error”

The part in the config.yaml:
hub:
config:
JupyterHub:
authenticator_class: ldapauthenticator.LDAPAuthenticator
LDAPAuthenticator:
bind_dn_template:
- cn={username},ou=users,ou=department,dc=institute,dc=de
server_address: server.institute.de

The logs of the hub pod:
[I 2021-05-19 12:50:58.142 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fhub%2Flogo (@172.17.0.1) 26.51ms
[I 2021-05-19 12:50:58.538 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fhub%2Fstatic%2Ffavicon.ico%3Fv%3Dfde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@172.17.0.1) 2.94ms
[I 2021-05-19 12:51:10.354 JupyterHub proxy:319] Checking routes
[W 2021-05-19 12:51:27.809 JupyterHub ldapauthenticator:398] Invalid password for user ‘u.user’
[W 2021-05-19 12:51:27.810 JupyterHub base:763] Failed login for u.user
[I 2021-05-19 12:51:27.812 JupyterHub log:181] 200 POST /hub/login?next=%2Fhub%2F (@::ffff:172.17.0.1) 72.58ms

As I tried to look for an alternative way, which maybe works I also got a lot of other errors. I think, if even the minimal example doesn’t show a result, I don’t want to spam with the other occured errors.
If there is Interest, I can post them.

kind regard
tnecnivkcots

I forgot to post the Information about the helm chart version. It’s the 0.11.1.

Additionally I tried it with the helm chart version 0.11.0, but the issue is the same (it’s using the same version of jupyterhub and ldapauthenticator).

Unfortunately LDAP can be notoriously hard to debug as it’s so dependent on your LDAP/AD server. You could try turning on debug logging to see if that gives you more information:

debug:
  enabled: true

The logs of the hub pod, with debug enabled:

[D 2021-05-21 19:01:49.246 JupyterHub application:730] Looking for /etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
[D 2021-05-21 19:01:49.985 JupyterHub application:752] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2021-05-21 19:01:50.188 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Spawner: kubespawner.spawner.KubeSpawner-0.15.0
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-05-21 19:01:50.191 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2021-05-21 19:01:50.192 JupyterHub app:1587] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2021-05-21 19:01:50.219 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[D 2021-05-21 19:01:50.232 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[W 2021-05-21 19:01:50.241 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-05-21 19:01:50.242 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-21 19:01:50.242 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[D 2021-05-21 19:01:50.360 JupyterHub app:1877] Purging expired APITokens
[D 2021-05-21 19:01:50.366 JupyterHub app:1877] Purging expired OAuthAccessTokens
[D 2021-05-21 19:01:50.372 JupyterHub app:1877] Purging expired OAuthCodes
[D 2021-05-21 19:01:50.397 JupyterHub app:2004] Initializing spawners
[D 2021-05-21 19:01:50.400 JupyterHub app:2137] Loaded users:

[I 2021-05-21 19:01:50.400 JupyterHub app:2416] Initialized 0 spawners in 0.004 seconds
[I 2021-05-21 19:01:50.404 JupyterHub app:2628] Not starting proxy
[D 2021-05-21 19:01:50.406 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[D 2021-05-21 19:01:50.416 JupyterHub proxy:868] Omitting non-jupyterhub route '/'
[I 2021-05-21 19:01:50.417 JupyterHub app:2664] Hub API listening on http:/...
[I 2021-05-21 19:01:50.417 JupyterHub app:2666] Private Hub API connect url http:/...
[I 2021-05-21 19:01:50.417 JupyterHub app:2679] Starting managed service cull-idle
[I 2021-05-21 19:01:50.418 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http:/...', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2021-05-21 19:01:50.422 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http:/... --timeout=3600 --cull-every=600 --concurrency=10
[D 2021-05-21 19:01:50.436 JupyterHub spawner:1147] Polling subprocess every 30s
[D 2021-05-21 19:01:50.437 JupyterHub proxy:314] Fetching routes to check
[D 2021-05-21 19:01:50.437 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[D 2021-05-21 19:01:50.444 JupyterHub proxy:868] Omitting non-jupyterhub route '/'
[I 2021-05-21 19:01:50.445 JupyterHub proxy:319] Checking routes
[I 2021-05-21 19:01:50.447 JupyterHub proxy:399] Adding default route for Hub: / => http:/...
[D 2021-05-21 19:01:50.448 JupyterHub proxy:795] Proxy: Fetching POST http:/...
[I 2021-05-21 19:01:50.456 JupyterHub app:2739] JupyterHub is now running at http:/...
[D 2021-05-21 19:01:50.459 JupyterHub app:2342] It took 1.225 seconds for the Hub to start
[D 2021-05-21 19:01:50.679 JupyterHub base:283] Recording first activity for <APIToken('e234...', service='cull-idle')>
[I 2021-05-21 19:01:50.711 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 40.06ms
[D 2021-05-21 19:01:51.005 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.42ms
...
[D 2021-05-21 19:02:48.999 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.59ms
[I 2021-05-21 19:02:49.695 JupyterHub log:181] 302 GET / -> /hub/ (@::ffff:XXX.XXX.XXX.XXX) 1.72ms
[I 2021-05-21 19:02:49.841 JupyterHub log:181] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 1.73ms
[I 2021-05-21 19:02:50.000 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 82.00ms
[D 2021-05-21 19:02:50.458 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[I 2021-05-21 19:02:50.477 JupyterHub proxy:319] Checking routes
[D 2021-05-21 19:02:51.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.96ms
...
[D 2021-05-21 19:02:56.999 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.86ms
[D 2021-05-21 19:02:57.574 JupyterHub log:181] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:XXX.XXX.XXX.XXX) 2.54ms
[D 2021-05-21 19:02:59.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.43ms
...
[D 2021-05-21 19:03:19.004 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.15ms
[D 2021-05-21 19:03:20.069 JupyterHub ldapauthenticator:379] Attempting to bind u.user with cn=u.user,u=users,ou=department,dc=institute,dc=de
[D 2021-05-21 19:03:20.105 JupyterHub ldapauthenticator:392] Status of user bind u.user with cn=u.user,u=users,ou=department,dc=institute,dc=de : False
    LDAPBindError: automatic bind not successful - invalidCredentials
[W 2021-05-21 19:03:20.105 JupyterHub ldapauthenticator:398] Invalid password for user 'u.user'
[W 2021-05-21 19:03:20.106 JupyterHub base:763] Failed login for u.user
[I 2021-05-21 19:03:20.110 JupyterHub log:181] 200 POST /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 42.88ms
[D 2021-05-21 19:03:21.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 2.24ms
...
[D 2021-05-21 19:03:41.005 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.20ms

@manics Thanks for helping.

The status of user binding is the problem, right?

Yes, though that could be due to an incorrect configuration for connecting to your AD server. Have you spoken to your AD administrator?

No, not yet and not directly. But I will communicate, that the connection fails maybe due to settings the AD server.

Thanks.

Hey,

I talked to my supervisor, practiced in usage of ldap. We changed the conguration to

hub:
  config:
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
      bind_dn_template:
      - uid={username},ou=users,ou=department,dc=intitute,dc=de
      server_address: server.institute.de
      lookup_dn: true
      lookup_dn_search_filter: ({login_attr}={login})
      lookup_dn_search_password: secret_password
      lookup_dn_search_user: u.user
      lookup_dn_user_dn_attribute: cn
      user_attribute: sAMAccountName
      user_search_base: ou=users,ou=department,dc=institute,dc=de

Now in the browser the error is displayed “500, Internal Server Error”.
And in the logs occures:

[E 2021-05-26 07:04:49.588 JupyterHub web:1789] Uncaught exception POST /hub/login?next=%2Fhub%2F (::ffff:172.17.0.1)
    HTTPServerRequest(protocol='http', host='192.168.49.2:30132', method='POST', uri='/hub/login?next=%2Fhub%2F', version='HTTP/1.1', remote_ip='::ffff:XXX.XXX.XXX.XXX')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/login.py", line 144, in post
        user = await self.login_user(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 361, in authenticate
        username, resolved_dn = self.resolve_username(username)
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 236, in resolve_username
        conn = self.get_connection(
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 314, in get_connection
        conn = ldap3.Connection(
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 356, in __init__
        self._do_auto_bind()
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 405, in _do_auto_bind
        raise LDAPBindError(error)
    ldap3.core.exceptions.LDAPBindError: automatic bind not successful - invalidCredentials

[D 2021-05-26 07:04:49.595 JupyterHub base:1280] No template for 500
[E 2021-05-26 07:04:49.654 JupyterHub log:173] {
      "X-Forwarded-Host": "192.168.49.2:30132",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "30132",
      "X-Forwarded-For": "::ffff:XXX.XXX.XXX.XXX",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "_xsrf=[secret]",
      "Referer": "http://192.168.49.2:30132/hub/login?next=%2Fhub%2F",
      "Connection": "close",
      "Origin": "http://192.168.49.2:30132",
      "Content-Length": "40",
      "Content-Type": "application/x-www-form-urlencoded",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0",
      "Host": "192.168.49.2:30132"
    }
[E 2021-05-26 07:04:49.654 JupyterHub log:181] 500 POST /hub/login?next=%2Fhub%2F (@::ffff:172.17.0.1) 137.96ms

So the connection still fails, but not due to invalid credentials, as the error message says.

Maybe you have an idea. I would be thankful for any suggestion.

Best

After I looked in the code of the ldap3 package for the ldap3.connection() module, I tried to find out where the error comes from.
In the method _do_auto_bind, the value self.bound is tested in the last ìf-statement and the error is raised. I tried to get in the code block before the last if by setting the variable use_ssl to true and server_port to 389.
With this settings the error message has two additional lines:


      File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/sync.py", line 56, in open
        BaseStrategy.open(self, reset_usage, read_server_info)
      File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/base.py", line 145, in open
        raise exception_history[0][0]
    ldap3.core.exceptions.LDAPSocketOpenError: socket ssl wrapping error: [Errno 104] Connection reset by peer

Maybe it’s helpful.

I’m afraid I don’t have any other ideas. I think you’re investigating things in the right way though- try and connect using the low level ldap3 Python methods.

Thank you for trying to help.

I tried to make a ldapsearch on the command line, from the minikube and the ldapsearch (command: ldapsearch simple bind -H ldap://mpiedc1.mpie.de -b ou=Server,ou=Computers,ou=department,dc=institute,dc=de) worked, so I estimate there could be a problem in the used ldap3 or the ldapauthenticator class for jupyterhub authentication.

I’ll report back, if I get new results and maybe have solutions for this problem.

1 Like

Hey @manics,

I solved the problem.
My configuration has to be like this:

hub:
  config:
    bind_dn_template:
    - cn={username},ou=users,ou=department,dc=institute,dc=de
    valid_username_regex: ^[A-Z][a-z]*.[A-Za-z]*$ 

I found out, how to handle a connection to an AD server with ldap3 and what the ldapauthenticator is doing exactly. That’s why I found this solution.

Thank you again for you help.

1 Like

Thanks for sharing your solution, hopefully it’ll help others!