Jupyterhub on Kubernetes, LDAPAuthenticator used to connect via Active Directory, Error with minimal example

JupyterHub Zero to JupyterHub on Kubernetes
1

Hello Jupyter Community,

I want to connect Jupyterhub running on kubernetes (at the moment of course just on a minikube) to the Active Directory Server. I am using the newest helm chart (helm version 3.5.4) to deploy Jupyterhub (version 0.11.1).
The tutorial on Authentication and authorization - Zero to JupyterHub with Kubernetes documentation, under LDAP and Active Directory gives a nice looking minimal example I tried.
But the Result is an “Invalid Password Error”

The part in the config.yaml

hub:
  config:
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
      bind_dn_template:
      - cn={username},ou=users,ou=department,dc=institute,dc=de
      server_address: server.institute.de

The logs of the hub pod

    [I 2021-05-19 12:50:58.142 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fhub%2Flogo (@172.17.0.1) 26.51ms
    [I 2021-05-19 12:50:58.538 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fhub%2Fstatic%2Ffavicon.ico%3Fv%3Dfde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@172.17.0.1) 2.94ms
    [I 2021-05-19 12:51:10.354 JupyterHub proxy:319] Checking routes
    [W 2021-05-19 12:51:27.809 JupyterHub ldapauthenticator:398] Invalid password for user 'u.user'
    [W 2021-05-19 12:51:27.810 JupyterHub base:763] Failed login for u.user
    [I 2021-05-19 12:51:27.812 JupyterHub log:181] 200 POST /hub/login?next=%2Fhub%2F (@::ffff:172.17.0.1) 72.58ms

As I tried to look for an alternative way, which maybe works I also got a lot of other errors. I think, if even the minimal example doesn’t show a result, I don’t want to spam with the other occured errors.
If there is Interest, I can post them.

kind regard
tnecnivkcots

2

I forgot to post the Information about the helm chart version. It’s the 0.11.1.

Additionally I tried it with the helm chart version 0.11.0, but the issue is the same (it’s using the same version of jupyterhub and ldapauthenticator).

3

Unfortunately LDAP can be notoriously hard to debug as it’s so dependent on your LDAP/AD server. You could try turning on debug logging to see if that gives you more information:

debug:
  enabled: true
4

The logs of the hub pod, with debug enabled:

[D 2021-05-21 19:01:49.246 JupyterHub application:730] Looking for /etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
[D 2021-05-21 19:01:49.985 JupyterHub application:752] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2021-05-21 19:01:50.188 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Spawner: kubespawner.spawner.KubeSpawner-0.15.0
[I 2021-05-21 19:01:50.189 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-05-21 19:01:50.191 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2021-05-21 19:01:50.192 JupyterHub app:1587] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2021-05-21 19:01:50.219 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[D 2021-05-21 19:01:50.232 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[W 2021-05-21 19:01:50.241 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-05-21 19:01:50.242 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-21 19:01:50.242 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[D 2021-05-21 19:01:50.360 JupyterHub app:1877] Purging expired APITokens
[D 2021-05-21 19:01:50.366 JupyterHub app:1877] Purging expired OAuthAccessTokens
[D 2021-05-21 19:01:50.372 JupyterHub app:1877] Purging expired OAuthCodes
[D 2021-05-21 19:01:50.397 JupyterHub app:2004] Initializing spawners
[D 2021-05-21 19:01:50.400 JupyterHub app:2137] Loaded users:

[I 2021-05-21 19:01:50.400 JupyterHub app:2416] Initialized 0 spawners in 0.004 seconds
[I 2021-05-21 19:01:50.404 JupyterHub app:2628] Not starting proxy
[D 2021-05-21 19:01:50.406 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[D 2021-05-21 19:01:50.416 JupyterHub proxy:868] Omitting non-jupyterhub route '/'
[I 2021-05-21 19:01:50.417 JupyterHub app:2664] Hub API listening on http:/...
[I 2021-05-21 19:01:50.417 JupyterHub app:2666] Private Hub API connect url http:/...
[I 2021-05-21 19:01:50.417 JupyterHub app:2679] Starting managed service cull-idle
[I 2021-05-21 19:01:50.418 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http:/...', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2021-05-21 19:01:50.422 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http:/... --timeout=3600 --cull-every=600 --concurrency=10
[D 2021-05-21 19:01:50.436 JupyterHub spawner:1147] Polling subprocess every 30s
[D 2021-05-21 19:01:50.437 JupyterHub proxy:314] Fetching routes to check
[D 2021-05-21 19:01:50.437 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[D 2021-05-21 19:01:50.444 JupyterHub proxy:868] Omitting non-jupyterhub route '/'
[I 2021-05-21 19:01:50.445 JupyterHub proxy:319] Checking routes
[I 2021-05-21 19:01:50.447 JupyterHub proxy:399] Adding default route for Hub: / => http:/...
[D 2021-05-21 19:01:50.448 JupyterHub proxy:795] Proxy: Fetching POST http:/...
[I 2021-05-21 19:01:50.456 JupyterHub app:2739] JupyterHub is now running at http:/...
[D 2021-05-21 19:01:50.459 JupyterHub app:2342] It took 1.225 seconds for the Hub to start
[D 2021-05-21 19:01:50.679 JupyterHub base:283] Recording first activity for <APIToken('e234...', service='cull-idle')>
[I 2021-05-21 19:01:50.711 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 40.06ms
[D 2021-05-21 19:01:51.005 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.42ms
...
[D 2021-05-21 19:02:48.999 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.59ms
[I 2021-05-21 19:02:49.695 JupyterHub log:181] 302 GET / -> /hub/ (@::ffff:XXX.XXX.XXX.XXX) 1.72ms
[I 2021-05-21 19:02:49.841 JupyterHub log:181] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 1.73ms
[I 2021-05-21 19:02:50.000 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 82.00ms
[D 2021-05-21 19:02:50.458 JupyterHub proxy:795] Proxy: Fetching GET http:/...
[I 2021-05-21 19:02:50.477 JupyterHub proxy:319] Checking routes
[D 2021-05-21 19:02:51.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.96ms
...
[D 2021-05-21 19:02:56.999 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.86ms
[D 2021-05-21 19:02:57.574 JupyterHub log:181] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:XXX.XXX.XXX.XXX) 2.54ms
[D 2021-05-21 19:02:59.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 1.43ms
...
[D 2021-05-21 19:03:19.004 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.15ms
[D 2021-05-21 19:03:20.069 JupyterHub ldapauthenticator:379] Attempting to bind u.user with cn=u.user,u=users,ou=department,dc=institute,dc=de
[D 2021-05-21 19:03:20.105 JupyterHub ldapauthenticator:392] Status of user bind u.user with cn=u.user,u=users,ou=department,dc=institute,dc=de : False
    LDAPBindError: automatic bind not successful - invalidCredentials
[W 2021-05-21 19:03:20.105 JupyterHub ldapauthenticator:398] Invalid password for user 'u.user'
[W 2021-05-21 19:03:20.106 JupyterHub base:763] Failed login for u.user
[I 2021-05-21 19:03:20.110 JupyterHub log:181] 200 POST /hub/login?next=%2Fhub%2F (@::ffff:XXX.XXX.XXX.XXX) 42.88ms
[D 2021-05-21 19:03:21.000 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 2.24ms
...
[D 2021-05-21 19:03:41.005 JupyterHub log:181] 200 GET /hub/health (@XXX.XXX.XXX.XXX) 4.20ms

@manics Thanks for helping.

The status of user binding is the problem, right?

5

Yes, though that could be due to an incorrect configuration for connecting to your AD server. Have you spoken to your AD administrator?

6

No, not yet and not directly. But I will communicate, that the connection fails maybe due to settings the AD server.

Thanks.

7

Hey,

I talked to my supervisor, practiced in usage of ldap. We changed the conguration to

hub:
  config:
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
      bind_dn_template:
      - uid={username},ou=users,ou=department,dc=intitute,dc=de
      server_address: server.institute.de
      lookup_dn: true
      lookup_dn_search_filter: ({login_attr}={login})
      lookup_dn_search_password: secret_password
      lookup_dn_search_user: u.user
      lookup_dn_user_dn_attribute: cn
      user_attribute: sAMAccountName
      user_search_base: ou=users,ou=department,dc=institute,dc=de

Now in the browser the error is displayed “500, Internal Server Error”.
And in the logs occures:

[E 2021-05-26 07:04:49.588 JupyterHub web:1789] Uncaught exception POST /hub/login?next=%2Fhub%2F (::ffff:172.17.0.1)
    HTTPServerRequest(protocol='http', host='192.168.49.2:30132', method='POST', uri='/hub/login?next=%2Fhub%2F', version='HTTP/1.1', remote_ip='::ffff:XXX.XXX.XXX.XXX')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/login.py", line 144, in post
        user = await self.login_user(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 361, in authenticate
        username, resolved_dn = self.resolve_username(username)
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 236, in resolve_username
        conn = self.get_connection(
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 314, in get_connection
        conn = ldap3.Connection(
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 356, in __init__
        self._do_auto_bind()
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 405, in _do_auto_bind
        raise LDAPBindError(error)
    ldap3.core.exceptions.LDAPBindError: automatic bind not successful - invalidCredentials

[D 2021-05-26 07:04:49.595 JupyterHub base:1280] No template for 500
[E 2021-05-26 07:04:49.654 JupyterHub log:173] {
      "X-Forwarded-Host": "192.168.49.2:30132",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "30132",
      "X-Forwarded-For": "::ffff:XXX.XXX.XXX.XXX",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "_xsrf=[secret]",
      "Referer": "http://192.168.49.2:30132/hub/login?next=%2Fhub%2F",
      "Connection": "close",
      "Origin": "http://192.168.49.2:30132",
      "Content-Length": "40",
      "Content-Type": "application/x-www-form-urlencoded",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0",
      "Host": "192.168.49.2:30132"
    }
[E 2021-05-26 07:04:49.654 JupyterHub log:181] 500 POST /hub/login?next=%2Fhub%2F (@::ffff:172.17.0.1) 137.96ms

So the connection still fails, but not due to invalid credentials, as the error message says.

Maybe you have an idea. I would be thankful for any suggestion.

Best

8

After I looked in the code of the ldap3 package for the ldap3.connection() module, I tried to find out where the error comes from.
In the method _do_auto_bind, the value self.bound is tested in the last ìf-statement and the error is raised. I tried to get in the code block before the last if by setting the variable use_ssl to true and server_port to 389.
With this settings the error message has two additional lines:


      File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/sync.py", line 56, in open
        BaseStrategy.open(self, reset_usage, read_server_info)
      File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/base.py", line 145, in open
        raise exception_history[0][0]
    ldap3.core.exceptions.LDAPSocketOpenError: socket ssl wrapping error: [Errno 104] Connection reset by peer

Maybe it’s helpful.

9

I’m afraid I don’t have any other ideas. I think you’re investigating things in the right way though- try and connect using the low level ldap3 Python methods.

11

Hey @manics,

I solved the problem.
My configuration has to be like this:

hub:
  config:
    bind_dn_template:
    - cn={username},ou=users,ou=department,dc=institute,dc=de
    valid_username_regex: ^[A-Z][a-z]*.[A-Za-z]*$

I found out, how to handle a connection to an AD server with ldap3 and what the ldapauthenticator is doing exactly. That’s why I found this solution.

Thank you again for you help.

2 Likes
12

Thanks for sharing your solution, hopefully it’ll help others!

1 Like
13

Yesterday I found another solution. Maybe it helps others to use the ldapauthenticator with lookup_dn and using credentials in form of “f.lastname”:

   hub:
     config:
       JupyterHub:
         authenticator_class: ldapauthenticator.LDAPAuthenticator
       LDAPAuthenticator:
         valid_username_regex: ^[A-Za-z]*.[A-Za-z]*$
         server_address: server.institute.de
         allowed_groups:
           - CN=groupname,OU=groups,OU=department,DC=institute,DC=de
         bind_dn_template:
           - cn={username},OU=users,OU=department,DC=institute,DC=de
         lookup_dn: true
         lookup_dn_search_filter: ({login_attr}={login})
         lookup_dn_search_password: secret
         lookup_dn_search_user: tech.user@INSTITUTE.DE
         lookup_dn_user_dn_attribute: cn
         user_attribute: sAMAccountName
         user_search_base: OU=users,OU=department,DC=institute,DC=de
14

Hi,

Based on your requirements: “f.lastname”, your regex, while allowing this kind of user name does not look completely correct if you only want to allow that format.

With this one you allow 0 or more alphabetical letter followed by one character that can be anything and then again 0 or more alphabetical characters. Which means that usernames like “0” or “?asdf” would match.

In its python form, this one is closer to your format:
r"^[A-Za-z]\.[A-Za-z]+$"

One letter followed by a point followed by one or more letters.

If your usernames are only lowercase:
r"^[a-z]\.[a-z]+$"

1 Like