LDAP Not working: LDAPStartTLSError

Hi!

Trying to setup a new Z2JH kubernetes cluster updated to the latest versions of jupyter hub and jupyterlab. But LDAP authentication appears to be giving me some grief.

Additional details will be below … but the essential issue appears to be that whenever a login attempt is made, a 500 error is returned and the hub pod (eg hub-7f5fb6d49f-wzj2t) reports an ldap3.core.exceptions.LDAPStartTLSError: ('wrap socket error: EOF occurred in violation of protocol (_ssl.c:1131)',) error.

A more complete log extract is below.

I’m not well versed in LDAP, but my initial searches online didn’t turn up any apparently relevant help.

If anyone here has any advice it would be most welcome, even it involves a nicer alternative to LDAP for flexible authentication.

Details

  • Jupyter Version: 1.4.2 / Helm Chart Version: 1.1.3
  • LDAP server: geek-cookbook/openldap with mostly default configuration (it runs the osixia docker image).
    • I have confirmed that the LDAP server is running appropriately by successfully interacting with it through Apache Directory Studio and ldap CLI tools over a kubectl port-forward.

config.yaml

proxy:
  https:
    enabled: true
    hosts:
      - DOMAIN
    letsencrypt:
      contactEmail: EMAIL
hub:
  config:
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
      use_ssl: true
      server_address: ldap-proto-openldap.default.svc.cluster.local
      bind_dn_template:
        - 'mail={username},o=DELETED,ou=DELETED,dc=DELETED,dc=com'
        - 'mail={username},o=DELETED,ou=DELETED,dc=DELETED,dc=com'
      escape_userdn: true
      lookup_dn: false
      valid_username_regex: '^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$'
  defaultUrl: "/lab"

Log Extract from hub Pod for when login attempted

[E 2021-09-29 09:52:51.564 JupyterHub web:1789] Uncaught exception POST /hub/login?next=%2Fhub%2F (::ffff:192.168.34.48)
    HTTPServerRequest(protocol='http', host='aac510edbf3624e84a65b2a742db10b8-2112802575.ap-southeast-2.elb.amazonaws.com', method='POST', uri='/hub/login?next=%2Fhub%2F', version='HTTP/1.1', remote_ip='::ffff:192.168.34.48')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/login.py", line 151, in post
        user = await self.login_user(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 382, in authenticate
        conn = self.get_connection(userdn, password)
      File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 314, in get_connection
        conn = ldap3.Connection(
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 363, in __init__
        self._do_auto_bind()
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 391, in _do_auto_bind
        if self.start_tls(read_server_info=False):
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 1314, in start_tls
        if self.server.tls.start_tls(self) and self.strategy.sync:  # for asynchronous connections _start_tls is run by the strategy
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/tls.py", line 280, in start_tls
        return self._start_tls(connection)
      File "/usr/local/lib/python3.8/dist-packages/ldap3/core/tls.py", line 289, in _start_tls
        raise start_tls_exception_factory(e)(connection.last_error)
    ldap3.core.exceptions.LDAPStartTLSError: ('wrap socket error: EOF occurred in violation of protocol (_ssl.c:1131)',)

I also have issues with this, but I think it’s because the newer versions of the ldapauthenticator do not support really support the use_ssl: true option. This is based off of my own experience with using it, and this discussion: v1.3.0 worked, but v1.3.2 gives error "automatic start_tls befored bind not successful" · Issue #186 · jupyterhub/ldapauthenticator · GitHub

As a workaround, I create a custom authenticator (really just an older version of ldapauthenticator - 1.2.2) in extraConfig that the hub can use. Here’s a minimal example:

hub:
    # Just doesn't work...
    # JupyterHub:
    #   authenticator_class: ldapauthenticator.LDAPAuthenticator
    # LDAPAuthenticator:
    #   bind_dn_template:
    #     - 'CN={username},OU=xxx,DC=xx,DC=xxxx,DC=xxx'
    #   server_address: xxx.xxx.xxx.xxx
    #   server_port: xxx
    #   use_ssl: False
  extraConfig:
    myConfig: |
      # Source: https://github.com/jupyterhub/ldapauthenticator/blob/630c512f6fea871be137cf902ce5d0f20ceec0c7/ldapauthenticator/ldapauthenticator.py
      import re

      from jupyterhub.auth import Authenticator
      import ldap3
      from ldap3.utils.conv import escape_filter_chars
      from tornado import gen
      from traitlets import Unicode, Int, Bool, List, Union


      class LDAPAuthenticator(Authenticator):
          server_address = Unicode(
              config=True,
              help="""
              Address of the LDAP server to contact.
              Could be an IP address or hostname.
              """
          )
          server_port = Int(
              config=True,
              help="""
              Port on which to contact the LDAP server.
              Defaults to `636` if `use_ssl` is set, `389` otherwise.
              """
          )

          def _server_port_default(self):
              if self.use_ssl:
                  return 636  # default SSL port for LDAP
              else:
                  return 389  # default plaintext port for LDAP

          use_ssl = Bool(
              False,
              config=True,
              help="""
              Use SSL to communicate with the LDAP server.
              Deprecated in version 3 of LDAP. Your LDAP server must be configured to support this, however.
              """
          )

          bind_dn_template = Union(
              [List(),Unicode()],
              config=True,
              help="""
              Template from which to construct the full dn
              when authenticating to LDAP. {username} is replaced
              with the actual username used to log in.
              If your LDAP is set in such a way that the userdn can not
              be formed from a template, but must be looked up with an attribute
              (such as uid or sAMAccountName), please see `lookup_dn`. It might
              be particularly relevant for ActiveDirectory installs.
              Unicode Example:
                  uid={username},ou=people,dc=wikimedia,dc=org
              List Example:
                  [
                      uid={username},ou=people,dc=wikimedia,dc=org,
                      uid={username},ou=Developers,dc=wikimedia,dc=org
                      ]
              """
          )

          allowed_groups = List(
              config=True,
              allow_none=True,
              default=None,
              help="""
              List of LDAP group DNs that users could be members of to be granted access.
              If a user is in any one of the listed groups, then that user is granted access.
              Membership is tested by fetching info about each group and looking for the User's
              dn to be a value of one of `member` or `uniqueMember`, *or* if the username being
              used to log in with is value of the `uid`.
              Set to an empty list or None to allow all users that have an LDAP account to log in,
              without performing any group membership checks.
              """
          )

          # FIXME: Use something other than this? THIS IS LAME, akin to websites restricting things you
          # can use in usernames / passwords to protect from SQL injection!
          valid_username_regex = Unicode(
              r'^[a-z][.a-z0-9_-]*$',
              config=True,
              help="""
              Regex for validating usernames - those that do not match this regex will be rejected.
              This is primarily used as a measure against LDAP injection, which has fatal security
              considerations. The default works for most LDAP installations, but some users might need
              to modify it to fit their custom installs. If you are modifying it, be sure to understand
              the implications of allowing additional characters in usernames and what that means for
              LDAP injection issues. See https://www.owasp.org/index.php/LDAP_injection for an overview
              of LDAP injection.
              """
          )

          lookup_dn = Bool(
              False,
              config=True,
              help="""
              Form user's DN by looking up an entry from directory
              By default, LDAPAuthenticator finds the user's DN by using `bind_dn_template`.
              However, in some installations, the user's DN does not contain the username, and
              hence needs to be looked up. You can set this to True and then use `user_search_base`
              and `user_attribute` to accomplish this.
              """
          )

          user_search_base = Unicode(
              config=True,
              default=None,
              allow_none=True,
              help="""
              Base for looking up user accounts in the directory, if `lookup_dn` is set to True.
              LDAPAuthenticator will search all objects matching under this base where the `user_attribute`
              is set to the current username to form the userdn.
              For example, if all users objects existed under the base ou=people,dc=wikimedia,dc=org, and
              the username users use is set with the attribute `uid`, you can use the following config:
              ```
              c.LDAPAuthenticator.lookup_dn = True
              c.LDAPAuthenticator.lookup_dn_search_filter = '({login_attr}={login})'
              c.LDAPAuthenticator.lookup_dn_search_user = 'ldap_search_user_technical_account'
              c.LDAPAuthenticator.lookup_dn_search_password = 'secret'
              c.LDAPAuthenticator.user_search_base = 'ou=people,dc=wikimedia,dc=org'
              c.LDAPAuthenticator.user_attribute = 'sAMAccountName'
              c.LDAPAuthenticator.lookup_dn_user_dn_attribute = 'cn'
              ```
              """
          )

          user_attribute = Unicode(
              config=True,
              default=None,
              allow_none=True,
              help="""
              Attribute containing user's name, if `lookup_dn` is set to True.
              See `user_search_base` for info on how this attribute is used.
              For most LDAP servers, this is uid.  For Active Directory, it is
              sAMAccountName.
              """
          )

          lookup_dn_search_filter = Unicode(
              config=True,
              default_value='({login_attr}={login})',
              allow_none=True,
              help="""
              How to query LDAP for user name lookup, if `lookup_dn` is set to True.
              """
          )

          lookup_dn_search_user = Unicode(
              config=True,
              default_value=None,
              allow_none=True,
              help="""
              Technical account for user lookup, if `lookup_dn` is set to True.
              If both lookup_dn_search_user and lookup_dn_search_password are None, then anonymous LDAP query will be done.
              """
          )

          lookup_dn_search_password = Unicode(
              config=True,
              default_value=None,
              allow_none=True,
              help="""
              Technical account for user lookup, if `lookup_dn` is set to True.
              """
          )

          lookup_dn_user_dn_attribute = Unicode(
              config=True,
              default_value=None,
              allow_none=True,
              help="""
              Attribute containing user's name needed for  building DN string, if `lookup_dn` is set to True.
              See `user_search_base` for info on how this attribute is used.
              For most LDAP servers, this is username.  For Active Directory, it is cn.
              """
          )

          escape_userdn = Bool(
              False,
              config=True,
              help="""
              If set to True, escape special chars in userdn when authenticating in LDAP.
              On some LDAP servers, when userdn contains chars like '(', ')', '\' authentication may fail when those chars
              are not escaped.
              """
          )

          def resolve_username(self, username_supplied_by_user):
              if self.lookup_dn:
                  server = ldap3.Server(
                      self.server_address,
                      port=self.server_port,
                      use_ssl=self.use_ssl
                  )

                  search_filter = self.lookup_dn_search_filter.format(
                      login_attr=self.user_attribute,
                      login=username_supplied_by_user
                  )
                  self.log.debug(
                      "Looking up user with search_base={search_base}, search_filter='{search_filter}', attributes={attributes}".format(
                          search_base=self.user_search_base,
                          search_filter=search_filter,
                          attributes=self.user_attribute
                      )
                  )

                  conn = ldap3.Connection(server, user=self.escape_userdn_if_needed(self.lookup_dn_search_user), password=self.lookup_dn_search_password)
                  is_bound = conn.bind()
                  if not is_bound:
                      self.log.warn("Can't connect to LDAP")
                      return None

                  conn.search(
                      search_base=self.user_search_base,
                      search_scope=ldap3.SUBTREE,
                      search_filter=search_filter,
                      attributes=[self.lookup_dn_user_dn_attribute]
                  )

                  if len(conn.response) == 0 or 'attributes' not in conn.response[0].keys():
                      self.log.warn('username:%s No such user entry found when looking up with attribute %s', username_supplied_by_user,
                                    self.user_attribute)
                      return None
                  return conn.response[0]['attributes'][self.lookup_dn_user_dn_attribute]
              else:
                  return username_supplied_by_user

          def escape_userdn_if_needed(self, userdn):
              if self.escape_userdn:
                  return escape_filter_chars(userdn)
              else:
                  return userdn

          search_filter = Unicode(
              config=True,
              help="LDAP3 Search Filter whose results are allowed access"
          )

          attributes = List(
              config=True,
              help="List of attributes to be searched"
          )


          @gen.coroutine
          def authenticate(self, handler, data):
              username = data['username']
              password = data['password']
              # Get LDAP Connection
              def getConnection(userdn, username, password):
                  server = ldap3.Server(
                      self.server_address,
                      port=self.server_port,
                      use_ssl=self.use_ssl
                  )
                  self.log.debug('Attempting to bind {username} with {userdn}'.format(
                          username=username,
                          userdn=userdn
                  ))
                  conn = ldap3.Connection(
                      server,
                      user=self.escape_userdn_if_needed(userdn),
                      password=password,
                      auto_bind=self.use_ssl and ldap3.AUTO_BIND_TLS_BEFORE_BIND or ldap3.AUTO_BIND_NO_TLS,
                  )
                  return conn

              # Protect against invalid usernames as well as LDAP injection attacks
              if not re.match(self.valid_username_regex, username):
                  self.log.warn('username:%s Illegal characters in username, must match regex %s', username, self.valid_username_regex)
                  return None

              # No empty passwords!
              if password is None or password.strip() == '':
                  self.log.warn('username:%s Login denied for blank password', username)
                  return None

              isBound = False
              self.log.debug("TYPE= '%s'",isinstance(self.bind_dn_template, list))

              resolved_username = self.resolve_username(username)
              if resolved_username is None:
                  return None

              if self.lookup_dn:
                  if str(self.lookup_dn_user_dn_attribute).upper() == 'CN':
                      # Only escape commas if the lookup attribute is CN
                      resolved_username = re.subn(r"([^\\]),", r"\1\,", resolved_username)[0]

              bind_dn_template = self.bind_dn_template
              if isinstance(bind_dn_template, str):
                  # bind_dn_template should be of type List[str]
                  bind_dn_template = [bind_dn_template]

              for dn in bind_dn_template:
                  userdn = dn.format(username=resolved_username)
                  msg = 'Status of user bind {username} with {userdn} : {isBound}'
                  try:
                      conn = getConnection(userdn, username, password)
                  except ldap3.core.exceptions.LDAPBindError as exc:
                      isBound = False
                      msg += '\n{exc_type}: {exc_msg}'.format(
                          exc_type=exc.__class__.__name__,
                          exc_msg=exc.args[0] if exc.args else ''
                      )
                  else:
                      isBound = conn.bind()
                  msg = msg.format(
                      username=username,
                      userdn=userdn,
                      isBound=isBound
                  )
                  self.log.debug(msg)
                  if isBound:
                      break

              if isBound:
                  if self.allowed_groups:
                      self.log.debug('username:%s Using dn %s', username, userdn)
                      for group in self.allowed_groups:
                          groupfilter = (
                              '(|'
                              '(member={userdn})'
                              '(uniqueMember={userdn})'
                              '(memberUid={uid})'
                              ')'
                          ).format(userdn=escape_filter_chars(userdn), uid=escape_filter_chars(username))
                          groupattributes = ['member', 'uniqueMember', 'memberUid']
                          if conn.search(
                              group,
                              search_scope=ldap3.BASE,
                              search_filter=groupfilter,
                              attributes=groupattributes
                          ):
                              return username
                      # If we reach here, then none of the groups matched
                      self.log.warn('username:%s User not in any of the allowed groups', username)
                      return None
                  elif self.search_filter:
                      conn.search(
                          search_base=self.user_search_base,
                          search_scope=ldap3.SUBTREE,
                          search_filter=self.search_filter.format(userattr=self.user_attribute,username=username),
                          attributes=self.attributes
                      )
                      if len(conn.response) == 0:
                          self.log.warn('User with {userattr}={username} not found in directory'.format(
                              userattr=self.user_attribute, username=username))
                          return None
                      elif len(conn.response) > 1:
                          self.log.warn('User with {userattr}={username} found more than {len}-fold in directory'.format(
                              userattr=self.user_attribute, username=username, len=len(conn.response)))
                          return None
                      return username
                  else:
                      return username
              else:
                  self.log.warn('Invalid password for user {username}'.format(
                      username=username,
                  ))
                  return None

      # Here is the actual configuration...
      c.JupyterHub.authenticator_class = LDAPAuthenticator
      c.LDAPAuthenticator.server_address = 'xxx.xxx.xxx.xxx'
      c.LDAPAuthenticator.server_port = xxx
      c.LDAPAuthenticator.bind_dn_template = 'CN={username},OU=xxx,DC=xx,DC=xxxx,DC=xxx'

Thanks for the reply!

I haven’t tried this myself. But I’ve got the feeling that using LDAP+LDAPAuthenticator on Jupyter is a potential landmine moving forward. Not to criticise the Jupyter developers! It just seems complex enough a stack to be prone to such problems. For my small-medium sized demands, I’ve looked instead to using the NativeAuthenticator and customising it where necessary (see eg my Feature Request on the NativeAuthenticator for more flexibility, which could be achieved with customisation just like yours).

I’ve pretty much moved any production deployments to oauth authentication instead, but still use LDAP some for development. Still, it is true that a non-ssl or otherwise out-dated security setup will eventually stop being supported…

1 Like

The main problem with the LDAPauthenticator is the current JupyterHub GitHub team have very limited experience with LDAP. This makes creating and reviewing PRs difficult since we don’t have the background knowledge to know if a change (even a bug fix) has side effects that could break other users.

There’s nothing inherently complicated about integrating LDAP or any other authentication mechanism with JupyterHub. In the simplest case it’s just a method that takes in (username, password) and checks whether the user is allowed or not!

The complication is almost entirely on the LDAP side of things, so if you’ve got extensive LDAP experience feel free to help, especially with reviewing PRs.

Thanks for the background!

And while this is a very understandable scenario for the JupyterHub developers (thanks so much BTW), it is the reason I’m looking to move away from relying on LDAP, as I also don’t have experience developing with LDAP, only using already built systems.

In this regard, just a shoutout to the NativeAuthenticator. The essential design fits my needs much better, and presumably many others too, while providing a good set of features that should be stable over time (as it’s integrated with, or native to JupyterHub.