JupyterHub 0.11.1 LDAP not working

So, I had a JupyterHub 0.9.0 that worked great; snippet of configuration (this will not work external to our network, but it did work for me, internally which means this is the correct address and the DNS is correctly resolving it):
auth:
admin:
users:
- hc7
ldap:
dn:
templates:
- uid={username},ou=people,dc=sanger,dc=ac,dc=uk
server:
address: ldap-ro.internal.sanger.ac.uk
type: ldap

I thought to upgrade to 0.11.1, on the same k8s cluster, it said I needed to update my config so it is now
hub:
config:
JupyterHub:
authenticator_class: ldapauthenticator.LDAPAuthenticator
LDAPAuthenticator:
bind_dn_template:
- uid={username},ou=people,dc=sanger,dc=ac,dc=uk
server_address: ldap-ro.internal.sanger.ac.uk
user_search_base: ou=people,dc=sanger,dc=ac,dc=uk

(this ought to be the SAME server, although I am unsure where to put admin users in this config)

The install appeared to work. I went to my hub website and it gave a login page, I provided my LDAP credentials (as previously working) and got 500: Internal server error. In the hub pod logs:

[E 2021-01-20 10:20:19.204 JupyterHub web:1789] Uncaught exception POST /hub/login?next= (192.168.199.234)

HTTPServerRequest(protocol='https', host='hc7-demo.internal.sanger.ac.uk', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='192.168.199.234')

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute

    result = await result

  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/login.py", line 144, in post

    user = await self.login_user(data)

  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user

    authenticated = await self.authenticate(data)

  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user

    authenticated = await maybe_future(self.authenticate(handler, data))

  File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 382, in authenticate

    conn = self.get_connection(userdn, password)

  File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 314, in get_connection

    conn = ldap3.Connection(

  File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 356, in __init__

    self._do_auto_bind()

  File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 380, in _do_auto_bind

    self.open(read_server_info=False)

  File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/sync.py", line 56, in open

    BaseStrategy.open(self, reset_usage, read_server_info)

  File "/usr/local/lib/python3.8/dist-packages/ldap3/strategy/base.py", line 153, in open

    raise LDAPSocketOpenError('invalid server address')

ldap3.core.exceptions.LDAPSocketOpenError: invalid server address

I do net believe it is the wrong address, although I may have provided it in the wrong format, I tried to obey the documentation. Unsure how to proceed

oh, curses, I was sure I pasted WITH FORMATING… try again.
old (working) config:

auth:
admin:
  users:
  - hc7
ldap:
  dn:
    templates:
    - uid={username},ou=people,dc=sanger,dc=ac,dc=uk
  server:
    address: ldap-ro.internal.sanger.ac.uk
type: ldap

new (boken) config:

hub:
config:
  JupyterHub:
    authenticator_class: ldapauthenticator.LDAPAuthenticator
  LDAPAuthenticator:
    bind_dn_template:
      - uid={username},ou=people,dc=sanger,dc=ac,dc=uk
  server_address: ldap-ro.internal.sanger.ac.uk
  user_search_base: ou=people,dc=sanger,dc=ac,dc=uk

Is that an exact copy? The last two lines need to be indented by one more level.

ooooh, yes, I’ll try that… ooooh it works ta much. Well, it let me log in, and started my user pod, but the website refuses to go to my notebook. Unable to find any useful error.

[I 2021-01-21 09:11:29.373 JupyterHub log:181] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-hc7&redirect_uri=%2Fuser%2Fhc7%2Foauth_callback&response_type=code&state=[secret] -> /user/hc7/oauth_callback?code=[secret]&state=[secret] (hc7@192.168.199.234) 76.64ms

[I 2021-01-21 09:12:12.488 JupyterHub proxy:319] Checking routes

[I 2021-01-21 09:12:19.485 JupyterHub log:181] 200 GET /hub/error

(hub logs, unhelpful)

error: 503 GET /user/hc7/oauth_callback?code=PFoUomU1f6gxOy5DaSajtPftp4Uuuw&state=eyJ1dWlkIjogIjJhYTdhYjVhMDAyNjRlNTA5MDRlYjE3OWJiOGFmNWJiIiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyIsICJjb29raWVfbmFtZSI6ICJqdXB5dGVyaHViLXVzZXItaGM3LW9hdXRoLXN0YXRlLVZOZmxxTFRiIn0 socket hang up

proxy log; I did think maybe security groups; but I have opened the nodes up and it has not helped

Can you turn on debug logging debug.enabled: true zero-to-jupyterhub-k8s/values.yaml at ca859aafb41d4081388ea5790c26e69b106e91e3 · jupyterhub/zero-to-jupyterhub-k8s · GitHub
And if possible show us your full configuration, with secrets redacted?

I have enabled debug, I don’t know what part of the output is useful.

Config (I really hope I can paste with formatting)

hub:
  config:
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
      bind_dn_template: 
        - uid={username},ou=people,dc=sanger,dc=ac,dc=uk
      server_address: ldap-ro.internal.sanger.ac.uk 
      user_search_base: ou=people,dc=sanger,dc=ac,dc=uk
ingress:
  annotations:
    ingress.kubernetes.io/proxy-body-size: 64m
    ingress.kubernetes.io/proxy-connect-timeout: "30"
    ingress.kubernetes.io/proxy-read-timeout: "3600"
    ingress.kubernetes.io/proxy-send-timeout: "3600"
    kubernetes.io/ingress.class: nginx
  enabled: true
  hosts:
  - hc7-demo.internal.sanger.ac.uk
  tls:
  - hosts:
    - hc7-demo.internal.sanger.ac.uk
    secretName: tls-jupyter
debug:
  enabled: true
proxy:
  secretToken: NNNNNNNNNNN
  service:
    type: ClusterIP
    scheduling:
  userPods:
    nodeAffinity:
      matchNodePurpose: ignore
1 Like

Could you maybe show us all the logs corresponding to one login attempt?

these are proxy logs, which is where the errors are. (why are the logs in so many places and so hard to get at and argh) also I have had to edit the “links” to not be, because the forum hates me, assume they were one correct.

10:25:42.034 [ConfigProxy] info: 200 GET /api/routes

10:26:38.001 [ConfigProxy] debug: PROXY WEB /user/hc7/oauth_callback?code=WTW5ro60JZ5ZUja7xbAhQQnIeRkaIx&state=eyJ1dWlkIjogIjZjODg5YzcxZWIxMTQxNDU5ZjhiYTQ4NzcyYmRjYzc3IiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlP3JlZGlyZWN0cz0xIn0 to [hub:8081 link]

10:26:38.128 [ConfigProxy] debug: PROXY WEB /hub/user/hc7/oauth_callback?code=WTW5ro60JZ5ZUja7xbAhQQnIeRkaIx&state=eyJ1dWlkIjogIjZjODg5YzcxZWIxMTQxNDU5ZjhiYTQ4NzcyYmRjYzc3IiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlP3JlZGlyZWN0cz0xIn0 to [hub:8081 link]

10:26:38.355 [ConfigProxy] debug: PROXY WEB /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 to [hub:8081 link]
10:26:38.366 [ConfigProxy] debug: PROXY WEB /hub/static/js/not_running.js?v=20210125151538 to [hub:8081 link]

10:26:42.034 [ConfigProxy] info: 200 GET /api/routes

10:26:43.326 [ConfigProxy] debug: PROXY WEB /hub/logout to [hub:8081 link]

10:26:43.388 [ConfigProxy] debug: PROXY WEB /hub/login to [hub:8081 link]

10:26:43.500 [ConfigProxy] debug: PROXY WEB /hub/logo to [hub:8081 link]

10:26:46.207 [ConfigProxy] debug: PROXY WEB /hub/login?next= to [hub:8081 link]

10:26:46.302 [ConfigProxy] debug: PROXY WEB /hub/spawn to [hub:8081 link]

10:26:46.660 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/hc7 to [hub:8081 link]

10:26:48.723 [ConfigProxy] debug: PROXY WEB /hub/spawn/hc7 to [hub:8081 link]

10:26:48.983 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/hc7 to [hub:8081 link]

10:26:49.122 [ConfigProxy] debug: PROXY WEB /hub/api/users/hc7/server/progress to [hub:8081 link]

10:26:53.771 [ConfigProxy] info: Adding route /user/hc7 ->[10.42.2.21:8888 link]

10:26:53.771 [ConfigProxy] info: Route added /user/hc7 → [10.42.2.21:8888 link]

10:26:53.772 [ConfigProxy] info: 201 POST /api/routes/user/hc7

10:26:53.907 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/hc7 to [hub:8081 link]

10:26:53.955 [ConfigProxy] debug: PROXY WEB /user/hc7/ to [10.42.2.21:8888 link]

10:26:53.992 [ConfigProxy] debug: PROXY WEB /user/hc7/tree? to [10.42.2.21:8888 link]

10:26:54.031 [ConfigProxy] debug: PROXY WEB /hub/api/oauth2/authorize?client_id=jupyterhub-user-hc7&redirect_uri=%2Fuser%2Fhc7%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjFjOTNiYjVjODQ2MDQ2MTA5YTgxMzU0OTFmOGJmYWI3IiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyJ9 to [hub:8081 link]

10:26:54.220 [ConfigProxy] debug: PROXY WEB /user/hc7/oauth_callback?code=N6Pd9dTVLcfnPkU89gHBBddgPxkySV&state=eyJ1dWlkIjogIjFjOTNiYjVjODQ2MDQ2MTA5YTgxMzU0OTFmOGJmYWI3IiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyJ9 to [10.42.2.21:8888 link]

10:27:42.035 [ConfigProxy] info: 200 GET /api/routes

10:27:44.222 [ConfigProxy] error: 503 GET /user/hc7/oauth_callback?code=N6Pd9dTVLcfnPkU89gHBBddgPxkySV&state=eyJ1dWlkIjogIjFjOTNiYjVjODQ2MDQ2MTA5YTgxMzU0OTFmOGJmYWI3IiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyJ9 socket hang up

Could you paste your hub logs too?
If you use triple backticks on the lines before and after your logs they’ll be formatted as a code block, see Creating and highlighting code blocks - GitHub Docs

Might Adjust ldapauthenticator to work with ldap3 v2.8 · Issue #174 · jupyterhub/ldapauthenticator · GitHub be of interest?

oh, I went away for a weekend and suddenly it works. the LDAP stuff may very well be of interest

and I thought to see that it works ground up… and no, it does not

If it started working without any changes on your end is it possible the problem is with your LDAP server? Seeing your hub logs might be helpful.

I am a bit confused, it is possible I accidentally (I mean while not paying attention) switched version back to 0.9.0.

And no, the LDAP servr did not change and is not the problem. I took it out of the config, and the problem persisted.
Here, a minimal config


ingress:
  annotations:
    ingress.kubernetes.io/proxy-body-size: 64m
    ingress.kubernetes.io/proxy-connect-timeout: "30"
    ingress.kubernetes.io/proxy-read-timeout: "3600"
    ingress.kubernetes.io/proxy-send-timeout: "3600"
    kubernetes.io/ingress.class: nginx
  enabled: true
  hosts:
  - hc7-demo.internal.sanger.ac.uk
  tls:
  - hosts:
    - hc7-demo.internal.sanger.ac.uk
    secretName: tls-jupyter
proxy:
  secretToken: NNNNNNNNNNNNNNN
  service:
    type: ClusterIP

it does the same,
in the proxy log
08:55:16.591 [ConfigProxy] error: 503 GET /user/hc7/oauth_callback?code=2nwBxqVsWzFQdD79nxMtrAApt3kYok&state=eyJ1dWlkIjogImE5ZGNmNTQ4OWZmNzRlODBiMjY5YzZlY2EwNjhmZGMxIiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyJ9 socket hang up

in the hub log
I 2021-02-03 08:54:26.523 JupyterHub log:181] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-hc7&redirect_uri=%2Fuser%2Fhc7%2Foauth_callback&response_type=code&state=[secret] → /user/hc7/oauth_callback?code=[secret]&state=[secret] (hc7@192.168.199.103) 30.54ms

[I 2021-02-03 08:54:53.141 JupyterHub proxy:319] Checking routes

[I 2021-02-03 08:55:16.604 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fuser%2Fhc7%2Foauth_callback%3Fcode%3D2nwBxqVsWzFQdD79nxMtrAApt3kYok%26state%3DeyJ1dWlkIjogImE5ZGNmNTQ4OWZmNzRlODBiMjY5YzZlY2EwNjhmZGMxIiwgIm5leHRfdXJsIjogIi91c2VyL2hjNy90cmVlPyJ9 (@10.42.2.75) 8.21ms

the user pod launched (with no password, of course, because there is no authentication). This is presumably a proxy issue