Zero to BinderHub docs leads to JupyterHub issue: Failed to create temporary user

I’m working through the Zero to BinderHub instructions, trying to get a binderhub/jupyterhub off the ground and I’m running into issues. I have a feeling that the problem may be stale documentation (which I’m happy to help fix) but perhaps I’m not following the docs correctly either.

I’m setting everything up on Digital Ocean, using DockerHub as the registry. It looks like the problem is definitely JupyterHub.

BinderHub seems to be building everything correctly:

Found built image, launching...
Launching server...
Launch attempt 1 failed, retrying...
Launch attempt 2 failed, retrying...
Launch attempt 3 failed, retrying...
Failed to create temporary user for sophsea/hub-binder-2dexamples-2drequirements-55ab5c:82543a814cdd4cb125d03112a193dd4137d8751c

Here’s the pods

> kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
default       binder-7bdc8cf7fc-6qk6f            1/1     Running   0          8m59s
default       binder-image-cleaner-dwjvs         1/1     Running   1          21h
default       binder-image-cleaner-pkmv6         1/1     Running   0          20h
default       hub-6cd86945b-wdpc4                0/1     Running   98         17h
default       proxy-795dfdbd9-tvphx              1/1     Running   0          17h
default       user-scheduler-78776cd96b-sgmg8    1/1     Running   0          20h
default       user-scheduler-78776cd96b-zpb7l    1/1     Running   1          21h
kube-system   cilium-czbg2                       1/1     Running   0          20h
kube-system   cilium-khw9x                       1/1     Running   1          21h
kube-system   cilium-operator-84bdd6f7b6-b2wj7   1/1     Running   0          20h
kube-system   cilium-operator-84bdd6f7b6-qj9d2   1/1     Running   0          20h
kube-system   coredns-55ff57f948-lsk6x           1/1     Running   0          20h
kube-system   coredns-55ff57f948-rl9bb           1/1     Running   0          20h
kube-system   csi-do-node-24vtx                  2/2     Running   2          21h
kube-system   csi-do-node-nqp6f                  2/2     Running   0          20h
kube-system   do-node-agent-chnc6                1/1     Running   0          20h
kube-system   do-node-agent-dsnrd                1/1     Running   1          21h
kube-system   kube-proxy-8xvgs                   1/1     Running   0          20h
kube-system   kube-proxy-sb984                   1/1     Running   1          21h

I lost the specific stack trace from that hub-<foo> pod but here’s a representative log from a previous run:

> kubectl logs hub-65c5d5987-2v9fw
No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
Loading extra config: 00-binder
[I 2021-05-28 13:13:28.637 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Authenticator: nullauthenticator.NullAuthenticator-1.0.0
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Spawner: builtins.BinderSpawner
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-05-28 13:13:28.639 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[W 2021-05-28 13:13:28.664 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-05-28 13:13:28.664 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-28 13:13:28.664 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[I 2021-05-28 13:13:28.980 JupyterHub app:2416] Initialized 0 spawners in 0.002 seconds
[I 2021-05-28 13:13:28.982 JupyterHub app:2628] Not starting proxy
[I 2021-05-28 13:13:28.992 JupyterHub app:2664] Hub API listening on http://:8081/hub/
[I 2021-05-28 13:13:28.993 JupyterHub app:2666] Private Hub API connect url http://hub:8081/hub/
[I 2021-05-28 13:13:28.993 JupyterHub app:2679] Starting managed service cull-idle
[I 2021-05-28 13:13:28.993 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10', '--cull-users']
[I 2021-05-28 13:13:28.997 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10 --cull-users
[I 2021-05-28 13:13:29.004 JupyterHub app:2688] Adding external service binder
[I 2021-05-28 13:13:29.008 JupyterHub proxy:319] Checking routes
[I 2021-05-28 13:13:29.009 JupyterHub app:2739] JupyterHub is now running at http://:8000
[I 2021-05-28 13:13:29.175 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@::1) 50.03ms
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-9uh8kd0p (inactive for 19:48:17.838276)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-kx0yoqvp (inactive for 19:22:28.146680)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-ulcc8nb6 (inactive for 19:16:59.410097)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-6l2aqu7t (inactive for 19:08:35.027401)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-ebo4bf5w (inactive for 19:08:34.835891)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-clj6q0ti (inactive for 19:04:06.844356)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-4prj6s97 (inactive for 18:58:27.393068)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-o540bcn9 (inactive for 18:49:22.953381)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-6cbcqisp (inactive for 18:17:17.593578)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-dfkck6qg (inactive for 18:05:21.955716)
[E 210528 13:13:49 __init__:348] Error processing artificialsoph--3ba16ff0a85852b-9uh8kd0p
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 346, in cull_idle
        result = yield f
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
        value = future.result()
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 769, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 338, in handle_user
        yield fetch(req)
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
        value = future.result()
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 775, in run
        yielded = self.gen.send(value)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
        return (yield client.fetch(req))
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
        value = future.result()
    tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20001 milliseconds with 0 bytes received

It looks like there’s a bunch of users that JHub cannot cull from previous runs (artificialsoph is the username of a previous GH repo I was trying to test).

My secret.yaml:

jupyterhub:
  hub:
    services:
      binder:
        apiToken: "<first token>"
  proxy:
    secretToken: "<second token>"
registry:
  username: <dockerhub username>
  password: <dockerhub password>

My config.yaml:

config:
  BinderHub:
    use_registry: true
    image_prefix: sophsea/hub-
    hub_url: http://159.65.214.50

The Z2BH docs include minimal configuration details for JHub. Most likely causes seem to me to be:

  • Z2BH seems to only ask users to follow the “Setup Kubernetes” section of the Z2JH docs, but perhaps the Z2BH docs expect users to follow the entire setup for JH. I’ve avoided testing this because it seems like doing so would lead to me installing JHub twice, unless I’m misunderstanding something.
  • The default configuration of JHub previously worked out of the box but no longer does, and users need to do some extra work to get JHub working.
  • I’ve incorrectly followed the Z2BH docs.

Any help is appreciated. Hopefully this is an easy problem to spot for anyone who’s gone through this first time setup before.