I’m working through the Zero to BinderHub instructions, trying to get a binderhub/jupyterhub off the ground and I’m running into issues. I have a feeling that the problem may be stale documentation (which I’m happy to help fix) but perhaps I’m not following the docs correctly either.
I’m setting everything up on Digital Ocean, using DockerHub as the registry. It looks like the problem is definitely JupyterHub.
BinderHub seems to be building everything correctly:
Found built image, launching...
Launching server...
Launch attempt 1 failed, retrying...
Launch attempt 2 failed, retrying...
Launch attempt 3 failed, retrying...
Failed to create temporary user for sophsea/hub-binder-2dexamples-2drequirements-55ab5c:82543a814cdd4cb125d03112a193dd4137d8751c
Here’s the pods
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default binder-7bdc8cf7fc-6qk6f 1/1 Running 0 8m59s
default binder-image-cleaner-dwjvs 1/1 Running 1 21h
default binder-image-cleaner-pkmv6 1/1 Running 0 20h
default hub-6cd86945b-wdpc4 0/1 Running 98 17h
default proxy-795dfdbd9-tvphx 1/1 Running 0 17h
default user-scheduler-78776cd96b-sgmg8 1/1 Running 0 20h
default user-scheduler-78776cd96b-zpb7l 1/1 Running 1 21h
kube-system cilium-czbg2 1/1 Running 0 20h
kube-system cilium-khw9x 1/1 Running 1 21h
kube-system cilium-operator-84bdd6f7b6-b2wj7 1/1 Running 0 20h
kube-system cilium-operator-84bdd6f7b6-qj9d2 1/1 Running 0 20h
kube-system coredns-55ff57f948-lsk6x 1/1 Running 0 20h
kube-system coredns-55ff57f948-rl9bb 1/1 Running 0 20h
kube-system csi-do-node-24vtx 2/2 Running 2 21h
kube-system csi-do-node-nqp6f 2/2 Running 0 20h
kube-system do-node-agent-chnc6 1/1 Running 0 20h
kube-system do-node-agent-dsnrd 1/1 Running 1 21h
kube-system kube-proxy-8xvgs 1/1 Running 0 20h
kube-system kube-proxy-sb984 1/1 Running 1 21h
I lost the specific stack trace from that hub-<foo> pod but here’s a representative log from a previous run:
> kubectl logs hub-65c5d5987-2v9fw
No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
Loading extra config: 00-binder
[I 2021-05-28 13:13:28.637 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Authenticator: nullauthenticator.NullAuthenticator-1.0.0
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Spawner: builtins.BinderSpawner
[I 2021-05-28 13:13:28.638 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-05-28 13:13:28.639 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[W 2021-05-28 13:13:28.664 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-05-28 13:13:28.664 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-28 13:13:28.664 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[I 2021-05-28 13:13:28.980 JupyterHub app:2416] Initialized 0 spawners in 0.002 seconds
[I 2021-05-28 13:13:28.982 JupyterHub app:2628] Not starting proxy
[I 2021-05-28 13:13:28.992 JupyterHub app:2664] Hub API listening on http://:8081/hub/
[I 2021-05-28 13:13:28.993 JupyterHub app:2666] Private Hub API connect url http://hub:8081/hub/
[I 2021-05-28 13:13:28.993 JupyterHub app:2679] Starting managed service cull-idle
[I 2021-05-28 13:13:28.993 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10', '--cull-users']
[I 2021-05-28 13:13:28.997 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10 --cull-users
[I 2021-05-28 13:13:29.004 JupyterHub app:2688] Adding external service binder
[I 2021-05-28 13:13:29.008 JupyterHub proxy:319] Checking routes
[I 2021-05-28 13:13:29.009 JupyterHub app:2739] JupyterHub is now running at http://:8000
[I 2021-05-28 13:13:29.175 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@::1) 50.03ms
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-9uh8kd0p (inactive for 19:48:17.838276)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-kx0yoqvp (inactive for 19:22:28.146680)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-ulcc8nb6 (inactive for 19:16:59.410097)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-6l2aqu7t (inactive for 19:08:35.027401)
[I 210528 13:13:29 __init__:311] Culling user artificialsoph--3ba16ff0a85852b-ebo4bf5w (inactive for 19:08:34.835891)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-clj6q0ti (inactive for 19:04:06.844356)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-4prj6s97 (inactive for 18:58:27.393068)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-o540bcn9 (inactive for 18:49:22.953381)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-6cbcqisp (inactive for 18:17:17.593578)
[I 210528 13:13:29 __init__:311] Culling user binder-examples-requirements-dfkck6qg (inactive for 18:05:21.955716)
[E 210528 13:13:49 __init__:348] Error processing artificialsoph--3ba16ff0a85852b-9uh8kd0p
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 346, in cull_idle
result = yield f
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 338, in handle_user
yield fetch(req)
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 775, in run
yielded = self.gen.send(value)
File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
return (yield client.fetch(req))
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20001 milliseconds with 0 bytes received
It looks like there’s a bunch of users that JHub cannot cull from previous runs (artificialsoph is the username of a previous GH repo I was trying to test).
My secret.yaml:
jupyterhub:
hub:
services:
binder:
apiToken: "<first token>"
proxy:
secretToken: "<second token>"
registry:
username: <dockerhub username>
password: <dockerhub password>
My config.yaml:
config:
BinderHub:
use_registry: true
image_prefix: sophsea/hub-
hub_url: http://159.65.214.50
The Z2BH docs include minimal configuration details for JHub. Most likely causes seem to me to be:
- Z2BH seems to only ask users to follow the “Setup Kubernetes” section of the Z2JH docs, but perhaps the Z2BH docs expect users to follow the entire setup for JH. I’ve avoided testing this because it seems like doing so would lead to me installing JHub twice, unless I’m misunderstanding something.
- The default configuration of JHub previously worked out of the box but no longer does, and users need to do some extra work to get JHub working.
- I’ve incorrectly followed the Z2BH docs.
Any help is appreciated. Hopefully this is an easy problem to spot for anyone who’s gone through this first time setup before.