Binderhub install gives 400 error (bad request)

Binder BinderHub
1

Hello! I’m trying to install Binderhub v0.2.0-b593363 and I get a 400 error. I saw Github issue #611 on about this, but I don’t have any comments in starting with < that may be causing the error. This might be a silly mistake or oversight on my part, but I’ve tried to deploy the Binderhub more than once (including a tear-down and re-start) and I still get the error. Could someone help me with this?

Here is the kubernetes pod log:

Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/usr/local/lib/python3.6/site-packages/binderhub/builder.py", line 273, in get
        image_manifest = await self.registry.get_image_manifest(*'/'.join(image_name.split('/')[-2:]).split(':', 1))
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/usr/local/lib/python3.6/site-packages/binderhub/registry.py", line 199, in get_image_manifest
        auth_resp = yield client.fetch(auth_req)
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
    tornado.httpclient.HTTPError: HTTP 400: Bad Request

Here is the secret.yaml file:

jupyterhub:
  hub:
    services:
      binder:
        apiToken: "redacted"
  proxy:
    secretToken: "redacted"
registry:
  url: https://gcr.io
  password: |
    {
    "type": "service_account",
    "project_id": "helioml",
    "private_key_id": "redacted",
    "private_key": "-----BEGIN PRIVATE KEY-----\nredacted=\n-----END PRIVATE KEY-----\n",
    "client_email": "binderhub-builder@helioml.iam.gserviceaccount.com",
    "client_id": "redacted",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/binderhub-builder%40helioml.iam.gserviceaccount.com"
    }
  config:
    GitHubRepoProvider:
      access_token: "redacted"

Here is the config.yaml file:

config:
  BinderHub:
    use_registry: true
    image_prefix: gcr.io/<helioml>/<binder-dev->-
    hub_url: http://35.245.153.145

Thanks!

cc @jmason86 @choldgraf

2

Cc also @sgibson91 and @KirstieJane who I believe have been working on a binderhub deployment as well

3

Hmmm, looking at the config, I wanna quickly confirm something:

here:

    image_prefix: gcr.io/<helioml>/<binder-dev->-

you have < around ‘helioml’ etc. Those should not be in the actual config file, it should instead be

    image_prefix: gcr.io/helioml/binder-dev-

Could you confirm this is what your config file looks like and give it a shot?

4

Yaaay, it works!!! Thank you! It was the < key after all. I’m really, really sorry – I should have seen that. I think it might be helpful to show an example secret.yaml file and config.yaml file with fake key values in the BinderHub docs at the end of Step 3.7.

Although the BinderHub works (I tested it out with a simple repository unrelated to the jupyter-book), it doesn’t build the master branch of my jupyter-book:

/bin/sh: 1: conda: Invalid argument
Removing intermediate container 09fe881d9b66
The command '/bin/sh -c conda env update -n root -f "environment.yml" && conda clean -tipsy && conda list -n root' returned a non-zero code: 2

However, I can successfully build this master branch on mybinder.org, so I’m not sure if there’s anything wrong with my environment.yml file. Is this a resource allocation problem?

2 Likes
5

hmmmm, I wonder if your binderhub is running and older version of conda…when you ran helm upgrade ... what was the version of the binderhub chart you used?

6

I was using 0.2.0-b593363 (from the February 4 release) but I re-created a BinderHub with the 0.2.0-6bfd93b chart and it works! :tada:

However, it took a really long time to build (~15 minutes). When I click the interact button in the jupyter-book, I get a timeout error (on Chrome, Safari, and Firefox).

I checked the Kubernetes pods and I am using a minimal amount (less than 10%) of CPU and memory. I also upped my Github rate limit to 5k. Is our codebase just too big?

7

hmmmm, it could be that the size of the images is really big. Are you using a “kitchen-sink”-style repository? Maybe you can check the size of the docker image to see if that’s prohibitive.

8

Yes, the virtual size of the image is 2.7GB.

I don’t know what kitchen-sink means, but I assume it means a single-layer image – if so, the answer is yes. Is there a way around this problem (other than lightening the environment.yml file)?

I looked at the repo2docker and binderhub docs and couldn’t find anything obvious (but I don’t understand exactly what to look for, so I may have missed something).

9

hmmm 2.7GB doesn’t seem to be too bad. (by “kitchen sink” I mean some folks that just put every package imaginable in their envt.yml file, just in case they’ll need it later :slight_smile: )

in kubernetes, you could check the logs of the build pod while it’s building…first try building a repo, then do something like:

# List all pods
kubectl --namespace=yournamespace get pod

# Then find the build pod

# List its logs
kubectl --namespace=yournamespace logs <build-pod-name>
10

It works!!! :tada::tada::tada: It turns out I was using https:// and then the IP for the binderhub_url field in the jupyter-book _config.yml file, when I should have been using http://.

The book is slow to load the first time. It took 5 minutes for BinderHub to successfully build the image, 2.5 minutes to push the image, and another 2 minutes to launch the server. In all, it took 9.5 minutes to go from clicking ‘Interact’ to being able to execute cells. After the first time, it loads quickly.

I am super happy with the end result and I think the Jupyter Book is genuinely one of the best educational tools of all time! Thanks for helping me through all these details. I really appreciate the support.

11

wow! that’s great news! It is always something small like that :slight_smile:

You should be able to reduce the first-time launch by using pre-puller configuration for the image that you want available:

https://zero-to-jupyterhub.readthedocs.io/en/latest/advanced.html?highlight=pre%20puller#pre-pulling-images-for-faster-startup

The reason it takes a long time is because it has to do a docker pull if the image doesn’t exist already on the node where the session is being launched.

and thanks for your kind words about Jupyter Book! Please feel free to make suggestions, edits, PRs, etc! It’s still early days for this project so there are lots of improvements to be made!