Self.registry.get_image_manifest() failing after initial helm chart setup

ekasprzy · August 7, 2025, 2:11pm

After installing the binderhub helm chart, and going in the browser to the start page
the trivial github repo GitHub - EmmanuelKasper/my-first-binder: Binder hub test repo
spits out in the log of the binderhub pod the following error:

I 250807 13:22:35 registry:115] Loading docker config /root/.docker/config.json
[E 250807 13:22:35 web:1875] Uncaught exception GET /build/gh/EmmanuelKasper/my-first-binder/HEAD (172.20.81.192)
HTTPServerRequest(protocol=‘http’, host=‘my binder hostname’, method=‘GET’, uri=‘/build/gh/EmmanuelKasper/my-first-binder/HEAD’, version=‘HTTP/1.1’, remote_ip=‘172.20.81.192’)
Traceback (most recent call last):
File “/usr/local/lib/python3.13/site-packages/tornado/web.py”, line 1790, in _execute
result = await result
^^^^^^^^^^^^
File “/usr/local/lib/python3.13/site-packages/binderhub/builder.py”, line 411, in get
image_manifest = await self.registry.get_image_manifest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
image_without_tag, image_tag
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File “/usr/local/lib/python3.13/site-packages/binderhub/registry.py”, line 331, in get_image_manifest
return json.loads(resp.body.decode(“utf-8”))
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.13/json/init.py”, line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File “/usr/local/lib/python3.13/json/decoder.py”, line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.13/json/decoder.py”, line 363, in raw_decode
raise JSONDecodeError(“Expecting value”, s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

now this might look like a authentication problem to the registry (get_image_manifest() is a suspect name ) however the auth file looks fine:

$ oc exec pod/binder-69999b7ddf-g8f5s – cat /root/.docker/config.json > authfile.json

$ podman login --authfile authfile.json my-registry/binderhub-staging --get-login

robot$binderhub-staging-robot

My container image registry is docker harbour, does it play a role here, or am I missing something else ?

manics · August 8, 2025, 2:41pm

Can you share your full BinderHub and JupyterHub configurations, and show us the commands you used to install everything?

ekasprzy · August 11, 2025, 9:09am

In the meantime I think I made some progress narrowing down the issue.
First while setting a setting a development environement, running python3 -m binderhub -f testing/local-binder-k8s-hub/binderhub_config.py to start binder, I could not reproduce the issue.

Second after activating the debug mode, I see the following calls being mode in registry.py:

[I 250811 08:30:19 repoproviders:1062] Using cached ref for https://api.github.com/repos/EmmanuelKasper/my-first-binder/commits/HEAD: 8b84d28e07fe807c7b263264f0792915bf8cdc80
[D 250811 08:30:19 registry:302] Getting image manifest from https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80

now the problem is that this last URL
https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80
is returning in my container registry (Docker Harbour) a HTML 404 if I try it in the browser.

But binder is expecting a JSON output here according to:

github.com/jupyterhub/binderhub

binderhub/registry.py

16ca1e6fd


      
                  ):
                      # Unauthorised. If we don't have a token, try and get one using
                      # information from the WWW-Authenticate header
                      # https://stackoverflow.com/questions/56193110/how-can-i-use-docker-registry-http-api-v2-to-obtain-a-list-of-all-repositories-i/68654659#68654659
                      www_auth_header = e.response.headers["www-authenticate"]
                      return await self._get_image_manifest_from_www_authenticate(
                          client, www_auth_header, url
                      )
                  else:
                      raise
              return json.loads(resp.body.decode("utf-8"))
          
          async def get_credentials(self, image, tag):
              """
              If a dynamic token is required for pushing an image to the registry
              return a dictionary of login credentials, otherwise return None
              (caller should get credentials from some other source)
              """
              return None

What do you think, does this analys make sense ?
Is there an option to deactivate the registry to only use a local docker instance running in Kubernetes ?

I am installing Binder this way:

helm upgrade \
binderhub \
jupyterhub/binderhub \
  --install \
  --version=1.0.0-0.dev.git.3754.hb05f032e \
  --create-namespace \
  --namespace=binderhub \
  -f secret.yaml \
  -f config.yaml

I have the following config.yml

service:
  type: ClusterIP
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx"
  https:
    enabled: false
  hosts:
    - binder-binder.staging.jaas.example.com

jupyterhub:
  proxy:
    service:
      type: ClusterIP
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
    hosts:
      - juniper-binder.staging.jaas.example.com

config:
  BinderHub:
    debug: true

and secret.yml

registry:
  url: https://registry.example.com/binderhub-staging
  username: robot$binderhub-staging-robot
  password: xxx

manics · August 13, 2025, 12:51pm

Does it make a difference if you remove the double // after v2?

ekasprzy · August 14, 2025, 8:13am

Hi Simon
Thanks for your help, I managed to resolve issue, it was an error on my side, after re-reviewing the install parameters, I noticed I missed the DockerRegistry key in the config.yaml of the helm chart:

  DockerRegistry:
    url: https://example.com
    token_url: example.com/service/token?service=harbor-registry

My error was that after I read in paragraph 3.2.5 of 3. Set up BinderHub — BinderHub documentation a reference to Docker Harbor, and I thought 3.2.5 was enough to configure registry access to my Docker Harbor.

I will see if I can make a pull request to fix the documentation in this regard, IMHO if the section in 3.4 should not be labelled as “Expand the config” because “Expand the config” would imply that the parameters mentioned there are not mandatory.

I have also seen that our helm chart for binderhub config.yaml puts the configuration in a Kubernetes Secret: this should be a ConfigMap, not a Secret, because there are no credentials in that part so it breaks the assumption Kubernetes users/admins have around the object,
and it complicates troubleshooting for no good reason since you have to encode / decode via base64 everychange you make to the config.

I will also propose a pull request for that when I am finished with my setup.

ekasprzy · August 26, 2025, 9:19am

Before providing a pull request for using a Kubernetes ConfigMap instead of a secret for config data, I have started with something smaller, as to cut my teeth on the binder development env:

github.com/jupyterhub/binderhub

Update the commands of the helm post install notes to map helm chart fields correctly:

main ← EmmanuelKasper:main

opened 01:39PM - 25 Aug 25 UTC

EmmanuelKasper

+3 -2

When binder is installed with the helm chart, and is only exposed via a ClusterI…P Kubernetes Service, the `kubectl` commands of the post install notes are erroring in subtle ways: ~~~ kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "fullname" . }}" -o jsonpath="{.items[0].metadata.name}" ~~~ can fail because `{{ template "fullname" . }}` does not always match the hardcoded labels we set in https://github.com/jupyterhub/binderhub/blob/33bcc37c2fd18573d19a13c5a41b1d8d563c3663/helm-chart/binderhub/templates/deployment.yaml#L20 and ~~~ kubectl port-forward $POD_NAME 8080:{{ .Values.service.externalPort }} ~~~ always fails because we don't expose set or expose an externalPort field in https://github.com/jupyterhub/binderhub/blob/33bcc37c2fd18573d19a13c5a41b1d8d563c3663/helm-chart/binderhub/values.yaml#L34 This commit fixes the two errors by using immutables, non-overridables values from the helm templates.

Feel free to flame or LGTM !

minrk · August 26, 2025, 9:55am

this should be a ConfigMap, not a Secret, because there are no credentials

Before opening a pull request, I would suggest checking why this is a Secret, since this statement isn’t true in general. This used to be a configmap, but was switched to a Secret because there absolutely are credentials in some config fields. This follows a similar pattern to helm itself, which stores release state in Secrets.

ekasprzy · August 26, 2025, 10:15am

This used to be a configmap, but was switched to a Secret because there absolutely are credentials in some config fields
then I understand this absolutely makes sense to have the parameters in a Kubernetes secrets.

Topic		Replies	Views
Caching problem on OVH k8s cluster deployment BinderHub	2	716	May 23, 2019
Debugging BinderHub: Immediately fails to build a binder BinderHub	6	1714	March 20, 2019
Binderhub install the Helm Chart BinderHub	0	804	March 2, 2020
Binderhub install gives 400 error (bad request) BinderHub	10	1751	February 20, 2019
Binderhub with private docker registry and token_url Binder	3	56	April 27, 2025

Self.registry.get_image_manifest() failing after initial helm chart setup

Related topics