# Self.registry.get\_image\_manifest() failing after initial helm chart setup

**URL:** <https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411>\
**Category:** Binder\
**Created:** [August 7, 2025, 2:11pm UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411 "2025-08-07T14:11:28Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ekasprzy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ekasprzy/32/20291_2.png) [@ekasprzy](https://discourse.jupyter.org/u/ekasprzy)\
**Post date:** [August 7, 2025, 2:11pm UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/1 "2025-08-07T14:11:28Z")

</div>

After installing the binderhub helm chart, and going in the browser to the start page  
the trivial github repo [GitHub - EmmanuelKasper/my-first-binder: Binder hub test repo](https://github.com/EmmanuelKasper/my-first-binder)  
spits out in the log of the binderhub pod the following error:

`I 250807 13:22:35 registry:115] Loading docker config /root/.docker/config.json`  
`[E 250807 13:22:35 web:1875] Uncaught exception GET /build/gh/EmmanuelKasper/my-first-binder/HEAD (172.20.81.192)`  
`HTTPServerRequest(protocol=‘http’, host=‘my binder hostname’, method=‘GET’, uri=‘/build/gh/EmmanuelKasper/my-first-binder/HEAD’, version=‘HTTP/1.1’, remote_ip=‘172.20.81.192’)`  
`Traceback (most recent call last):`  
`File “/usr/local/lib/python3.13/site-packages/tornado/web.py”, line 1790, in _execute`  
`result = await result`  
`^^^^^^^^^^^^`  
`File “/usr/local/lib/python3.13/site-packages/binderhub/builder.py”, line 411, in get`  
`image_manifest = await self.registry.get_image_manifest(`  
`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^`  
`image_without_tag, image_tag`  
`^^^^^^^^^^^^^^^^^^^^^^^^^^^^`  
`)`  
`^`  
`File “/usr/local/lib/python3.13/site-packages/binderhub/registry.py”, line 331, in get_image_manifest`  
`return json.loads(resp.body.decode(“utf-8”))`  
` ~~~~~~~~~~ ^^^^^^^^^^^^^^^^^^^^^^^^^^^`  
`File “/usr/local/lib/python3.13/json/` **`init`** `.py”, line 346, in loads`  
`return _default_decoder.decode(s)`  
` ~~~~~~~~~~~~~~~~~~~~~~~ ^^^`  
`File “/usr/local/lib/python3.13/json/decoder.py”, line 345, in decode`  
`obj, end = self.raw_decode(s, idx=_w(s, 0).end())`  
` ~~~~~~~~~~~~~~~ ^^^^^^^^^^^^^^^^^^^^^^^`  
`File “/usr/local/lib/python3.13/json/decoder.py”, line 363, in raw_decode`  
`raise JSONDecodeError(“Expecting value”, s, err.value) from None`  
`json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

now this might look like a authentication problem to the registry (get\_image\_manifest() is a suspect name 🙂 ) however the auth file looks fine:

`$ oc exec pod/binder-69999b7ddf-g8f5s – cat /root/.docker/config.json > authfile.json`

`$ podman login --authfile authfile.json my-registry/binderhub-staging --get-login`

`robot$binderhub-staging-robot`

My container image registry is docker harbour, does it play a role here, or am I missing something else ?

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [August 8, 2025, 2:41pm UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/2 "2025-08-08T14:41:37Z")

</div>

Can you share your full BinderHub and JupyterHub configurations, and show us the commands you used to install everything?

---

<div class="post-metadata">

**Author:** ![ekasprzy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ekasprzy/32/20291_2.png) [@ekasprzy](https://discourse.jupyter.org/u/ekasprzy)\
**Post date:** [August 11, 2025, 9:09am UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/3 "2025-08-11T09:09:03Z")

</div>

In the meantime I think I made some progress narrowing down the issue.  
First while setting a setting a development environement, running `python3 -m binderhub -f testing/local-binder-k8s-hub/binderhub_config.py` to start binder, I could not reproduce the issue.

Second after activating the debug mode, I see the following calls being mode in registry.py:

```auto
[I 250811 08:30:19 repoproviders:1062] Using cached ref for https://api.github.com/repos/EmmanuelKasper/my-first-binder/commits/HEAD: 8b84d28e07fe807c7b263264f0792915bf8cdc80
[D 250811 08:30:19 registry:302] Getting image manifest from https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80

```

now the problem is that this last URL  
[https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80](https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80)  
is returning in my container registry (Docker Harbour) a HTML 404 if I try it in the browser.

But binder is expecting a JSON output here according to:

> <https://github.com/jupyterhub/binderhub/blob/16ca1e6fdd8c38bf3a959eb3d329900f99d6edef/binderhub/registry.py#L331>

What do you think, does this analys make sense ?  
Is there an option to deactivate the registry to only use a local docker instance running in Kubernetes ?

I am installing Binder this way:

```auto
helm upgrade \
binderhub \
jupyterhub/binderhub \
  --install \
  --version=1.0.0-0.dev.git.3754.hb05f032e \
  --create-namespace \
  --namespace=binderhub \
  -f secret.yaml \
  -f config.yaml

```

I have the following config.yml

```auto
service:
  type: ClusterIP
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx"
  https:
    enabled: false
  hosts:
    - binder-binder.staging.jaas.example.com

jupyterhub:
  proxy:
    service:
      type: ClusterIP
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
    hosts:
      - juniper-binder.staging.jaas.example.com

config:
  BinderHub:
    debug: true

```

and secret.yml

```auto
registry:
  url: https://registry.example.com/binderhub-staging
  username: robot$binderhub-staging-robot
  password: xxx

```

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [August 13, 2025, 12:51pm UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/4 "2025-08-13T12:51:45Z")

</div>

> [@ekasprzy](#):
>
> now the problem is that this last URL  
> [https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80](https://registry.example.com/binderhub-staging/v2//manifests/8b84d28e07fe807c7b263264f0792915bf8cdc80)  
> is returning in my container registry (Docker Harbour) a HTML 404 if I try it

Does it make a difference if you remove the double `//` after `v2`?

---

<div class="post-metadata">

**Author:** ![ekasprzy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ekasprzy/32/20291_2.png) [@ekasprzy](https://discourse.jupyter.org/u/ekasprzy)\
**Post date:** [August 14, 2025, 8:13am UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/5 "2025-08-14T08:13:12Z")

</div>

Hi Simon  
Thanks for your help, I managed to resolve issue, it was an error on my side, after re-reviewing the install parameters, I noticed I missed the DockerRegistry key in the config.yaml of the helm chart:

```auto
  DockerRegistry:
    url: https://example.com
    token_url: example.com/service/token?service=harbor-registry

```

My error was that after I read in paragraph 3.2.5 of [3. Set up BinderHub — BinderHub documentation](https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/setup-binderhub.html) a reference to Docker Harbor, and I thought 3.2.5 was enough to configure registry access to my Docker Harbor.

I will see if I can make a pull request to fix the documentation in this regard, IMHO if the section in 3.4 should not be labelled as “Expand the config” because “Expand the config” would imply that the parameters mentioned there are not mandatory.

I have also seen that our helm chart for binderhub config.yaml puts the configuration in a Kubernetes Secret: this should be a ConfigMap, not a Secret, because there are no credentials in that part so it breaks the assumption Kubernetes users/admins have around the object,  
and it complicates troubleshooting for no good reason since you have to encode / decode via base64 everychange you make to the config.

I will also propose a pull request for that when I am finished with my setup.

---

<div class="post-metadata">

**Author:** ![ekasprzy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ekasprzy/32/20291_2.png) [@ekasprzy](https://discourse.jupyter.org/u/ekasprzy)\
**Post date:** [August 26, 2025, 9:19am UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/6 "2025-08-26T09:19:15Z")

</div>

Before providing a pull request for using a Kubernetes ConfigMap instead of a secret for config data, I have started with something smaller, as to cut my teeth on the binder development env:

> <https://github.com/jupyterhub/binderhub/pull/2008>
>
> When binder is installed with the helm chart, and is only exposed via a ClusterI…P Kubernetes Service, the \`kubectl\` commands of the post install notes are erroring in subtle ways:
> 
> ~~~
> kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "fullname" . }}" -o jsonpath="{.items\[0\].metadata.name}"
> ~~~
> 
> can fail because \`{{ template "fullname" . }}\` does not always match the hardcoded labels we set in https://github.com/jupyterhub/binderhub/blob/33bcc37c2fd18573d19a13c5a41b1d8d563c3663/helm-chart/binderhub/templates/deployment.yaml#L20
> 
> and
> 
> ~~~
> kubectl port-forward $POD\_NAME 8080:{{ .Values.service.externalPort }} 
> ~~~
> 
> always fails because we don't expose set or expose an externalPort field in https://github.com/jupyterhub/binderhub/blob/33bcc37c2fd18573d19a13c5a41b1d8d563c3663/helm-chart/binderhub/values.yaml#L34
> 
> This commit fixes the two errors by using immutables, non-overridables values from the helm templates.

Feel free to flame or LGTM !

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [August 26, 2025, 9:55am UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/7 "2025-08-26T09:55:45Z")

</div>

> this should be a ConfigMap, not a Secret, because there are no credentials

Before opening a pull request, I would suggest checking why this is a Secret, since this statement isn’t true in general. This used to be a configmap, but was switched to a Secret because there absolutely are credentials in some config fields. This follows a similar pattern to helm itself, which stores release state in Secrets.

---

<div class="post-metadata">

**Author:** ![ekasprzy](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ekasprzy/32/20291_2.png) [@ekasprzy](https://discourse.jupyter.org/u/ekasprzy)\
**Post date:** [August 26, 2025, 10:15am UTC](https://discourse.jupyter.org/t/self-registry-get-image-manifest-failing-after-initial-helm-chart-setup/37411/8 "2025-08-26T10:15:46Z")

</div>

> This used to be a configmap, but was switched to a Secret because there absolutely are credentials in some config fields  
> then I understand this absolutely makes sense to have the parameters in a Kubernetes secrets.