Image build crashes while pushing layers

Binder mybinder.org ops
1

Dear experts,
I’m trying to spawn a session based on a git repo, but I get a “Failed” error while repo2docker is pushing the layers.
Attaching the log below.
Do you have any idea of why this happens?

Thanks a lot!

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2dockergl4onoxb'...
HEAD is now at 480fb4b update compatibility release for coffea
Building conda environment for python=3.11
Using CondaBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 5.26kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 DONE 0.7s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [ 1/18] FROM docker.io/library/buildpack-deps:jammy@sha256:1a63cc50ec6f4f45440121af59c47f78ca390607714ac0886c1588d118526b40
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 51.50MB 0.4s done
#5 DONE 0.5s

#6 [ 2/18] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#6 CACHED

#7 [ 8/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-637204 /tmp/install-base-env.bash
#7 CACHED

#8 [ 6/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e67d51 /etc/profile.d/activate-conda.sh
#8 CACHED

#9 [ 5/18] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        gettext-base        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#9 CACHED

#10 [13/18] RUN chown jovyan:jovyan /home/jovyan
#10 CACHED

#11 [12/18] WORKDIR /home/jovyan
#11 CACHED

#12 [10/18] RUN mkdir -p /srv/npm && chown -R jovyan:jovyan /srv/npm
#12 CACHED

#13 [ 9/18] RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
#13 CACHED

#14 [11/18] RUN if [ ! -d "/home/jovyan" ]; then         /usr/bin/install -o jovyan -g jovyan -d "/home/jovyan";     fi
#14 CACHED

#15 [14/18] COPY --chown=1000:1000 src/binder/environment.yml /home/jovyan/binder/environment.yml
#15 CACHED

#16 [ 4/18] RUN groupadd         --gid 1000         jovyan &&     useradd         --comment "Default user"         --create-home         --gid 1000         --no-log-init         --shell /bin/bash         --uid 1000         jovyan
#16 CACHED

#17 [ 3/18] RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
#17 CACHED

#18 [ 7/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e11-2dlinux-2d64-2elock-2bb04a /tmp/env/environment.lock
#18 CACHED

#19 [15/18] RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "binder/environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '
#19 CACHED

#20 [16/18] COPY --chown=1000:1000 src/ /home/jovyan/
#20 DONE 0.5s

#21 [17/18] COPY /python3-login /usr/local/bin/python3-login
#21 DONE 0.1s

#22 [18/18] COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
#22 DONE 0.1s

#23 exporting to image
#23 exporting layers
#23 exporting layers 0.3s done
#23 writing image sha256:5b213548b70f7aacf949e2e3a1b86bc2e9f6335ae9a70613cde8d30b5822b982 done
#23 naming to registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60 done
#23 DONE 0.3s

#24 pushing registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60 with docker
#24 pushing layer 2747f4af6b8b
#24 pushing layer 6f6dd9ef05aa
#24 pushing layer fbff668d81e2
#24 pushing layer aa6db46f3737
#24 pushing layer 65755242094d
#24 pushing layer 1a8fef2709e7
#24 pushing layer 5f70bf18a086
#24 pushing layer 2ef9b3b1f242
#24 pushing layer d782d5201cca
#24 pushing layer 227058c97c1c
#24 pushing layer e764bf2f319f
#24 pushing layer 58054042142d
#24 pushing layer e661d9cf12f9
#24 pushing layer 4a46c519ba72
#24 pushing layer 03a32a93505d
#24 pushing layer 65283cd2cdc8
#24 pushing layer a3e1a9f8dd1f
#24 pushing layer 3088d1636682
#24 pushing layer b7d4ab99ea66
#24 pushing layer 270a1170e7e3
#24 pushing layer 2747f4af6b8b 512B / 3.36kB 0.6s
#24 pushing layer 6f6dd9ef05aa 512B / 374B 0.7s
#24 pushing layer 2747f4af6b8b 6.66kB / 3.36kB 0.7s
#24 pushing layer 6f6dd9ef05aa 3.58kB / 374B 0.7s
#24 pushing layer fbff668d81e2 4.72MB / 51.44MB 0.9s
#24 pushing layer fbff668d81e2 8.91MB / 51.44MB 1.2s
#24 pushing layer fbff668d81e2 13.11MB / 51.44MB 1.4s
#24 pushing layer fbff668d81e2 18.35MB / 51.44MB 1.8s
#24 pushing layer fbff668d81e2 21.50MB / 51.44MB 2.0s
#24 pushing layer fbff668d81e2 25.21MB / 51.44MB 2.3s
#24 pushing layer fbff668d81e2 28.39MB / 51.44MB 2.5s
#24 pushing layer fbff668d81e2 31.55MB / 51.44MB 2.8s
#24 pushing layer fbff668d81e2 34.69MB / 51.44MB 3.1s
#24 pushing layer 6f6dd9ef05aa 3.2s done
#24 pushing layer 2747f4af6b8b 3.4s done
#24 pushing layer fbff668d81e2 38.38MB / 51.44MB 3.9s
#24 pushing layer fbff668d81e2 41.00MB / 51.44MB 4.4s
#24 pushing layer fbff668d81e2 43.62MB / 51.44MB 4.9s
#24 pushing layer aa6db46f3737 48.11MB / 3.65GB 5.0s
#24 pushing layer fbff668d81e2 46.24MB / 51.44MB 5.4s
#24 pushing layer fbff668d81e2 51.57MB / 51.44MB 5.9s
#24 pushing layer aa6db46f3737 95.63MB / 3.65GB 10.1s
#24 pushing layer fbff668d81e2 12.0s done
#24 pushing layer aa6db46f3737 176.59MB / 3.65GB 15.3s
...
#24 pushing layer aa6db46f3737 1.77GB / 3.65GB 157.2s
2

This is what I am seeing when I try that provided launch link now:

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2docker3um6mdtv'...
HEAD is now at 480fb4b update compatibility release for coffea
Reusing existing image (registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60), not building.Built image, launching...
Launching server...
Server requested
2025-03-24T17:24:05.717723Z [Normal] Successfully assigned hetzner-2i2c/jupyter-atlas-outreach-ec--x---dad890f8 to mybinder-01
2025-03-24T17:24:06Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.5524.ha41b617b" already present on machine
2025-03-24T17:24:06Z [Normal] Created container tc-init
2025-03-24T17:24:06Z [Normal] Started container tc-init
2025-03-24T17:24:07Z [Normal] Pulling image "registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60"
2025-03-24T17:24:07Z [Warning] Failed to pull image "registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60": rpc error: code = NotFound desc = failed to pull and unpack image "registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60": failed to resolve reference "registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60": registry.2i2c.mybinder.org/i-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60: not found
2025-03-24T17:24:07Z [Warning] Error: ErrImagePull

I will say though that I saw something like this just the end of last week with a repo I was developing. During a build or pushing laters, MyBinder had gone down entriely. And shortly after that I was seeing something like I see with yours now, specifically launch mostly seemed to go normal but then cease with Error: ErrImagePull. No matter what, the launch kept getting stuck there in all subsequent attempts.

I happened to be working on things at that time and wanted to make additional changes anyway, and so later when I pushed new changes the next commit was back to working to launch after uninterrupted build and push steps.

So my question is have you verified that MyBinder was working and then pushed a change and tried the new commit?

UPDATE:

I deleted my old fork with the intention of trying my suggestion on my own fork; however, there is another current issue now with MyBinder that it thwarting my attempt. (See below.)

FURTHER UPDATE:

I am chasing an unrelated issue at this point:

Nothing builds from scratch right now that I can find.

I just used the binder-examples/conda as a template.
Here is the result: repo
Launch that should work but doesn’t:
Binder

I see the following:

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2dockerbdn_v8ix'...
HEAD is now at 1795a68 Initial commit
Building conda environment for python=3.11
Using CondaBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 5.21kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 DONE 0.4s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [ 1/18] FROM docker.io/library/buildpack-deps:jammy@sha256:1a63cc50ec6f4f45440121af59c47f78ca390607714ac0886c1588d118526b40
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 100.84kB 0.0s done
#5 DONE 0.0s

#6 [ 9/18] RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
#6 CACHED

#7 [ 3/18] RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
#7 CACHED

#8 [10/18] RUN mkdir -p /srv/npm && chown -R jovyan:jovyan /srv/npm
#8 CACHED

#9 [ 7/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e11-2dlinux-2d64-2elock-2bb04a /tmp/env/environment.lock
#9 CACHED

#10 [ 4/18] RUN groupadd         --gid 1000         jovyan &&     useradd         --comment "Default user"         --create-home         --gid 1000         --no-log-init         --shell /bin/bash         --uid 1000         jovyan
#10 CACHED

#11 [ 2/18] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#11 CACHED

#12 [11/18] RUN if [ ! -d "/home/jovyan" ]; then         /usr/bin/install -o jovyan -g jovyan -d "/home/jovyan";     fi
#12 CACHED

#13 [ 8/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-637204 /tmp/install-base-env.bash
#13 CACHED

#14 [ 6/18] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e67d51 /etc/profile.d/activate-conda.sh
#14 CACHED

#15 [12/18] WORKDIR /home/jovyan
#15 CACHED

#16 [ 5/18] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        gettext-base        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#16 CACHED

#17 [13/18] RUN chown jovyan:jovyan /home/jovyan
#17 CACHED

#18 [14/18] COPY --chown=1000:1000 src/environment.yml /home/jovyan/environment.yml
#18 ERROR: failed to prepare v8jkd17xl6034sts0nfurf5yb as ql8to62cdja5tknzhhgsyt3sq: open /var/lib/docker/overlay2/0vsujg8ls01jh8dxzux42w6jn/.tmp-committed3464481049: no such file or directory
------
 > [14/18] COPY --chown=1000:1000 src/environment.yml /home/jovyan/environment.yml:
------
Dockerfile:116
--------------------
 114 |     # If scripts required during build are present, copy them
 115 |     
 116 | >>> COPY --chown=1000:1000 src/environment.yml ${REPO_DIR}/environment.yml
 117 |     USER ${NB_USER}
 118 |     RUN TIMEFORMAT='time: %3R' \
--------------------
ERROR: failed to solve: failed to prepare v8jkd17xl6034sts0nfurf5yb as ql8to62cdja5tknzhhgsyt3sq: open /var/lib/docker/overlay2/0vsujg8ls01jh8dxzux42w6jn/.tmp-committed3464481049: no such file or directory
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'registry.2i2c.mybinder.org/i-fomightez-2dtest-5fmar-5f24-5f2025-29f7a5:1795a68ab169b76f925dc9086ecee217fb5f7130', '--platform', 'linux/amd64', '/tmp/tmprq2038ii']' returned non-zero exit status 1.Error in event stream: Error

I reported it over in a Zulip discussion about an outage this morning just after Michał Krassowski has also reported seeing the same thing.

3

Hey @fomightez, thanks a lot for the quick reaction!
Upon retrying immediately after failing to build, I had the same message you showed about ErrImagePull.
It looks like after a while existing (and broken?) images gets purged; I waited ~1h and tried again, and I got again in the building phase which failed (again).

However, I’ve tried to build the image also in other places (e.g. Binder from GESIS)
Now if I use the same link I’ve attached before, I get:

Found built image, launching...
Launching server...
Server requested
2025-03-25T08:47:11.985235Z [Normal] Successfully assigned gesis/jupyter-atlas-outreach-ec--x---a8db0152 to spko-css-app03
2025-03-25T08:47:12Z [Normal] Container image "quay.io/jupyterhub/k8s-network-tools:4.0.0" already present on machine
2025-03-25T08:47:12Z [Normal] Created container: block-cloud-metadata
2025-03-25T08:47:12Z [Normal] Started container block-cloud-metadata
2025-03-25T08:47:13Z [Normal] Pulling image "gesiscss/binder-r2d-g5b5b759-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60"
Spawn failed: pod gesis/jupyter-atlas-outreach-ec--x---a8db0152 did not start in 300 seconds!
Launch attempt 1 failed, retrying...
Server requested
2025-03-25T08:56:31.103956Z [Normal] Successfully assigned gesis/jupyter-atlas-outreach-ec--x---ab2084fe to spko-css-app03
2025-03-25T08:56:31Z [Normal] Container image "quay.io/jupyterhub/k8s-network-tools:4.0.0" already present on machine
2025-03-25T08:56:31Z [Normal] Created container: block-cloud-metadata
2025-03-25T08:56:32Z [Normal] Started container block-cloud-metadata
2025-03-25T08:56:32Z [Normal] Pulling image "gesiscss/binder-r2d-g5b5b759-atlas-2doutreach-2ddata-2dtools-2dnotebooks-2dcollection-2dopendata-40f4f5:480fb4b1edef0a4d7b79f085ac8bc8fcbf60ce60"
4

Yes, there were more issues after that one that were being discussed in the Zulip conversation.
All seem resolved for now.
Importantly, that link launches just fine for me at the present time.

When you are having issues, it is always best to try some independent repo. Preferably one that needs to build if you are stuck at the building step. Or likewise for different stages. That way you aren’t thinking your repo is the problem when it is a more general issue. (See below for some tips about this.)
The infrastructure running MyBinder has a different approach of late and they are working out some of the kinks still with running in a new way on a new resource, see here and here for just an indication of what is happening. There’s been some bumps the last few weeks; however, Yuvi is tackling them pretty quickly and all combined it should make it more resilient.

Plus, now you know if things go out in the middle of a launch and then you see the Error: ErrImagePull to wait or try a new commit.

Tips for testing independent repos:

For finding places to try, beyond the examples at the main github, there’s a good list in the gists of Jeremy Tuloup. The ones that Jeremy lists tend to get less used than the main examples and so you may have better luck finding one that matches your state of buildable/stored layers/ launching well.

5

Hey @fomightez,
Many thanks for the very helpful insights!
Will follow the tips in the future!

1 Like