Containers created, but spawn fails on mybinder.org

Binder mybinder.org ops
1

We use mybinder.org as a backup scaffold for learners in our workshops if they do not have the ability to install the required software in time. We have had a recent report that the instances are failing to spawn: swc-shell-novice scaffolds fail to build in binder · Issue #7 · carpentries/scaffolds · GitHub

Is there any indication of what could be going on?

This is an example of some of the output:

Found built image, launching...
Launching server...
Server requested
2021-10-19T15:37:46.813352Z [Normal] Successfully assigned prod/jupyter-carpentries-2dscaffolds-2dniahnsqi to gke-prod-user-202009-b9c03ca0-6qx4
2021-10-19T15:37:47Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-n655.hfe65496" already present on machine
2021-10-19T15:37:47Z [Normal] Created container tc-init
2021-10-19T15:37:47Z [Normal] Started container tc-init
2021-10-19T15:37:48Z [Normal] Pulling image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc"
2021-10-19T15:38:16Z [Normal] Successfully pulled image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc" in 27.748927076s
2021-10-19T15:38:16Z [Normal] Created container notebook
2021-10-19T15:38:16Z [Normal] Started container notebook
Spawn failed: pod prod/jupyter-carpentries-2dscaffolds-2dniahnsqi did not start in 300 seconds!
Launch attempt 1 failed, retrying...
Server requested
2021-10-19T15:42:45.713490Z [Normal] Successfully assigned prod/jupyter-carpentries-2dscaffolds-2dnftf9xuv to gke-prod-user-202009-b9c03ca0-ku80
2021-10-19T15:42:47Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-n655.hfe65496" already present on machine
2021-10-19T15:42:47Z [Normal] Created container tc-init
2021-10-19T15:42:47Z [Normal] Started container tc-init
2021-10-19T15:42:48Z [Normal] Pulling image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc"
2021-10-19T15:43:16Z [Normal] Successfully pulled image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc" in 27.769661977s
2021-10-19T15:43:19Z [Normal] Created container notebook
2021-10-19T15:43:19Z [Normal] Started container notebook
Spawn failed: pod prod/jupyter-carpentries-2dscaffolds-2dnftf9xuv did not start in 300 seconds!
Launch attempt 2 failed, retrying...
Server requested
2021-10-19T15:47:32.330215Z [Normal] Successfully assigned prod/jupyter-carpentries-2dscaffolds-2dp957w176 to gke-prod-user-202009-b9c03ca0-nrh6
2021-10-19T15:47:33Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-n655.hfe65496" already present on machine
2021-10-19T15:47:33Z [Normal] Created container tc-init
2021-10-19T15:47:33Z [Normal] Started container tc-init
2021-10-19T15:47:33Z [Normal] Pulling image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc"
2021-10-19T15:47:51Z [Normal] Successfully pulled image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc" in 17.768284863s
2021-10-19T15:47:51Z [Normal] Created container notebook
2021-10-19T15:47:52Z [Normal] Started container notebook
Spawn failed: Timeout
Launch attempt 3 failed, retrying...
Server requested
2021-10-19T15:52:48.510152Z [Normal] Successfully assigned prod/jupyter-carpentries-2dscaffolds-2dc4bkd2b3 to gke-prod-user-202009-b9c03ca0-nrh6
2021-10-19T15:52:49Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-n655.hfe65496" already present on machine
2021-10-19T15:52:49Z [Normal] Created container tc-init
2021-10-19T15:52:49Z [Normal] Started container tc-init
2021-10-19T15:52:50Z [Normal] Container image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-carpentries-2dscaffolds-a0305a:264cd6761c4079de4ee2aea4779b5a66b26564bc" already present on machine
2021-10-19T15:52:50Z [Normal] Created container notebook
2021-10-19T15:52:50Z [Normal] Started container notebook
Spawn failed: pod prod/jupyter-carpentries-2dscaffolds-2dc4bkd2b3 did not start in 300 seconds!
Spawn failed: pod prod/jupyter-carpentries-2dscaffolds-2dc4bkd2b3 did not start in 300 seconds!
2

I forked it and did some stuff and it works.

Launch from here
(URL I’m using for launch is https://mybinder.org/v2/gh/fomightez/scaffolds/swc-shell-novice)

Updated version (my fork) at GitHub - fomightez/scaffolds at swc-shell-novice

So far I did two things:

  • deleted postBuild as the Table of Contents (TOC) extension is no longer separate and is a feature built-in now, and so installing that was unnecessary. That was the only thing the postBuild did, and so I removed it.
  • replaced start file with one of mine that I knew worked and then replaced contents with all but last line of swc-shell-novice’s start file. The last line didn’t 100% match the recommended format and the icon my system used to represent it with looked different than working ones I had on my local machine, and so I just thought I’d swap and then replace all but the last line with your contents.

So the question becomes did one of those two things fix it, or rebuilding fix it?

As a test, I changed the last line of the start file back to have eval "$@" and then gets the reported issue where it fails to spawn again. So I think that was the issue. You now need exec "$@" there, and cannot substitute eval for exec at present.
For easy copying, the following is good as last line of a start file:

exec "$@"
1 Like