Pre loading package on binder with start

Binder
1

Hello all, the doc of our open source solver is using binder to provide example notebooks.
Everything is working fine so far (with an apt.txt , an environment.yml to install the dependencies that change rarely and postBuild for releases which install the wheel of our library).
We have recently introduced nightly distributions and we would like the documentation for the master branch to use a binder using the nightlys.
My idea was to use start to dynamically download a zip from github (where the nightlys are stored) and install it.
Does that seem workable ?
And if yes would someone have an example of start script … mine is not working :frowning:
Thanks for your help

2

Yes, and yes.

3 Likes
3

:pray: and :pray:

Thanks my start file was already quite similar but I still have an issue.
Everything seems to go fine, each layer to be built … but I have this error:

2021-12-17T07:57:21Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-n655.hfe65496" already present on machine
2021-12-17T07:57:22Z [Normal] Created container tc-init
2021-12-17T07:57:22Z [Normal] Started container tc-init
2021-12-17T07:57:22Z [Normal] Pulling image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-galleon-2dscikit-2ddecide-e69c4e:e9c9012dcb052e24a1154fc811beb68638c5d4c8"
2021-12-17T07:58:11Z [Normal] Successfully pulled image "gcr.io/binderhub-288415/r2d-staging-g5b5b759-galleon-2dscikit-2ddecide-e69c4e:e9c9012dcb052e24a1154fc811beb68638c5d4c8" in 49.027263279s
2021-12-17T07:58:14Z [Normal] Created container notebook
2021-12-17T07:58:14Z [Normal] Started container notebook
Spawn failed: Server at http://10.0.14.69:8888/user/galleon-scikit-decide-ys3h21m8/ didn't respond in 30 seconds
Launch attempt 3 failed, retrying...

How can I debug this? Did I miss something?

Thanks

4

yep, this is a danger of hacking on start, as you won’t get any help from docker for it getting cached (even though it will re-run postBuild every change). Without seeing more of the start script, there won’t be much to see.

Another approach, if your system package is used only in a kernel, and not in the jupyter server, would be to write a tiny, demo-only config that installed stuff during startup. This is totally abuse of the config system, but heck, this is binder. You can also use some pip flags to limit the blast radius of a hot install in an otherwise working system.

# .binder/jupyter_server_config.py

import subprocess, sys, pathlib
error = none
try:
    p = subprocess.Popen([
        sys.executable, "-m", "pip", "install", 
        "--upgrade", "--ignore-installed", "--no-deps", 
        "-r", ".binder/nightly-requirements.txt"
    ], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    p.wait()
except Exception as observed_error:
    error = observed_error
finally:
   with(".pip-install.log", "w+").open() as fd:
       fd.write(p.stdout)
       fd.write(p.stderr)
       fd.write(str(error))

Being able to use more familiar error handling and logging than some of the “special” features on binder, might better allow local debugging.

1 Like
5

I just forked your repo to investigate your start file. Mistakenly I didn’t push to the branch and launched anyway. (In other words, just forked yours and did a new build with no changes.) My fork launched fine though? Did you resolve your issue? Or at least find the offending line(s)?
You can try it at https://mybinder.org/v2/gh/fomightez/scikit-decide/gal/binder.

The repo is at https://github.com/fomightez/scikit-decide/blob/gal/binder/binder/start

1 Like
6

Hehe, Thanks for investigating … Yes I played a lot commenting lines … and I found the culprit !
The clone you are using is the one working :wink: and the line I would like to uncomment is line 27

# pip install --pre --find-links ./dist/ "scikit-decide[all]"

If is it uncommented I hit in the 30s non response time …

May be I should try to pre-install as much dependencies as possible (but I am always at risk that they might change) ?

Any other option?

7

Thank you all for your help.
I moved all I could move (dependencies) in environment.yml and finally it works :man_dancing:
I would nevertheless be interested by any feedback you can have !

Thank you
Guillaume

1 Like