# Tip: speed up Binder launches by pulling github content in a Binder link with nbgitpuller

**URL:** <https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922>\
**Category:** discuss\
**Tags:** tip\
**Created:** [April 30, 2019, 12:44am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922 "2019-04-30T00:44:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/choldgraf/32/1264_2.png) [@choldgraf](https://discourse.jupyter.org/u/choldgraf)\
**Post date:** [April 30, 2019, 12:44am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/1 "2019-04-30T00:44:13Z")

</div>

Something people often want is to de-couple the **content** of a repository from the **environment** that is needed to run it. This would allow you to update the content of a repo without needing to re-build the Binder needed for it.

One option is to use a tool called [nbgitpuller](https://github.com/jupyterhub/nbgitpuller). This a tool for quickly pulling in GitHub content into a JupyterHub. You can create links that, when clicked, will automatically pull in content into a user’s workspace.

`nbgitpuller` lets you share a link with this structure:

```auto
<your-jupyterhub-url>/<user-server>/git-pull?repo=https://github.com/data-8/materials-fa17

```

We can take advantage of this in Binder to share a similar link with a BinderHub. Here’s how:

## Step 0: Your repository structure

In this example, we’ll have two repositories. The **environment repository** will have all the Binder configuration files to define the environment you’d like. It’s what Binder will “build”. The **content repository** will _only_ have the content you want to share, _not_ the environment files.

## Step 1: Prep your environment repository

First set up the repository with whatever environment you wish. Then, make  
sure that it works with `nbgitpuller` by following these steps:

In a `requirements.txt` file, make sure this line is there:

```auto
...
nbgitpuller
...

```

In a `postBuild` file make sure this line is there to activate `nbgitpuller`

```auto
...
jupyter serverextension enable --py nbgitpuller --sys-prefix
...

```

## Create an nbgitpuller Binder link

Next, **create a custom Binder link** that points to the content you want users to see when they click the link. To do so, use the same `nbgitpuller` syntax described above along with the BinderHub `urlpath` parameter.

```auto
https://mybinder.org/v2/gh/<your-username>/<your-environment-repo>/master?urlpath=git-pull?repo=<url-of-your-content-repo>

```

## For example

For example, here’s a Binder link that uses my `binder-sandbox` repository to define the environment, and that pulls in the content from the data-8 Fall 2017 course:

> **[GitHub: choldgraf/binder-sandbox/master](https://mybinder.org/v2/gh/choldgraf/binder-sandbox/master?urlpath=git-pull?repo=https:%2F%2Fgithub.com%2Fdata-8%2Fmaterials-fa17)**
>
> Click to run this interactive environment. From the Binder Project: Reproducible, sharable, interactive computing environments.

You can use any URL you’d like for the “content repository”!

## That’s it!

This should let you share Binder links that all have the same environment, but that serve users _arbitrary_ content!

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 1, 2019, 6:27pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/2 "2019-05-01T18:27:50Z")

</div>

This is really neat… but a little arcane, perhaps, and _yet another thing to remember_™?

I could imagine a separate UI for this - “Binder environments”, maybe? — where you have a form that lets you specify:

- a build repository
- a content repository.

The build repository is the one that gets run through repo2docker; the content repository is the one that gets gitpulled.

Adding more form elements and switches to the current page would just overcomplicate the logic of the page and its appearance to users?

A “Binder environments” page might also specify various default build repos (cf the Jupyter stacks, but you could maybe have base containers for different subject areas: stats, chemistry, astronomy, cartography etc)?

---

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/choldgraf/32/1264_2.png) [@choldgraf](https://discourse.jupyter.org/u/choldgraf)\
**Post date:** [May 2, 2019, 2:18am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/3 "2019-05-02T02:18:37Z")

</div>

Totally - this post was totally a “Chris just realized this was possible and wanted to write down how to do it because otherwise he’d forget” post 🙂

It’d be cool to prototype UI for this kinda thing in an “unofficial” sense. E.g., you could whip together whatever HTML+JS you want that’d build the proper Binder URL to launch links like this (though you’d need to ensure that the build images had nbgitpuller)

also re: base images and stacks, you may also be interested in this issue:

> <https://github.com/jupyter/repo2docker/issues/487>

would love your thoughts!

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 2, 2019, 11:38am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/4 "2019-05-02T11:38:11Z")

</div>

Could / should repo2docker inject nbgitpuller to the build ? MyBinder could detect it in the calling URL, but how would that propagate? Can repo2docker force additional requirements into a build eg via a command line argument?

The user defined base image for repo2docker is interesting, but what then happens to all the things that repo2docker might normally add in to the container on top of the base container. Would it continue to do that? Should it?

How would changing the workflow to use a different base container compare with using a custom Dockerfile in `binder/`?

---

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/choldgraf/32/1264_2.png) [@choldgraf](https://discourse.jupyter.org/u/choldgraf)\
**Post date:** [May 2, 2019, 2:43pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/5 "2019-05-02T14:43:36Z")

</div>

A few thoughts there:

> Could / should repo2docker inject nbgitpuller to the build ? MyBinder could detect it in the calling URL, but how would that propagate? Can repo2docker force additional requirements into a build eg via a command line argument?

I think if it’s both lightweight and a common-enough use-case, we could make a case for it. For now I think it’s too uncommon to justify adding it in there by default, just my 2 cents tho.

> The user defined base image for repo2docker is interesting, but what then happens to all the things that repo2docker might normally add in to the container on top of the base container. Would it continue to do that? Should it?

It would - the idea is just that you can configure what the starting point is, but the r2d machinery is the same on top of it.

> How would changing the workflow to use a different base container compare with using a custom Dockerfile in `binder/` ?

The goal of “start with a custom Docker image” is more for administrators to choose this rather than users. E.g. if I set up a BinderHub aimed at the GeoSciences, I can choose a base image that has a few very common packages installed, then either a lot of users don’t need to install them, or it’s a much shorter process.

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 2, 2019, 3:12pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/6 "2019-05-02T15:12:52Z")

</div>

> [@choldgraf](#):
>
> E.g. if I set up a BinderHub aimed at the GeoSciences, I can choose a base image that has a few very common packages installed, then either a lot of users don’t need to install them, or it’s a much shorter process.

Right… the repos I branched [here](https://github.com/psychemedia/showntell) are variously flavoured in the builds for demo packages in different subject areas. Putting all the dependencies from the different subject areas into a single Binder image would have made for a very large image and dependency management hell…

---

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/choldgraf/32/1264_2.png) [@choldgraf](https://discourse.jupyter.org/u/choldgraf)\
**Post date:** [May 2, 2019, 3:27pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/7 "2019-05-02T15:27:11Z")

</div>

for sure - this I don’t think would ever be deployed for [mybinder.org](http://mybinder.org), it’s more for people deploying group-specific BinderHubs where a lot more assumptions can be made about the space of possible packages needed

---

<div class="post-metadata">

**Author:** ![yuvipanda](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yuvipanda/32/17_2.png) [@yuvipanda](https://discourse.jupyter.org/u/yuvipanda)\
**Post date:** [May 2, 2019, 5:09pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/8 "2019-05-02T17:09:11Z")

</div>

This is great! Thanks for digging this up, Chris.

> [@psychemedia](#):
>
> I could imagine a separate UI for this - “Binder environments”, maybe? — where you have a form that lets you specify:
> 
> - a build repository
> - a content repository.

There’s an nbgitpuller link generator at [https://jupyterhub.github.io/nbgitpuller/link](https://jupyterhub.github.io/nbgitpuller/link), and it just got a canvas option. We could add a [mybinder.org](http://mybinder.org) option?

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 2, 2019, 5:12pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/9 "2019-05-02T17:12:01Z")

</div>

> [@yuvipanda](#):
>
> nbgitpuller link generator

Ooh… that looks handy…

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/jupyter/original/1X/82751e98fa1d3ee6bc832c46bddb9322b0748830.png)

---

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/choldgraf/32/1264_2.png) [@choldgraf](https://discourse.jupyter.org/u/choldgraf)\
**Post date:** [May 2, 2019, 5:30pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/10 "2019-05-02T17:30:24Z")

</div>

I was thinking the same thing actually!

---

<div class="post-metadata">

**Author:** ![betatim](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/betatim/32/10_2.png) [@betatim](https://discourse.jupyter.org/u/betatim)\
**Post date:** [May 2, 2019, 8:28pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/11 "2019-05-02T20:28:10Z")

</div>

> [@psychemedia](#):
>
> The user defined base image for repo2docker is interesting, but what then happens to all the things that repo2docker might normally add in to the container on top of the base container. Would it continue to do that? Should it?

I think we have consensus on what “custom base images in repo2docker” should look like: [Make it possible to configure the base image · Issue #487 · jupyterhub/repo2docker · GitHub](https://github.com/jupyter/repo2docker/issues/487#issuecomment-479794426) and the next few comments after the linked one (plus really the whole thread for several options that were considered, trade-offs, prototypes and alternatives. I like the converged upon idea and would recommend we try to implement it before re-opening the discussion. Otherwise we spend forever talking and not so much doing. We aren’t yet a committee 🙂

I’d be -1 on adding nbgitpuller to repo2docker because it is niche. A constant battle is the perception that repo2docker created images contain “bloat” or “for real uses cases one needs a custom `Dockerfile` to remove bloat” etc etc. So we should make an effort to keep thing slim (because they are!) which means only adding things to core repo2docker that are used very widely, even if (like nbgitpuller) they don’t actually increase the image size all that much. Instead more documentation and “cookie cutter” repos specialised to these use cases.

Those are my thoughts on how to address this.

On a more positive note: having new, other, more user interfaces and user experiences for how to create your “binder link” is very cool. They can and should be hosted/built separately of BinderHub. The fact that it doesn’t need the central oversight committee to agree to any of it is a feature 🙂

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 3, 2019, 9:05am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/12 "2019-05-03T09:05:37Z")

</div>

Just caught up with that thread: extensions / plugins for `repo2docker`, brilliant. Makes for easier community contributions. Here’s a related line of thinking from Simon Willison on [datasette plugins](https://github.com/simonw/datasette/issues/14).

---

<div class="post-metadata">

**Author:** ![betatim](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/betatim/32/10_2.png) [@betatim](https://discourse.jupyter.org/u/betatim)\
**Post date:** [May 3, 2019, 12:02pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/13 "2019-05-03T12:02:51Z")

</div>

Thanks for the link to datasette. Simon seems to have settled on pluggy/likes it so I am adding that to my list of things to checkout. I really like the idea of (one day) having something like [plugincompat.herokuapp.com](http://plugincompat.herokuapp.com) for repo2docker.

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 3, 2019, 12:13pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/14 "2019-05-03T12:13:44Z")

</div>

datasette also has a range of tools for packaging and deploying datasettes: [`datasette publish`](https://datasette.readthedocs.io/en/stable/publish.html). There are various issues in the repo where Simon runimated on various aspects of this I think.

The use case is much more limited / constrained than `repo2docker`, but again, ways of doing things and if different ecosystems merge on good practice or common utils, that’s handy for future projects. (eg datasette uses click for command line interface IIRC).

_Apols if this is a distraction; I try to make sense of Jupyter / where it might be developing, by trying to make sense of it in context of other things I don’t really understand either!😉_

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 7, 2019, 7:02pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/15 "2019-05-07T19:02:11Z")

</div>

@choldgraf Just been playing with this quickly and it’s absolutely bonkers in a brilliant way:-)

eg [https://mybinder.org/v2/gh/ouseful-testing/binder-graphviz/master/?urlpath=git-pull?repo=https://github.com/hchasestevens/show\_ast](https://mybinder.org/v2/gh/ouseful-testing/binder-graphviz/master/?urlpath=git-pull?repo=https://github.com/hchasestevens/show_ast)

eg provides a mechanism for showing repo maintainers how their repo looks in mybinder, and can also be used to demo requirements for making it runnable ex- of their repo and ex- of requiring a PR on it.

Reminds me of URL hacking to chain different things across different APIs. Can it also accept a redirect to open into a specified notebook?

@yuvipanda Adding a tab for Binderhub to the `nbgitpuller` link generator would be really useful I think… especially _if_ folk found out about it…

---

<div class="post-metadata">

**Author:** ![betatim](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/betatim/32/10_2.png) [@betatim](https://discourse.jupyter.org/u/betatim)\
**Post date:** [May 8, 2019, 8:20am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/16 "2019-05-08T08:20:20Z")

</div>

> [@psychemedia](#):
>
> @choldgraf Just been playing with this quickly and it’s absolutely bonkers in a brilliant way:-)
> 
> eg [https://mybinder.org/v2/gh/ouseful-testing/binder-graphviz/master/?urlpath=git-pull?repo=https://github.com/hchasestevens/show\_ast](https://mybinder.org/v2/gh/ouseful-testing/binder-graphviz/master/?urlpath=git-pull?repo=https://github.com/hchasestevens/show_ast)

This is pretty wild 😃 So wild we should publicise it more!

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 8, 2019, 11:08am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/17 "2019-05-08T11:08:56Z")

</div>

I’m wondering as well as `nbgitpuller`, there could be a generic (curl, wget etc) pull that could pull eg data files from a URL?

---

<div class="post-metadata">

**Author:** ![betatim](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/betatim/32/10_2.png) [@betatim](https://discourse.jupyter.org/u/betatim)\
**Post date:** [May 8, 2019, 12:21pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/18 "2019-05-08T12:21:29Z")

</div>

> [@psychemedia](#):
>
> I’m wondering as well as `nbgitpuller` , there could be a generic (curl, wget etc) pull that could pull eg data files from a URL?

Not sure what I think about making it possible to fetch arbitrary URLs. For getting data my personal view is that one should use [GitHub - binder-examples/getting-data: How to get data into your Binder](https://github.com/binder-examples/getting-data#how-to-get-data-into-your-binder).

One thing that feels uncomfortable to me with my mybinder.org-operator hat on is that if we let people construct URLs that make [mybinder.org](http://mybinder.org) take action via its high bandwidth connection we become a more attractive target for being the source of a DOS. You can trigger a [mybinder.org](http://mybinder.org) launch from a very slow dial up connection and if that launch then starts a (very large) download you nnow have a way to amplify your impact. (This is probably also true for letting people pull from git repos and generally true if we let people perform outgoing network connections from within [mybinder.org](http://mybinder.org) but it still feels like making it “too easy” to do :-/)

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [May 8, 2019, 12:28pm UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/19 "2019-05-08T12:28:45Z")

</div>

Ah, right… yes… understood. Other factors… ☹

---

<div class="post-metadata">

**Author:** ![psychemedia](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/psychemedia/32/24_2.png) [@psychemedia](https://discourse.jupyter.org/u/psychemedia)\
**Post date:** [June 10, 2019, 11:06am UTC](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922/20 "2019-06-10T11:06:01Z")

</div>

Pondering this a bit more eg in context of [this repo](https://github.com/colinleach/astro-Jupyter) which is gitpulled into [this Binder build](https://github.com/colinleach/binder-box) ([discussion](https://github.com/colinleach/astro-Jupyter/issues/2)), I wonder…

Github conventionally uses the `gh-pages` branch as a “reserved” branch for constructing Github Pages docs related to a particular repo.

The `binder/` directory in a repo can be used to partition Binder build requirements in a repo, but there are a couple of problems associated with this:

- a maintainer may not want to have the `binder/` directory cluttering their package repo;
- any updates to the repo will force a rebuild of the Binder image next time the repo is run on a particular Binder node. (With Binder federation, if there are N hosts in the federation, after updating a repo, is it possible that my next N attempts to run the repo on MyBinder may require a rebuild if I am directed to a different host each time?)

If by convention something like a `binder-build` branch was used to contain the build requirements for a repo, then the process for calling a build (by default) could be simplified.

Eg rather than having something like:

[https://mybinder.org/v2/gh/colinleach/binder-box/master/?urlpath=git-pull?repo=https://github.com/colinleach/astro-Jupyter](https://mybinder.org/v2/gh/colinleach/binder-box/master/?urlpath=git-pull?repo=https://github.com/colinleach/astro-Jupyter)

we would have something like:

[https://mybinder.org/v2/gh/colinleach/astro-Jupyter/binder-build/?urlpath=git-pull?repo=https://github.com/colinleach/astro-Jupyter](https://mybinder.org/v2/gh/colinleach/astro-Jupyter/binder-build/?urlpath=git-pull?repo=https://github.com/colinleach/astro-Jupyter)

which could simplify to something that defaults to a build from `binder-build` branch (the “build” branch) and `nbgitpull` from `master` (the “content” branch):

[https://mybinder.org/v2/gh/colinleach/astro-Jupyter?binder-build=True](https://mybinder.org/v2/gh/colinleach/astro-Jupyter?binder-build=True)

Complications could be added to support changing the build branch, the `nbgitpull` branch, the commit/ID of a particular build, etc?

It might overly complicate things further, but I could also imagine:

- automatically injecting `nbgitpuller` into the Binder image and enabling it;
- providing some sort of directive support so that if the content directory has a `setup.py` file the package from that content directory is installed.

[Next page](https://discourse.jupyter.org/t/tip-speed-up-binder-launches-by-pulling-github-content-in-a-binder-link-with-nbgitpuller/922.md?page=2)