I’ve been brainstorming with a few others on ways to link JupyterHub/BinderHub with a web-based data repository (in this case https://idr.openmicroscopy.org/ though the pattern is useful for other repositories). For example:
User finds a dataset or item of interest in the repository
That item is associated with one or more recommended notebooks.
User clicks a mybinder.org link associated with the item that launches the notebook, configured to run the analysis on the selected item. In the case of IDR this link would be in the right-hand pane of the dataset browser along with other metadata associated with the dataset or image.
Assume the notebook in step 3 is generic and can work with multiple datasets/items- e.g. a notebook to calculate image statistics, or to fetch metadata, can be run on any image. It therefore needs a parameter identifying the image to analyse.
A user can obviously edit the parameter, or jupyter-widgets can be used to request input, but this requires the user to copy and paste an ID from the data browser to the notebook in mybinder.
What would be really nice is to have a way to pass those parameters as a URL query. For example, instead of
These links would be auto-generated for each item.
It’s partly related to what Papermill can do on the CLI, but in the browser. Is the best way to implement this a Jupyter notebook/lab extension, or are there better ways?
If I understand you correctly you want to add parameters for the notebook, not the spawner (or BinderHub itself). This is what makes it a bit different from the issue I linked.
There are (at least?) two things to solve:
recognise that the query parameters aren’t meant for the spawner and pass them on
use a notebook to extract them from the final URL
No good idea right now but wanted to link the existing discussion.
One exciting but likely impractical idea: run a git repository server where https://gitrepos.idr.example.com/<datasetID> spits out a repo that is the same for all datasets except for the notebooks that it contains and what the dataset parameters are set to in the notebooks. Then your BinderHub link would refer to the repo for that dataset. (I have no idea if you can generate git repositories “on the fly” … that is why it is exciting and impractical )
If you go to the app /urlparams and pass two query parameters paramsfile=config.json and params={"a":1,"b":false} it writes the file, then redirects you to urlpath
Example on mybinder:
Double escpaing the query params for BinderHub makes this difficult to hand-craft URLs, it would be nice if BinderHub could process just the parameters it recognises and pass the remainder unchanged to the app. nbgitpuller has the same problem.
filepath is okay, and usually better, because it encodes the location of the file within the repo, which binder then resolves to a url. This doesn’t, however, allow specifying url-specific things like params, so you have to fallback on urlpath if you want to do url things. Any and all url things should be supported, though. The url for a notebook is /notebooks/ not /tree/, and apparently the redirect from /tree to /notebooks doesn’t preserve params, which is why your link isn’t working.
Thanks! That works very well for Jupyter notebooks and the code looks relatively simple. The autorun=true option is pretty neat too!
I had a quick look at the JupyterLab extension docs (first time I’ve looked into extensions), am I correct in thinking this sort of extension would be quite complicated because the notebook isn’t loaded directly, instead it goes via JupyterLab?
@manics This is awesome and much needed so thank you! I think there may be one or two bugs (I am definitely not an expert so hopefully this is not just my own ignorance). I notice in your example you pass two “?” in your URL. I am not sure if two messes up things or if it will just recognize the second ? as an &. The website I am using redirects the user and it recognizes that the URL provided already has a ? and so it appends the URL with an & instead of a second ?. The second bug is that if you pass & instead of %26 the remaining variables are not imported. I have not control over the website I am trying to interact with. Any suggestions on how I can get this to work?
You probably have more control than you realize about the URL since you say you are using MyBinder. In fact the example on the jupyter-notebookparams page uses MyBinder.
If you study the example launch badge URL there (i.e., https://mybinder.org/v2/gh/manics/jupyter-notebookparams/master?urlpath=notebooks%2Fexample.ipynb%3Fa%3D1%26b%3D%22whatever%22%26autorun%3Dtrue) you’ll see there is no second question mark or quotes because the text is ‘escaped’ after the ?urlpath= portion.
If you go to here and then paste in the box on the left the end of the URL the launch badge uses notebooks%2Fexample.ipynb%3Fa%3D1%26b%3D%22whatever%22%26autorun%3Dtrue, and then finally press ‘Decode URL’ button, you’ll see that ‘escaped’ part decodes to notebooks/example.ipynb?a=1&b="whatever"&autorun=true. So for MyBinder use you should be constructing your URL similar to that badge as well. And you can use that same site to encode it for the link you’ll use to launch MyBInder with your notebook. You’ll use the ‘Encode URL’ button to convert that final portion that includes your notebook path and name and the variables you want to set, and place it after the ?urlpath= portion of your typical MyBinder URL.
For example, to pass different parameters, a = 42 and b = 'foobar' to the test notebook, you would construct the URL Binder as a URL. Go ahead and click on that link and you’ll see it launches via MyBinder with those settings now.
Thanks for the response! Maybe I should clarify a bit more what I mean about that I don’t have control over a website. I am build a notebook that analyzes some data. In order to get that data, I have to go through OAuth2. So far, I have been the only one using my notebook, and so I go through an arduous process to get authenticated by hand just so I can get my data. I have some friends that would like to use my notebook and so instead of going through my ‘hack’ to get access, I am wanting to host my notebook on mybinder. To do OAuth2, you go to a certain URL and authenticate. After authenticating, it sends you back to my website with code=authentication_code appended at the end.
As far as I understand as I have looked into it more, I do not think this extension will work for me because it doesn’t quite read the parameters and then pass them in again. Instead, you append the URL with a second set of parameters. So only 1 parameter is read and that parameter is the urlpath=[URLpath]?[parameters=values]. The extenstion reads that urlpath takes you to the URLpath and builds the first cell with the extra parameters. Because I don’t have control of the authentication, my added parameters is added as an actual added parameter and not appended to the URLpath.
Maybe I could find a way to bounce the URL in the right format. But I have found a method where I open the authentication in a IFrame and then display the authentication code. The user has to copy and paste it, but that is easy enough to do for my friends. Any suggestions on how to do this better is welcome! I am definitely not a programer, so I am just faking my way through this.
The solution posted by @manics is great, but the need to encode the URL parameters makes the links a bit brittle (e.g. the encoding gets lost when used as links on Twitter, or when pasted into forums as here).
@betatim - would it be possible for the MyBinder launcher to pass through unknown URL parameters? e.g. mybinder/repo?filepath=bob.ipynb&a=1&b=2 ends up at binder.host/instance/bob.ipynb?a=1&b=2
A downside of passing unknown parameters is the notebook could break if a future release of BinderHub starts making use of those parameters.
One option could be to reserve a prefix for query parameters that are passed through, e.g. param_*, and the Jupyter extension could either strip the prefix or keep it. It’s not as nice for a user constructing a URL, but it’s easier than encoding it.
We had discussed that parameters like x-foo-bar-param would be passed through/never used by Binder itself. Similar to how X-Foobar will never be an official HTTP header.
Could be a nice (little) PR to add. I’d expect most of the work being in writing tests for all the wonderfully weird edge cases and possible legal and illegal combinations wrt quoting/not quoting/escaping/etc.
I was able to adapt it for a project I’m working on where we want to link users to a notebook and pass in a parameter (an id for a dataset to be analyzed). Here is our fork of your repository, which is still a bit in flux: https://github.com/lawlesst/jupyter-notebookparams
In our case, we don’t want to pass arbitrary data, just a single id parameter. Rather than rely on a “Parameter:” comment we look for a consistent variable name, “dataset_id”, and replace the complete line with the passed value. And we may have a use for the “autorun” functionality so we kept that.
)