Build errors when using fully pinned requirements.txt coming from pip-tools

Binder repo help
1

When building GitHub - astrojuanlu/ie-mbd-advanced-python: "Advanced Python" subject from the Master in Big Data @ IE, build fails at this step:

Attempting uninstall terminado
Found existing installation: terminado 0.8.3
ERROR: Cannot unintstall 'terminado'. It is a distutils installed project and thus...

(by the way, I can’t seem to be able to copy the text from the build logs…)

The requirements.txt contains these lines:

$ grep terminado requirements.txt 
ptyprocess==0.6.0         # via pexpect, terminado
terminado==0.9.1          # via notebook
tornado==6.1              # via ipykernel, jupyter-client, notebook, terminado

Because they come from pip-compile:

$ cat requirements.in 
black==20.8b1
notebook
rise

I guess that pinning terminado to the version that Binder uses would fix the problem, but I don’t think this was needed in the past.

This was already discussed in the chat, this was @betatim answer (thanks!):

Generally we recommend to not list all packages ever in your requirements.txt but only those you “care” about. in particular for things which are dependencies of your dependencies and such i wouldn’t specify an explicit version
…
it is a bit particular to binder. i think in general having a “fully pinned” requirements.txt is a good idea/practice but in binder it gets used “on top of” an existing environment so the recommendation changes a bit

However, @minrk notes:

I think there is an issue in that the “distutils installed” bit shouldn’t happen. This is probably a problem in the conda package not installing the right metadata.

For the moment, I think I will stick to “do not pip-compile Jupyter-related dependencies” rule, or not use pip-tools at all (sadly)

2 Likes
2

Coming back to this: perhaps Binder could provide in their docs some sort of constraints files for these cases?

https://pip.pypa.io/en/stable/user_guide/#constraints-files

In that way, I could pin my requirements without breaking my Binder environment. I guess that these constraints would essentially be packages that repo2docker is installing in the “base” environment (not sure where to look at). @betatim @choldgraf what do you think?

1 Like
3

Hmmm - so if you added a constraints file with your repository, it would list versions needed but not install them? How would you ensure that this wouldn’t break the repository (e.g., if it actually needed a newer version of Tornado)?

4

Meh, I was trying several combinations of “constraints” and pip-tools to give a more specific answer but they don’t seem to work very well. For example:

(venv) $ ls
constraints.in  requirements.in
(venv) $ cat constraints.in 
# This is what Binder sets
notebook==5.7.10
(venv) $ cat requirements.in 
-c constraints.txt

black==20.8b1
notebook
rise
(venv) $ pip-compile constraints.in > /dev/null 2>&1
(venv) $ pip-compile requirements.in 
Could not find a version that matches notebook==5.7.10,>=6.0 (from -r requirements.in (line 4))
Tried: 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.0.1, 4.0.2, 4.0.2, 4.0.2, 4.0.4, 4.0.4, 4.0.4, 4.0.5, 4.0.5, 4.0.6, 4.0.6, 4.0.6, 4.1.0, 4.1.0, 4.1.0, 4.2.0, 4.2.0, 4.2.0, 4.2.1, 4.2.1, 4.2.1, 4.2.2, 4.2.2, 4.2.2, 4.2.3, 4.2.3, 4.3.0, 4.3.0, 4.3.1, 4.3.1, 4.3.2, 4.3.2, 4.4.0, 4.4.1, 4.4.1, 5.0.0, 5.0.0, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.2.2, 5.2.2, 5.3.0, 5.3.0, 5.3.1, 5.3.1, 5.4.0, 5.4.0, 5.4.1, 5.4.1, 5.5.0, 5.5.0, 5.6.0, 5.6.0, 5.7.0, 5.7.0, 5.7.1, 5.7.1, 5.7.2, 5.7.2, 5.7.3, 5.7.3, 5.7.4, 5.7.4, 5.7.5, 5.7.5, 5.7.6, 5.7.6, 5.7.8, 5.7.8, 5.7.9, 5.7.9, 5.7.10, 5.7.10, 6.0.0, 6.0.0, 6.0.1, 6.0.1, 6.0.2, 6.0.2, 6.0.3, 6.0.3, 6.1.0, 6.1.0, 6.1.1, 6.1.1, 6.1.2, 6.1.2, 6.1.3, 6.1.3, 6.1.4, 6.1.4, 6.1.5, 6.1.5, 6.1.6, 6.1.6, 6.2.0, 6.2.0
Skipped pre-versions: 4.2.0b1, 4.2.0b1, 4.2.0b1, 5.0.0b1, 5.0.0b1, 5.0.0b2, 5.0.0b2, 5.0.0rc1, 5.0.0rc2, 5.0.0rc2, 5.1.0rc1, 5.1.0rc1, 5.1.0rc2, 5.1.0rc2, 5.1.0rc3, 5.1.0rc3, 5.2.0rc1, 5.2.0rc1, 5.2.1rc1, 5.2.1rc1, 5.3.0rc1, 5.3.0rc1, 5.5.0rc1, 5.5.0rc1, 5.6.0rc1, 5.6.0rc1, 6.0.0rc1, 6.0.0rc1, 6.1.0rc1, 6.1.0rc1
There are incompatible versions in the resolved dependencies:
  notebook (from -r requirements.in (line 4))
  notebook==5.7.10 (from -c constraints.txt (line 56))
  notebook>=6.0 (from rise==5.7.1->-r requirements.in (line 5))

I will continue pursuing this when I have a better idea on whether this behavior is a bug or a feature. Thanks @choldgraf anyway!