`mamba env update` instead of `mamba env create`?

Binder discuss
1

Hello,

I have used mybinder.org for a repository for a couple of years now:

https://mybinder.org/v2/gh/AMIGA-IAA/hcg-16/udocker-1.3.3

It’s been working until recently but now I get

Encountered problems while solving:
  - package aplpy-2.0-py_0 requires reproject >=0.4, but none of the providers can be installed

In the logs I see:

Step 40/53 : RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '

Why is mamba doing env update instead of env create?

A local installation with mamba and the environment file works:

https://github.com/AMIGA-IAA/hcg-16/blob/8c110ee6f9e39e5d285c74678c806bf273b558b3/environment.yml

Many thanks,
Sebastian

2

This is mostly for speed and image size. The mamba env update command is used as often folk only need to specify a few packages, and get them installed into the default env, running with most of the jupyter stack ready to go on python3.7. The default stance leaves less of a delta on disk, even if some packages are out-of-date.

Your specific environment.yml has a lot of tight pins, and a lot of channels, including the dreaded conda-forge/labels/broken. The jupyter metapackage is also rather troublesome, and you should drop that if you can, as qt doesn’t buy you much on binder without a lot of other tricks.

You could sacrifice speed and cache-ability by forcing more behavior with a renamed environment-foo.yml and a postBuild, e.g.

For binder, you’d need to ensure you had the jupyterhub-singleuser package installed.

#!/usr/bin/env bash
mamba env create \
  --file .binder/environment-foo.yml \
  --prefix ${NB_PYTHON_PREFIX}
mamba clean -yaf # clean up downloaded files and indexes

I’d also highly recommend investigating conda-lock, and check a known-good, binder-compatible solve of your specific env in.

conda-lock --kind=explicit --platform linux-64 --file

this will generate a lockfile that you can check in and use with any conda or mamba install, and get exactly the packages you want, and install with the same mamba create technique above.

3

Thanks @bollwyvl

I added conda-forge/labels/broken after reading:

https://mybinder.readthedocs.io/en/latest/tutorials/reproducibility.html

The jupyter metapackage is there since we intend to use the same environment to execute the code both locally and on binder (and being consistent across). Reading https://jupyter.org/install I also tried jupyterlab but qt is also installed with it. Any other suggestion?

I just played a bit with conda-lock but it wasn’t a great experience, and I couldn’t get it to work properly. Not sure if I will use this at the moment.

I have also played a bit with the postBuild, adding:

mamba env create \
  --file hcg16.yml \
  --prefix ${NB_PYTHON_PREFIX}
mamba clean -yaf # clean up downloaded files and indexes

conda activate hcg-16

However, when I run the notebooks on binder, I don’t see the environment available?

!conda env list
# conda environments:
#
base                     /srv/conda
notebook              *  /srv/conda/envs/notebook

Do you know what would be the issue?
Thanks,
Sebastian

4

jupyterlab but qt

There are any number of packages that will pull it: jupyter is just a likely candidate due to qtconsole. Another candidate is matplotlib, vs matplotlib-base.

Really, unless your desktop work actively uses qt-based UIs, it’s worth trying to avoid the 200mb and rather arcane pins it sometimes forces.

conda-lock

Seems to work, with some of the above changes: a conda-lock · GitHub

note, it also needed some extra pins, as many newer conda-forge packages don’t take EOL python versions into account.

I don’t see the environment

--prefix overrides --name (or the name: field in the environment). The rest of the machinery is looking to start ${NB_PYTHON_PREFIX}/bin/jupyter.

If there is hard-coded stuff in the code referring to hcg-16 (vs $CONDA_PREFIX or sys.prefix) then the user is likely in for even more adventures.