How to compile cython files for use in binder?

Binder
1

I have a github repo with some .pyx files. I can compile them from the command line with python setup.py

setup.py looks like:

from distutils.core import setup
from Cython.Build import cythonize
setup(
    ext_modules=cythonize('./*.pyx', annotate = True), 
    script_args=['build_ext'],                                        
    options={'build_ext':{'inplace':True}}
)

Right now I am using environment.yml to set up my binder, is there some directive to include in that file? Or another file to include? I would like for binder to spin up the image with the files compiled.

1 Like
2

A postBuild file might be the simplest approach. It runs after the environment has been updated, so pip install -e ., for example, would work, provided cython (and any other build dependencies) are included in the environment.yml.

Of note: transitioning to setuptools is recommended over distutils by the cython docs… and the maintainers of distutils.

2 Likes
3

Of note: transitioning to setuptools is recommended over distutils by the cython docs… and the maintainers of distutils .

Interesting, I didn’t know that, thank you.

Would adding

- pip:
  - .

To the environment.yml file work? As in, will binder run pip install .?

4

Yes, that pip trick works.

From an entropy/cache perspective: in binder (well, docker in general), I don’t recommend doing anything “fancy” with environment.yml, as it triggers an undesirable code path which forces the environment to be resolved, re-downloaded, and re-linked every commit, instead of just when the environment.yml changes.

I find having a local- and docker-compatible Makefile or dodo.py which handles complex, dependency-based environment tasks, which either install or assume the “base” environment, leads to the most reproducible outcomes… especially when coupled with tools like conda-lock and pip-tools to generate lockfiles, as these are great sources of predictable entropy.

5

So, in summary, I should instead have a file called postBuild and in it should just be the line pip install -e .?

And forget doing anything in environment.yml that isn’t declaring simple dependencies?

6

Right, a cache-friendly, defensive setup might be:

  • environment.yml
    channels:
  - conda-forge
  - nodefaults
dependencies:
  - cython                # and any other build deps
  - python >=3.9,<3.10    # or whatever version
  - jupyterlab >=3.4.4,<4 # or whatever version
  - pip
  - pip:
    - any-deps-not-met-by-conda
  • postBuild
    #!/usr/bin/env bash
python -m pip install -vv --no-deps --ignore-installed -e .
python -m pip list   # so you have info for...
python -m pip check  # ...catching packaging issues during build with log
2 Likes