How to compile cython files for use in binder?

I have a github repo with some .pyx files. I can compile them from the command line with python looks like:

from distutils.core import setup
from Cython.Build import cythonize
    ext_modules=cythonize('./*.pyx', annotate = True), 

Right now I am using environment.yml to set up my binder, is there some directive to include in that file? Or another file to include? I would like for binder to spin up the image with the files compiled.

1 Like

A postBuild file might be the simplest approach. It runs after the environment has been updated, so pip install -e ., for example, would work, provided cython (and any other build dependencies) are included in the environment.yml.

Of note: transitioning to setuptools is recommended over distutils by the cython docs… and the maintainers of distutils.


Of note: transitioning to setuptools is recommended over distutils by the cython docs… and the maintainers of distutils .

Interesting, I didn’t know that, thank you.

Would adding

- pip:
  - .

To the environment.yml file work? As in, will binder run pip install .?

Yes, that pip trick works.

From an entropy/cache perspective: in binder (well, docker in general), I don’t recommend doing anything “fancy” with environment.yml, as it triggers an undesirable code path which forces the environment to be resolved, re-downloaded, and re-linked every commit, instead of just when the environment.yml changes.

I find having a local- and docker-compatible Makefile or which handles complex, dependency-based environment tasks, which either install or assume the “base” environment, leads to the most reproducible outcomes… especially when coupled with tools like conda-lock and pip-tools to generate lockfiles, as these are great sources of predictable entropy.

So, in summary, I should instead have a file called postBuild and in it should just be the line pip install -e .?

And forget doing anything in environment.yml that isn’t declaring simple dependencies?

Right, a cache-friendly, defensive setup might be:

  • environment.yml
      - conda-forge
      - nodefaults
      - cython                # and any other build deps
      - python >=3.9,<3.10    # or whatever version
      - jupyterlab >=3.4.4,<4 # or whatever version
      - pip
      - pip:
        - any-deps-not-met-by-conda
  • postBuild
    #!/usr/bin/env bash
    python -m pip install -vv --no-deps --ignore-installed -e .
    python -m pip list   # so you have info for...
    python -m pip check  # ...catching packaging issues during build with log