Using cibuildwheel to build project dependencies

[andyfaff]

The scipy/numpy packages use the OpenBLAS library whilst building their wheels. This requires the .so, include directories, etc.
At the moment those projects download a pre-built OpenBLAS library.

I was wondering if it is possible to use cibuildwheel to solely build OpenBLAS? i.e. I just want to run the CIBW_BEFORE_ALL/CIBW_BEFORE_BUILD steps to build OpenBLAS on a range of OS/architectures, I don't want to build any Python wheels, but then I want to upload an artefact containing the OpenBLAS library. (I don't want to build OpenBLAS during a wheel build, it takes way too long)

The reason this might be useful is that the build process would use the same environment (e.g the right Docker container) that would ultimately be used for building wheels in end-user packages, and cibuildwheel seems to have the infrastructure for specifying many of the relevant options.

Another way of putting my question would be: "What's the best way of using CI to build a C library across all the OS/architecture matrix that is ultimately going to be used for building Python wheels"?

[joerick]

cibuildwheel doesn't currently work for this use-case - we always require that a platform wheel is built, via either pip or build. And actually, we don't currently support a method of extracting files from each build (that's a long-running issue #363), but some people have had success with moving their files around manually (on Linux, to /output or /host/something to escape the Docker container).

I can think of a few things:

1. Write a small Python extension that exposes some of the OpenBLAS API, and build that at the same time. That would also let you write some basic tests in Python that cibuildwheel could call to check that the built library is working ok.

2. Don't use cibuildwheel, but reconstruct the build environment yourself. It's not too bad. Off the top of my head:

   Windows: we don't do anything special
   Mac: we set MACOSX_DEPLOYMENT_TARGET=10.9
   Linux: we run in a docker image, take your pick from pinned_docker_images.cfg

[henryiii]

I was thinking of option 2 as well when I saw this, FWIW.

[joerick]

Another thing that I wonder is whether it would be possible to build a C library and package it as a wheel - kind of like how the cmake wheels contain cmake executables. Then it could even be used by other projects that might want to call OpenBLAS in their Python extension. But I am just speculating.