BLD: issues while making macosx_arm64 natively on cirrus-ci (scipy)

[andyfaff]

Description

I'm trying to build scipy wheels for macosx_arm64 natively on cirrus-ci (in anticipation of scipy/scipy#17029, linux_aarch64 being merged). The work is being done in andyfaff/scipy#33.

Two matrix entries are for cp38 and cp39, see the build log here. For each of the builds meson thinks its doing a native build for x86_64, even though the CI is an arm64 machine:

  Build type: native build
  Project name: SciPy
  Project version: 1.10.0.dev0
  C compiler for the host machine: cc (clang 13.1.6 "Apple clang version 13.1.6 (clang-1316.0.21.2.3)")
  C linker for the host machine: cc ld64 762
  C++ compiler for the host machine: c++ (clang 13.1.6 "Apple clang version 13.1.6 (clang-1316.0.21.2.3)")
  C++ linker for the host machine: c++ ld64 762
  Host machine cpu family: x86_64
  Host machine cpu: x86_64

The cibuildwheel infrastructure thinks the build machine is arm64:

  platform: 'macos'
  architectures: {<Architecture.arm64: 'arm64'>}

Prior to the cibuildwheel step I also query the machine (with a different Python interpreter installed using brew):

> python -c "import platform;print(platform.python_version());print(platform.system());print(platform.machine())"
3.10.4
Darwin
arm64
> uname -m
arm64
> clang --version
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.4.0
Thread model: posix
InstalledDir: /Applications/Xcode-13.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

At the start of the build process (before meson runs) pip is trying to install numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl. This leads me to think there is something fishy going on with cibuildwheel/python setup/pip (that then causes the meson issue). Why would pip be trying to install an x86_64 wheel onto an arm64 machine?
pip gives the following output whilst considering pyproject.toml:

Ignoring numpy: markers 'python_version == "3.8" and platform_machine == "arm64" and platform_system == "Darwin"' don't match your environment

Given that ostensibly the python version installed by cibuildwheel is 3.8.10, the machine should be arm64, and the platform should be Darwin, I don't know why the specific numpy requirement (1.21.0) isn't being obeyed.

I should note that in other matrix entry I build for 3.10 and 3.11. There the correct numpy for machine is installed, i.e. macosx_arm64.

Build log

https://api.cirrus-ci.com/v1/task/4544934911934464/logs/cibuildwheel.log

CI config

https://github.com/andyfaff/scipy/blob/cirrus2/.cirrus.yml

[andyfaff]

After trying lots of CI debugging I decided to try and build the package on cirrus-ci outside of cibuildwheel, using the Python@3.9 installed by homebrew.
When I pip install numpy the correct architecture gets installed. meson also picks up the right build architecture.

https://cirrus-ci.com/task/5623569559322624

This leads me to believe something is broken in the way cibuildwheel sets up its Python environment for Python 3.8/3.9.

[andyfaff]

I've found the smoking gun. In this run I use

echo $(python -c "import platform;print(platform.python_version());print(platform.system());print(platform.machine())")
echo $(pip --version)

in the CIBW_BEFORE_BUILD script. The output from this is:

++ python -c 'import platform;print(platform.python_version());print(platform.system());print(platform.machine())'
+ echo 3.8.10 Darwin x86_64
3.8.10 Darwin x86_64
++ pip --version
+ echo pip 22.2.2 from /private/var/folders/_f/lyvxf0v13gs7984d7sf7j83c0000gn/T/cibw-run-otkzgjht/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/pip '(python' '3.8)'
pip 22.2.2 from /private/var/folders/_f/lyvxf0v13gs7984d7sf7j83c0000gn/T/cibw-run-otkzgjht/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/pip (python 3.8)

The Python that cibuildwheel is installing to do the build is reporting x86_64 for platform.machine() instead of arm64. I'm pretty sure this will be a bug in cibuildwheel.

[andyfaff]

Over on scipy/scipy#17077 (where I also asked this question) @rgommers pointed out that the official Python 3.8 installer doesn't have arm64 support. So perhaps it's not possible to build arm64 wheels on 3.8 using cibuildwheel?

[Czaki]

If I correctly remember it is possible by cross compilation but it is not possible to test it in test phase.

[andyfaff]

If it's only possible to build on 3.8 by cross-compilation, then this should be outlined in the documentation somewhere because the are probably a lot of projects that require a bunch of extra stuff for cross-compilation. e.g. scipy requires a gfortran cross compiler. (Please forgive me if it is in the documentation, I didn't see it.)

[joerick]

If I'm reading the linked issue right, it seems that this problem is localised to CPython 3.8? In a way, I'm not surprised. The universal2/arm64 support in that installer has always been janky from my perspective.

I think that CPython 3.8 always runs in Rosetta. So that's why your numpy download is wrong.

At least, we should probably warn when users are building 3.8 on an arm64 host.

I'm actually leaning towards removing that from cibuildwheel, as we get a lot of bug reports about it and I don't think many people will be consuming those wheels. I might look at PyPI download stats to see what the numbers really are.

[rgommers]

I'm actually leaning towards removing that from cibuildwheel, as we get a lot of bug reports about it and I don't think many people will be consuming those wheels. I might look at PyPI download stats to see what the numbers really are.

I think a clear error for a naive 3.8 arm64 build is preferred over what cibuildwheel is doing currently. That said, projects probably cannot drop 3.8 arm64 wheels before they drop 3.8 completely. Otherwise 3.8 users are going to get from-source builds, which are going to fail a lot. So if the docs could be updated to point to the cross-compiling method, that'd be great.

[Czaki]

If I correctly read log there is also problem for python 3.9?

[rgommers]

If I correctly read log there is also problem for python 3.9?

The last message on the issue says that that was fine, and checking this log file shows:

Created wheel for scipy: filename=scipy-1.10.0.dev0-cp39-cp39-macosx_11_0_arm64.whl

[andyfaff]

I think it is localised to 3.8. However, there's several errors that are clouding things, so I'm trying to work through those first. If it is localised to 3.8 I'll close this issue.

[andyfaff]

It's localised to 3.8, not present for 3.9.

[henryiii]

I'm actually leaning towards removing that from cibuildwheel, as we get a lot of bug reports about it and I don't think many people will be consuming those wheels. I might look at PyPI download stats to see what the numbers really are.

This was tricky to add, but it was extremely important - 3.8 was the default version of Python on macOS 11! So anyone using the system Python on macOS 11 needs 3.8 ARM wheels. It might not / probably won't work on an ARM runner, though, since the official support was adding the ability to cross compile from Intel, and I don't think there ever was an ARM runner. You can use Apple's custom macOS 11 copy, of course, but that will be OS dependent - macOS 12 has 3.9.

[rgommers]

Now that CirrusCI is a thing (it offers free arm64 runners for open source projects), that will likely quickly get popular.

Can this issue be reopened? There are two things to do here:

1. Raise an error on native arm64 runners,
2. Update the docs to state that cross-compiling is the way to go for 3.8 wheels

[henryiii]

I believe you can technically build on ARM, you just can't install non pure Python dependencies on ARM while building? (In other words, you can't run Python, since that will be Intel not ARM).

[henryiii]

If that's true, I'd say we should raise an error unless the tests are explicitly skipped. That would make this a clear opt-in.

[andyfaff]

When I try to build scipy using a naive cibuildwheel call on cirrus-ci for a target of cp38-macosx_arm64 the build does complete. However, when I download the artefact and unpack the wheel, examining the architectures of all the *.so, they're a mixture of x86_64 and arm64. I think the purely c-based codes come out as x86_64, but ones that depend on gfortran come out as arm64. This is because the gfortran compiler only targets arm64, whereas clang probably does arm64 and x86_64.

I think in order to make them all the correct architecture I'll probably have to specify a cross-file for meson (the build backend). I was hoping to get away with not doing that because it'll involve more fiddling. Given that I can already build cp38-macosx_arm64 via cross compilation (outside of cibuildwheel) on GHA I may stick with that, and only use cirrus-ci + cibuildwheel to do the native build of cp3(9,10,11)-macosx_arm64.

[joerick]

This was tricky to add, but it was extremely important - 3.8 was the default version of Python on macOS 11! So anyone using the system Python on macOS 11 needs 3.8 ARM wheels.

Thanks for the reminder, I had forgotten this. Indeed PyPI download stats do confirm that it's used pretty regularly:

link to Google sheet that produced this graph

[mayeut]

One thing you can try if you want to natively build cp38-macosx_arm64 is to install the universal2 version of CPython 3.8 before starting cibuildwheel.

This installer was marked as experimental but recommended for Apple Silicon.

cibuildwheel always use the Intel installer, even on AppleSilicon, because the universal2 installer restricted building wheels for macOS 11+ only whereas the Intel installer can be used for macOS 10.9+. It could be an issue when building x86_64 or universal2 wheels on an AppleSilicon machine but in your case, when only targeting an arm64 build, it's not.

However, even if you get the native build to pass, you won't be able to get cibuildwheel to run the tests for cp38-macosx_arm64 as it's a hardcoded exception (for now at least).

[henryiii]

I was thinking we could improve this, so here are my thoughts:

1. When building for AS, the min version of macOS is 11 anyway, so you want the U2 installer for AS on AS. The Intel installer "works", but not fully (issues in this thread).
2. When building for U2, then you want the U2 installer if you are targeting 11+, but you are forced to the Intel installer if you are targeting anything less, like the default 10.9+.

Given the above, the ideal situation would be for the AS on AS to use the AS installer, and for U2 on AS to pick one of the two (the current Intel is probably the better default), but have some emergency back-door to allow using the AS. (The 11+ limit would make U2 wheels pretty useless IMO - so this back-door is a very low priority). A much better way to build U2 wheels would be to fuze Intel (10.9) + AS (11) wheels anyway. It would be nice if we had better support or docs for doing that. :)

Testing on AS should probably always use the U2 installer - you are on 11+ anyway.

Also, for implementing a solution, an important thing to note is this is temporary. Once we drop 3.8 support, this will all go away. The URLs here should never change - new binary packages are not being released. So a solution could be hardcoded, and introducing new complexity in the url listing or new configuration options is not ideal.

The problem is that you can't install both (AFAICT), and you can run multiple identifiers at any time (AS + U2, for example), or target different macOS versions via overrides, etc. So trying to figure this out and do the right thing without user input is trickier than I thought. Ideally, building AS on AS would replace the Intel installer with a U2 one. Targeting 11+ could even replace the U2 one as well.

I think we could start by checking to see if the resolved macOS archetectures includes U2. If not, then we replace the Intel URL with the U2 URL. This would address point 1 above. If we wanted to, we could provide a backdoor envvar or maybe even full config setting to allow the same replacement even if U2 is included (with the caveat being targeting macOS 11+ is required, which maybe we could even check, or include in the envvar name) - but this is lower priority and a followup if requested, I think.

If you ran cibuildwheel twice, this would cause issues in the same run, but they are the same issues you'd get if you pre-install Python, so don't think that's too bad. But that's one reason I'm writing this before trying (very hard) to implement anything. :)

[rgommers]

A much better way to build U2 wheels would be to fuze Intel (10.9) + AS (11) wheels anyway. It would be nice if we had better support or docs for doing that. :)

It would be ideal if there was a separate tool for this. It could then be used inside cibuildwheel, but also by things like py2app. Longer-term universal2 wheels on PyPI aren't very useful, they just consume space while you can't even directly install them with pip. So longer term it'd be nice to not have to bother with them as a package author - for that very small fraction of users that need universal2 wheels, just run that separate tool or have py2app & co do that for you.

[joerick]

Thanks for the analysis @henryiii

I think we could start by checking to see if the resolved macOS archetectures includes U2. If not, then we replace the Intel URL with the U2 URL. This would address point 1 above.

This would be a good minimum. Though it's worth noting that functionally, as a workaround it's no different than the user manually curl'ing and installing the u2 pkg manually. Though I suppose it would become an officially supported mode of operation.

If we wanted to, we could provide a backdoor envvar or maybe even full config setting to allow the same replacement even if U2 is included (with the caveat being targeting macOS 11+ is required

If we're going to the trouble of adding a feature, do we know if installing the u2 pkg over a x86_64 installation would work? I know from experience that you can upgrade Python that way. Globally installed packages with native code might break but I don't think we install anything besides pip (and perhaps virtualenv?).

If that does, we can have an option native-macOS-cp38-builds and it would switch out the install during the run between the intel and arm builds, and make things more user-friendly and repeatable.

[perklet]

I'm getting the same error here, platform.machine got confused since python is x86_64 on arm64 machine using rosetta. My code detects that add selects corresponding dylibs, which fails for this reason.

Upgrading to cp39-macos-arm64 fixed it.

[agriyakhetarpal]

Commenting on this issue to notify that I am receiving a slightly similar problem too, of sorts, with using cibuildwheel on the new GitHub Actions M1 runners (specified by runs-on: macos-14), where Clang thinks that the target for the build is x86_64 instead of arm64. This fails my wheel builds with Python 3.8.

Outside of and without cibuildwheel, i.e., while running on a different PR build on macOS arm64 – Clang correctly assumes the correct target to be arm64 itself.

on a macOS M1 runner, with cibuildwheel

    Apple clang version 15.0.0 (clang-1500.0.40.1)
    Target: x86_64-apple-darwin23.2.0
    Thread model: posix

on a macOS M1 runner, without cibuildwheel

    Apple clang version 15.0.0 (clang-1500.0.40.1)
    Target: arm64-apple-darwin23.2.0
    Thread model: posix

I am unsure whether this is something that is coming from incorrect compiler configurations from the actions/runner-images end or from the cibuildwheel configuration (though I don't think cibuildwheel exercises any control over Clang). I thought it would be good to post here as well just in case someone else faces the issue – perhaps not building for Python 3.8 and proceeding from Python 3.9 and above should be what it would take to fix it, as mentioned previously in this thread.

If this is unrelated to this thread but still related enough for cibuildwheel, I apologise :) happy to open a new issue about this at the maintainers' discretion if this looks like a legitimate bug. Thanks!

---

Edit: I can confirm that Python 3.9+ with cibuildwheel works well and does not bring up any issues, but it is also to be noted that non-cibuildwheel Python 3.8 works too.

[cretz]

I am hitting this too where cp38-macosx_arm64 on macos-14 in GH builds wheel as cp38-macosx_x86_64 instead (and it is fixed by just updating to cp39-macosx_arm64 and I'm lucky I use abi3 limited API in my C extension so I can rename back to 3.8 after).

I wonder if this can be fixed by changing

cibuildwheel/cibuildwheel/resources/build-platforms.toml

Line 92 in 9cf99e7

{ identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },

. Today it downloads https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg, but maybe it should be https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg instead like it is with 3.9 builds.

[agriyakhetarpal]

I think because Python 3.8 never provided an architecture-specific arm64 installer (https://www.python.org/downloads/release/python-3810/), something seems to be off on that regard from cibuildwheel's side. I did download the Python distribution (universal2) from the above link and checked the command on an M-series machine mentioned in the topmost comment:

/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -c 'import platform;print(platform.python_version());print(platform.system());print(platform.machine())'

to receive

3.8.10
Darwin
arm64

which should be correct, so using this to run cibuildwheel should be okay, but isn't. Alhough, Python 3.8 is scheduled for EOL in just about a few more months by this time and wheels for that version will diminish in due course, I think one way to resolve the issue, with at least experimental bounds, is to let users download a Python distribution by themselves (with an environment variable CIBW_EXPERIMENTAL switched on, otherwise not). Edit: this also needs some figuring out as to how to install them (they are not .pkg files but just pre-built ones that just need ensurepip or a few other things like setting PATH correctly).

To provide an example for the above, https://github.com/indygreg/python-build-standalone/releases/tag/20240224 (unofficial releases) could be quite useful. However, I don't know if they are supposed to be used in CI and especially to build wheels to publish to PyPI, because there is a bit of nuance in setting MACOSX_DEPLOYMENT_TARGET based on the nature of the extension module, as supported by #1804 (comment) and many other issues that come up. But at least to build a wheel with cibuildwheel running locally, @cretz, I would definitely recommend trying these builds out, it would be helpful with isolating where the issue is I would think. You still won't be able to use CIBW_TEST_COMMAND because it is blocked by cibuildwheel for Python 3.8 on macOS arm64 explicitly, but they can be tested manually later on outside of cibuildwheel.

[larsoner]

@andyfaff you could try #1871 if you want. It just implements the suggestion by @cretz and worked on my repo at least!

[andyfaff]

@larsoner, thanks for the info. However numpy and scipy now have minimum python versions of 3.10 in main, so this issue isn't a concern for those projects any more.