Missing `ninja` package when building wheels for multiple Python versions on linux #2056

mtsokol · 2024-10-25T13:37:47Z

mtsokol
Oct 25, 2024

Hi!

I'm trying to build wheels for py3.10, py3.11, and py3.12 for my project on linux (it also uses CMake for the compiled part (MLIR dialect)).

The cibuildwheel builds wheel correctly for the first python version in:

[tool.cibuildwheel.linux]
build = "cp310-* cp311-* cp312-*"

but for the next one it fails due to missing dependency (package ninja that is a dependency installed there). It fails always on the second python version, regardless which version that is. Full logs in details below.

I see that the paths are somewhat incorrect in logs:

    Creating /tmp/pip-build-env-o3kxa489/overlay/bin
    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/ninja to 755

...

  CMake Error at CMakeLists.txt:47 (project):
    Running

     '/tmp/pip-build-env-g_8l_p9x/overlay/bin/ninja' '--version'

    failed with:

     no such file or directory

And here's a WIP file that I'm running: https://github.com/finch-tensor/Finch-mlir/blob/434389962d0e6de929ae0d222b29c1822949d3d1/pyproject.toml

I wonder how to debug it? Is there another place where I can define Python deps for these python environments that are used for building wheels on linux?

  Created temporary directory: /tmp/pip-unpack-dv8roaw6
  Looking up "https://files.pythonhosted.org/packages/31/2d/90165d51ecd38f9a02c6832198c13a4e48652485e2ccf863ebb942c531b6/setuptools-75.2.0-py3-none-any.whl" in the cache
  Current age based on date: 303
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 303
  Using cached setuptools-75.2.0-py3-none-any.whl (1.2 MB)
  Downloading link https://files.pythonhosted.org/packages/31/2d/90165d51ecd38f9a02c6832198c13a4e48652485e2ccf863ebb942c531b6/setuptools-75.2.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.8) to /tmp/pip-unpack-dv8roaw6/setuptools-75.2.0-py3-none-any.whl
  Looking up "https://files.pythonhosted.org/packages/1b/d1/9babe2ccaecff775992753d8686970b1e2755d21c8a63be73aba7a4e7d77/wheel-0.44.0-py3-none-any.whl" in the cache
  Current age based on date: 303
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 303
  Using cached wheel-0.44.0-py3-none-any.whl (67 kB)
  Downloading link https://files.pythonhosted.org/packages/1b/d1/9babe2ccaecff775992753d8686970b1e2755d21c8a63be73aba7a4e7d77/wheel-0.44.0-py3-none-any.whl (from https://pypi.org/simple/wheel/) (requires-python:>=3.8) to /tmp/pip-unpack-dv8roaw6/wheel-0.44.0-py3-none-any.whl
  Looking up "https://files.pythonhosted.org/packages/6d/92/8d7aebd4430ab5ff65df2bfee6d5745f95c004284db2d8ca76dcbfd9de47/ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl" in the cache
  Current age based on date: 330
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 330
  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
  Downloading link https://files.pythonhosted.org/packages/6d/92/8d7aebd4430ab5ff65df2bfee6d5745f95c004284db2d8ca76dcbfd9de47/ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (from https://pypi.org/simple/ninja/) to /tmp/pip-unpack-dv8roaw6/ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl
  Looking up "https://files.pythonhosted.org/packages/4b/87/3b01bc707c7a021f3c2c2d133af405618b1580613e22a9ead4a9d9d49903/cmake-3.30.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" in the cache
  Current age based on date: 330
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 330
  Using cached cmake-3.30.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.9 MB)
  Downloading link https://files.pythonhosted.org/packages/4b/87/3b01bc707c7a021f3c2c2d133af405618b1580613e22a9ead4a9d9d49903/cmake-3.30.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/cmake/) (requires-python:>=3.7) to /tmp/pip-unpack-dv8roaw6/cmake-3.30.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  Looking up "https://files.pythonhosted.org/packages/13/2f/0f24b288e2ce56f51c920137620b4434a38fd80583dbbe24fc2a1656c388/pybind11-2.13.6-py3-none-any.whl" in the cache
  Current age based on date: 330
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 330
  Using cached pybind11-2.13.6-py3-none-any.whl (243 kB)
  Downloading link https://files.pythonhosted.org/packages/13/2f/0f24b288e2ce56f51c920137620b4434a38fd80583dbbe24fc2a1656c388/pybind11-2.13.6-py3-none-any.whl (from https://pypi.org/simple/pybind11/) (requires-python:>=3.7) to /tmp/pip-unpack-dv8roaw6/pybind11-2.13.6-py3-none-any.whl
  Looking up "https://files.pythonhosted.org/packages/9b/b4/e3c7e6fab0f77fff6194afa173d1f2342073d91b1d3b4b30b17c3fb4407a/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" in the cache
  Current age based on date: 9
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 9
  Using cached numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.0 MB)
  Downloading link https://files.pythonhosted.org/packages/9b/b4/e3c7e6fab0f77fff6194afa173d1f2342073d91b1d3b4b30b17c3fb4407a/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/numpy/) (requires-python:>=3.10) to /tmp/pip-unpack-dv8roaw6/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  Looking up "https://files.pythonhosted.org/packages/b9/2b/614b4752f2e127db5cc206abc23a8c19678e92b23c3db30fc86ab731d3bd/PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" in the cache
  Current age based on date: 9
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 9
  Using cached PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (767 kB)
  Downloading link https://files.pythonhosted.org/packages/b9/2b/614b4752f2e127db5cc206abc23a8c19678e92b23c3db30fc86ab731d3bd/PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/pyyaml/) (requires-python:>=3.8) to /tmp/pip-unpack-dv8roaw6/PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  Installing collected packages: ninja, wheel, setuptools, PyYAML, pybind11, numpy, cmake

    Creating /tmp/pip-build-env-o3kxa489/overlay/bin
    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/ninja to 755

    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/wheel to 755



    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/pybind11-config to 755

    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/f2py to 755
    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/numpy-config to 755

    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/cmake to 755
    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/cpack to 755
    changing mode of /tmp/pip-build-env-o3kxa489/overlay/bin/ctest to 755
  Successfully installed PyYAML-6.0.2 cmake-3.30.5 ninja-1.11.1.1 numpy-2.1.2 pybind11-2.13.6 setuptools-75.2.0 wheel-0.44.0
  Removed build tracker: '/tmp/pip-build-tracker-tznxtqot'
  Running command Getting requirements to build wheel
  running egg_info
  writing finch_mlir.egg-info/PKG-INFO
  writing dependency_links to finch_mlir.egg-info/dependency_links.txt
  writing top-level names to finch_mlir.egg-info/top_level.txt
  reading manifest file 'finch_mlir.egg-info/SOURCES.txt'
  writing manifest file 'finch_mlir.egg-info/SOURCES.txt'
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info
  writing /tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/SOURCES.txt'
  writing manifest file '/tmp/pip-modern-metadata-48wkbbz4/finch_mlir.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-48wkbbz4/finch_mlir-0.0.1.dist-info'
  Running command Building wheel for finch-mlir (pyproject.toml)
  running bdist_wheel
  running build
  running build_ext
  CMake Deprecation Warning at /project/llvm-project-main/cmake/Modules/CMakePolicy.cmake:6 (cmake_policy):
    The OLD behavior for policy CMP0116 will be removed from a future version
    of CMake.

    The cmake-policies(7) manual explains that the OLD behaviors of all
    policies are deprecated and that a policy should be set to OLD only under
    specific short-term circumstances.  Projects should be ported to the NEW
    behavior and not rely on setting a policy to OLD.
  Call Stack (most recent call first):
    CMakeLists.txt:8 (include)


  CMake Error at CMakeLists.txt:47 (project):
    Running

     '/tmp/pip-build-env-g_8l_p9x/overlay/bin/ninja' '--version'

    failed with:

     no such file or directory


  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "/opt/python/cp312-cp312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/python/cp312-cp312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/python/cp312-cp312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 434, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', metadata_directory])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 422, in _build
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 516, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
      exec(code, locals())
    File "<string>", line 99, in <module>
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 183, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 407, in run
      self.run_command("build")
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 98, in run
      _build_ext.run(self)
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
      self._build_extensions_serial()
    File "/tmp/pip-build-env-o3kxa489/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
      self.build_extension(ext)
    File "<string>", line 48, in build_extension
    File "/opt/python/cp312-cp312/lib/python3.12/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cmake', '/project/llvm-project-main/llvm', '-G Ninja', '-B/project/build/llvm-build', '-DLLVM_ENABLE_PROJECTS=mlir', '-DLLVM_TARGETS_TO_BUILD=Native', '-DMLIR_ENABLE_BINDINGS_PYTHON=ON', '-DPython3_EXECUTABLE=/opt/python/cp312-cp312/bin/python', '-DLLVM_INSTALL_UTILS=ON', '-DLLVM_CCACHE_BUILD=ON', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PLATFORM_NO_VERSIONED_SONAME=ON']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for finch-mlir (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/python/cp312-cp312/bin/python /opt/python/cp312-cp312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmp91awvaqq
  cwd: /project
  ERROR: Failed building wheel for finch-mlir
ERROR: Failed to build one or more wheels

NOTE: Shared object (.so) files found in this project.

These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout.

If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.

setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.

Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.

Error: Command ['python', '-m', 'pip', 'wheel', PurePosixPath('/project'), '--wheel-dir=/tmp/cibuildwheel/built_wheel', '--no-deps', '-vvv'] failed with code 1.

Answered by mtsokol

Oct 25, 2024

After changing:

subprocess.run(["ninja"], cwd=finch_build_dir, check=True)

to:

import ninja
ninja_executable_path = Path(ninja.BIN_DIR) / "ninja"
subprocess.run([ninja_executable_path], cwd=finch_build_dir, check=True)

in my setup.py helped here.

View full answer

mtsokol · 2024-10-25T14:17:59Z

mtsokol
Oct 25, 2024
Author

After changing:

subprocess.run(["ninja"], cwd=finch_build_dir, check=True)

to:

import ninja
ninja_executable_path = Path(ninja.BIN_DIR) / "ninja"
subprocess.run([ninja_executable_path], cwd=finch_build_dir, check=True)

in my setup.py helped here.

2 replies

henryiii Oct 25, 2024
Maintainer

I'd highly recommend looking at scikit-build-core for building Python packages with CMake. There are a lot of caveats to extending setuptools, adding build-type deps, etc. (PS: do you actually need pyyaml at build time? Also, pybind11 doesn't need numpy at build time).

mtsokol Oct 25, 2024
Author

Thank you! I will take a look at it. I guess there might be a lot of improvements to this build setup, so far I'm mostly following MLIR Python bindings build setup (withing LLVM project), adjusting to my needs:
https://github.com/makslevental/mlir-wheels/blob/main/pyproject.toml
https://github.com/makslevental/mlir-wheels/blob/main/setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing `ninja` package when building wheels for multiple Python versions on linux #2056

{{title}}

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Missing ninja package when building wheels for multiple Python versions on linux #2056

mtsokol Oct 25, 2024

Replies: 1 comment · 2 replies

mtsokol Oct 25, 2024 Author

henryiii Oct 25, 2024 Maintainer

mtsokol Oct 25, 2024 Author

Missing `ninja` package when building wheels for multiple Python versions on linux #2056

mtsokol
Oct 25, 2024

Replies: 1 comment 2 replies

mtsokol
Oct 25, 2024
Author

henryiii Oct 25, 2024
Maintainer

mtsokol Oct 25, 2024
Author