Python 3.14t build

[kdschlosser]

I had opened issue CadQuery/cadquery#2029 a bunch of months ago and there wasn't a giant amount of support or help to get cadquery to compile for Python 3.14t. The build system that was in place at the time was causing me a very large amount of grief so I walked away from it for a while. Yesterday I decided to revisit trying to make this happen. I discovered that the build system had been completely rewritten to something that was a whole lot easier to work with.

I want to say TY for rewriting the build system, it has allowed me to make the modifications needed to get a working build for Python 3.14t. If you are interested in seeing what was done to get it to work you can find the code in my fork located HERE. There is value to doing this because it would allow an application to have a thread for handling OCP related tasks and another thread handling UI type stuff and the threads would be able to run parallel to each other. I posted some example code on how to handle making sure that OCP is only accessed from a single thread. I do however want to mention that not everything needs to run using a single thread. I was able to create shapes for every single glyph for for a font for each of the 4 styles using 4 threads at the same time. That process also involved tessellating the shape into a mesh array as well. by using 4 threads there was a 348% speedup which is a substantial increase.

The code should compile for all of the previous Python versions but I have disabled that from happening to speed up the compile times. I also set up a self hosted runner for the windows build because the CI would take about 5.5 hours to complete where as with the self hosted running it takes about 30 minutes. I also added caching of the pybind generated files as well which is something would would need to be removed. I am building against OCCT 8.0.0 that doesn't have VTK. there is no 3.14t build of VTK available and I personally didn't need to have that available. That is the only thing that would need to be dealt with and it might only have to be dealt with in cadquery and not in OCP. I don't know if OCP needs to use the Python c extension of VTK or if it only needs to link against a shared library of VTK not specifically compiled for use with Python.

If you have any questions about the code in that link above ask away. Most of it should be pretty self explanatory.

[adam-urbanczyk]

Thanks, good to know. I don't see any changes in your branch that are directly related to py3.14t only a lot of CI tweaks. Does it mean that with the right deps installed it just works (tm)?

[kdschlosser]

I will open up a PR once I get it to compile for all 3 platforms so we can look to see what exactly has changed. The PR will not be something that you would merge it simply be something where you can ask about specific code changes and also easily view the changes I made.

I do have a question. Is there a reason why the CI was coded to use the macos intel variant instead of using the apple silicon variant? the intel variant CPU's have not been used in quite some time which is why ask this. I changed it to use the apple silicon which is what all modern macs use.

[kdschlosser]

I also want to touch base on the VTK thing and why it causes an issue with 3.14t...

Compiling OCCT with VTK should be a non issue because OCCT links against the C++ code in VTK and really has nothing to do with python at all. The issue is because there is no version of VTK that is available on Conda that doesn't have the python bindings compiled with it. If there was then adding VTK would be a non issue because the build of OCCT doesn't care about VTK's Python bindings. But because we are compiling for a specific python version that information gets fed into Conda and that results on the VTK package not being available. If we could get the folks over at VTK to make a release that doesn't contain any Python bindings with it and to publish that to Conda then compiling with VTK is able to be done.

I have to do some more poking about with Conda to see if there is an explicit way to override the python requirements for a specific library.

[adam-urbanczyk]

To manage your expectations, I'll only accept minimal changes to the CI. If I understand you correctly, actually nothing free-threading specific is happening here and all the interesting stuff is in the relevant conda-forge feedstocks. If you can check with the maintainers of the vtk conda-forge feedstock regarding free-threaded, it would be certainly great.

[kdschlosser]

I am pretty sure that we don't have to bother the folks over at VTK for this. It's a matter of how we are requesting VTK from conda-forge. We should be able to tell conda-forge to ignore the python version from its search for a compatible library. since OCCT links against the C++ portion of VTK and OCP doesn't do anything with the python portion of VTK this would be the way of getting VTK support into the 3.14t build...

I do have a question and it's actually a really important one. Github Actions (the free variant) has a hard limit of 6 hours for the build time. The current CI build matrix compiles for python 3.10, 3.11 and 3.12 . It does this for each of those python versions with means an extremely long build time. If we could compile once for each platform that would cover Python 3.10, 3.12, 3.13 and 3.14 in a single go and then a compile for each platform for 3.14t (this has to be a separate build) that should be able to be done keeping the build time to under 6 hours. The longest build is for Windows which takes almost 5.5 hours and I think we can work the build matrix to it would compile everything needed in parallel so we don't eclipse that 6 hour time limit.. If cadquery has a paid github actions account then this might not apply and there may also be more cores and memory available which would speed up compile times. But if not this could be done and it would offer a greater range of supported python versions across all 3 platforms.

[adam-urbanczyk]

I'm not sure what you are actually suggesting above. If you can, please help the maintainers of the vtk-feedstock with this migration: conda-forge/vtk-feedstock#454 .

Also see: https://conda-forge.org/status/migration/?name=python314t

[kdschlosser]

alright. For VTK this is how conda-forge works.

Currently VTK on conda-forge is packaged together with it's python bindings. VTK is actually a C++ library and the Python bindings are a separate component. conda-forge doesn't have VTK without the Python bindings available.

OCCT can be compiled with or without VTK. Compiling with VTK only carries with it a need for the shared library and headers for VTK not the python bindings. This is because OCCT is compiled as a shared library and that is what OCP links to. OCP doesn't need VTK or it's python bindings and OCCT only needs VTK to link against and also has no use for it's python bindings. But because conda-forge doesn't have a package for VTK without the python bindings when the environment gets set up for 3.14t it naturally fails because what is being looked for is VTK with python bindings for 3.14t even tho the python bindings are not needed because they are not used.

Make sense?

Now because the Python bindings are irrelevant for VTK we simply need to instruct conda-forge to ignore trying to match the python version being used with a VTK distro. As long as we use a distro that is the correct version of VTK the python bindings portion doesn't make any difference.

[kdschlosser]

Now, as far as the other portions of my comment go regarding the python versions..

I just asked Claude to do some digging into being able to use the Python stable/limited ABI and while it is available for use in python since version 3.2 of Python, Pybind11 foes not support using it. nanobind which is similar to what PyBind11 does supports using it. Because OCP generates the binding code it would not be a giant undertaking to change the templates so that it would output nanobind shaped code instead of pybind shaped code. It would be a far easier task then if the binding code was hand written across hundreds of files.

Using the python Limited ABI means that you can compile a single extension module per platform and that extension module would work across many versions of Python. You can support python versions from 3.10 to the current version using a single extension module. For OCP you would need to compile an extension module for windows, macos and linux. You would also need to compile 3 extension modules for 3.14t, the free threaded builds are not supported in the limited ABI for python.

[jdegenstein]

I do have a question. Is there a reason why the CI was coded to use the macos intel variant instead of using the apple silicon variant? the intel variant CPU's have not been used in quite some time which is why ask this. I changed it to use the apple silicon which is what all modern macs use.

I created the PR that began the migration from Azure to Github Actions, so I wanted to respond to this specific question. The primary reason for this decision was to not unnecessarily drop support for macos x86_64. The second reason is that the migration from Azure to Github Actions was not simple and it was quite important to me to change as little as possible to help Adam feel some comfort that the PR was not a "boil the ocean" approach.

I am not an expert in this platform, so I would not be surprised if it is possible to compile for macos x86_64 on macos arm64. Once the macos-15-intel runner becomes EOL in around fall 2027 then it will become mandatory to re-evaluate.

Regarding the Github Actions build time limit of 6 hours -- yes it is worth paying attention to. The original version of my PR had caching enabled which helped me solve problems with the PR without waiting for the full time. If I recall correctly Adam quite reasonably disabled caching. This has the huge benefit of avoiding issues with cache invalidation which is IMHO nontrivial in the context of OCP.

[kdschlosser]

Thank you for taking the time to do the work to get it all moved over to using Github actions. It made things a lot easier when adding the 3.14t support to it....

Now in an ideal world the build system should be running in parallel so all windows builds across all versions of python.But as we know this is not an ideal world and having about 30 minutes of wiggle room can mean without a doubt that the 6 hour timeout can becomes an issue if the servers are under heavy load.

There are a few ways of handling this specific problem. the first way it to create a separate repo for the code generation. This is probably a good idea to do since it would allow for pinning specific CI revisions to specific releases made for the generated code. That would pull 30 minutes of the build time off the library compiling and put it to a completely different CI to run. That would make it far less likely to hit that 6 hour limit...

another thing that can be done is to change out using pybind11 for another code generator library. Because of how this library has been set up using templates it would not be all that much work to change those templates so it produces the format needed for another binding library It could also speed up compilation...

another suggestion would be creating individual binding code files and creating separate modules for each shared library that OCCT has. Mirroring the compiled shared library exactly.

[kdschlosser]

Oh and the whole thing with the macos deal is that mac's have not used intel since 2023. Apple started the migration in 2020 and everything except the pro line of computers was apple silicon (arm) since late 2022. so it's been 4 years now since apple has not been using x86_x64 architecture for the bulk of their computers being sold. This is the reason why I asked about the macos and the current state of the CI compiling only for the x86_x64 and not for arm. I did compile for arm so that does work, just so you know...