Installation for Triton

[parsifal-47]

Hello All,
This PR Integrates Triton (https://github.com/triton-lang/triton) to compiler explorer's infra. It also includes CPU backend from https://github.com/microsoft/triton-shared

It almost works, I see the following:

$ ./bin/ce_install install triton
...
Successfully built triton
Installing collected packages: filelock, triton
Successfully installed filelock-3.16.0 triton-2.1.0
2024-09-16 02:40:59,801 lib.installation_context INFO     Moving from staging (/opt/compiler-explorer/staging/ed9924e0-5a73-46b5-aa3b-6a24632e455d/triton-2.1.0) to final destination (/opt/compiler-explorer/triton-2.1.0)
2024-09-16 02:40:59,831 lib.installation_context INFO     Destination /opt/compiler-explorer/triton-2.1.0 exists, temporarily moving out of the way (to /opt/compiler-explorer/staging/ed9924e0-5a73-46b5-aa3b-6a24632e455d.orig)
2024-09-16 02:40:59,831 lib.installation_context INFO     Removing temporarily moved /opt/compiler-explorer/staging/ed9924e0-5a73-46b5-aa3b-6a24632e455d.orig
2024-09-16 02:41:00,036 lib.ce_install  ERROR    compilers/triton 2.1.0 installed OK, but doesn't appear as installed after
Installing libraries/triton/triton-shared main
2024-09-16 02:41:00,036 lib.ce_install  INFO     libraries/triton/triton-shared main is already installed, skipping
0 packages installed OK, 1 skipped, and 1 failed installation
Failed:
  compilers/triton 2.1.0

What am missing?

Besides that, I am open for discussion, let me know what you think!
Thank you!

[mattgodbolt]

Hi! sorry it's bad news here -- supporting compilers written in python hasn't been very easy for us to support in the past, and I think you're falling foul of that here.

[mattgodbolt]

This is probably the culprit here. We don't have a "real" python3 in our environment where the compilers run and/or are installed. As you know python has a rich history of complex package management systems and we don't currently well support compilres that are written in python.

The computer that creates the venv will not be the same one that executes the compilers, so a lot of system dependencies could change.

Unfortunately we don't have a great solution for this currently.

If you have Discord you could dro pinto our discord server for a chat about this, we're open to ideas. But python's installation system is a little hostile to our current setup as it stands.

[mattgodbolt]

This will build on our "admin" installer node, which isn't a great place for this to go. In other languages we try and build a "hermetic", relocatable tarball that we use as the source of truth.

From (bitter) experience in my day job, python and its various virtual environments aren't great at this kind of thing. pip and poetry don't handle native dependencies at all, conda is not relocatable.

Also hardcoding /opt/compiler-explorer here is not a great idea as we support other installation locations (though pragmatically this will work for the CE main install)

[parsifal-47]

Hi! sorry it's bad news here -- supporting compilers written in python hasn't been very easy for us to support in the past, and I think you're falling foul of that here.

Got it, let me think and get back to you on discord, maybe I could find a way to have less python, thank you for the review!