Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{2023.06}[foss/2023a] Clang v16.0.6 #737

Open
wants to merge 3 commits into
base: 2023.06-software.eessi.io
Choose a base branch
from

Conversation

casparvl
Copy link
Collaborator

I've seen that it failed in this PR #637 (comment) but that's been a while, so let me just retry.

Also: on our system we also build with RPATH, and have no issues installing this Clang version... It'll be interesting to see what might be different in EESSI.

Copy link

eessi-bot bot commented Sep 24, 2024

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

Copy link

eessi-bot bot commented Sep 24, 2024

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi.io-2023.06-compat

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

Copy link

eessi-bot bot commented Sep 24, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

Copy link

eessi-bot bot commented Sep 24, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Sep 24, 2024

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.09/pr_737/19698

date job status comment
Sep 24 14:08:05 UTC 2024 submitted job id 19698 awaits release by job manager
Sep 24 14:08:17 UTC 2024 released job awaits launch by Slurm scheduler
Sep 24 14:13:38 UTC 2024 running job 19698 is running
Sep 24 14:14:42 UTC 2024 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-19698.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
No artefacts were created or found.
Sep 24 14:14:42 UTC 2024 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-19698.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

Copy link

eessi-bot bot commented Sep 24, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

Copy link

eessi-bot bot commented Sep 24, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Sep 24, 2024

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.09/pr_737/19701

date job status comment
Sep 24 14:22:46 UTC 2024 submitted job id 19701 awaits release by job manager
Sep 24 14:23:07 UTC 2024 released job awaits launch by Slurm scheduler
Sep 24 14:24:16 UTC 2024 running job 19701 is running
Sep 24 16:44:35 UTC 2024 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-19701.out
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen3-1727195885.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen3/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen3
no other files in tarball
Sep 24 16:44:35 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-19701.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

Copy link

eessi-bot bot commented Sep 25, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

Updates by the bot instance boegel-bot-deucalion (click for details)
  • account casparvl has NO permission to send commands to the bot

Copy link

eessi-bot bot commented Sep 25, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Sep 25, 2024

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.09/pr_737/19833

date job status comment
Sep 25 08:48:41 UTC 2024 submitted job id 19833 awaits release by job manager
Sep 25 08:49:20 UTC 2024 released job awaits launch by Slurm scheduler
Sep 25 08:50:22 UTC 2024 running job 19833 is running
Sep 25 13:56:51 UTC 2024 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-19833.out
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen3-1727269891.tar.gzsize: 0 MiB (10752 bytes)
entries: 1
modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen3/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen3
2023.06/init/easybuild/eb_hooks.py
Sep 25 13:56:51 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-19833.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

Ok, same situation as reported here. Some executables have RPATH set, eg.

== 2024-09-25 12:50:06,242 run.py:689 DEBUG cmd "readelf -d /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-pdbutil" exited with exit code 0 and output:

Dynamic section at offset 0x766ce0 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libz3.so.4.12]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libzstd.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libtinfo.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000f (RPATH)              Library rpath: [/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/lib64
:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Z3/4.12.2-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GMP/6.2.1-GCCcore-12.3.0/lib64:/cvmfs/software.ees
si.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libxml2/2.11.4-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hwloc/2.9.1-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3
/software/Python/3.11.3-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Perl/5.36.1-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GCCcore/12.3.0/lib64:/cvmfs/software.eessi.i
o/versions/2023.06/software/linux/x86_64/amd/zen3/software/GCCcore/12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libpciaccess/0.17-GCCcore-12.3.0/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/softwar
e/numactl/2.0.16-GCCcore-12.3.0/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libffi/3.4.4-GCCcore-12.3.0/lib64/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/SQLite/3.42.0-GCCcore-12.3.0/lib/.
./lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Tcl/8.6.13-GCCcore-12.3.0/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libarchive/3.6.2-GCCcore-12.3.0/lib/../lib64:/cvmfs/software.eessi.io/versi
ons/2023.06/software/linux/x86_64/amd/zen3/software/cURL/8.0.1-GCCcore-12.3.0/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenSSL/1.1/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GCCcor
e/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0:/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/../lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libffi
/3.4.4-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib:/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libpciaccess/0.17-GCCcore-12.3.0/lib:/cvmf
s/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/numactl/2.0.16-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libffi/3.4.4-GCCcore-12.3.0/lib64:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x8
6_64/amd/zen3/software/SQLite/3.42.0-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Tcl/8.6.13-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libarchive/3.6.2-GCCcore-12.3.0/lib:
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/cURL/8.0.1-GCCcore-12.3.0/lib:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenSSL/1.1/lib:$ORIGIN/../lib]

looks just fine. But others dont:


== 2024-09-25 12:50:06,272 run.py:689 DEBUG cmd "readelf -d /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info" exited with exit code 0 and output:

Dynamic section at offset 0x2d70 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libomp.so]
 0x0000000000000001 (NEEDED)             Shared library: [libomptarget.so.16]
 0x0000000000000001 (NEEDED)             Shared library: [libhwloc.so.15]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]

They also have the wrong interpreter set, for the same files. This is correct:

== 2024-09-25 12:50:06,208 run.py:247 INFO running cmd: file /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-pdbutil
== 2024-09-25 12:50:06,216 run.py:689 DEBUG cmd "file /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-pdbutil" exited with exit code 0 and output:
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-pdbutil: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/
ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped

But this is wrong:

== 2024-09-25 12:50:06,243 run.py:247 INFO running cmd: file /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info
== 2024-09-25 12:50:06,248 run.py:689 DEBUG cmd "file /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info" exited with exit code 0 and output:
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not strippe
d

@casparvl
Copy link
Collaborator Author

I don't get it, on our own system, it is correct:

$ readelf -d /sw/arch/RHEL8/EB_production/2023/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info

Dynamic section at offset 0x3d40 contains 36 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libomp.so]
 0x0000000000000001 (NEEDED)             Shared library: [libomptarget.so.16]
 0x0000000000000001 (NEEDED)             Shared library: [libhwloc.so.15]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [/sw/arch/RHEL8/EB_production/2023/software/Clang/16.0.6-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/Clang/16.0.6-GCCcore-12.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/sw/arch/RHEL8/EB_production/2023/software/Z3/4.12.2-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/GMP/6.2.1-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/ncurses/6.4-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/libxml2/2.11.4-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/hwloc/2.9.1-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/binutils/2.40-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/Python/3.11.3-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/Perl/5.36.1-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/GCCcore/12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/GCCcore/12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0:/sw/arch/RHEL8/EB_production/2023/software/libpciaccess/0.17-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/numactl/2.0.16-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/libffi/3.4.4-GCCcore-12.3.0/lib64:/sw/arch/RHEL8/EB_production/2023/software/libffi/3.4.4-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/SQLite/3.42.0-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/Tcl/8.6.13-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/libreadline/8.2-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/libarchive/3.6.2-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/XZ/5.4.2-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/cURL/8.0.1-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/OpenSSL/1.1/lib:/sw/arch/RHEL8/EB_production/2023/software/bzip2/1.0.8-GCCcore-12.3.0/lib:/sw/arch/RHEL8/EB_production/2023/software/zlib/1.2.13-GCCcore-12.3.0/lib:$ORIGIN/../lib]

Of course, we are not using a sysroot, but we are using RPATH. And that seems to work here...

The installation has 3 stages... I'm guessing it does something like build Clang with GCC, then build Clang with Clang (not sure what the 3rd stage is). Maybe depending on which stage this is built in, does it set RPATH and interpreter correctly? I.e. maybe the rpath stuff gets lost somehow in later stages?

@casparvl
Copy link
Collaborator Author

I had the feeling I had seen this issue before. And I think I have. And fixed it. https://github.com/easybuilders/easybuild-easyblocks/pull/2799/files

The issue back then (if I remember correctly) was that the stage 2 and 3 compilers weren't being wrapped by the RPATH wrappers. I think the same might be happening now. For some reason, the logic in the EasyBlock is, I guess, not working when using --sysroot. That's one major difference with how we run locally, and might be why with just RPATH (and no --sysroot) this issue doesn't pop up.

I'll need to check the logs again for those specific debugging statements surrounding the wrapping of the compilers...

@casparvl
Copy link
Collaborator Author

Ah, the link line for llvm-omp-device-info indeed looks wrong:

[ 98%] Linking CXX executable /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/bin/llvm-omp-device-info
cd /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/runtimes/runtimes-bins/openmp/libomptarget/tools/deviceinfo && /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CMake/3.26.3-GCCcore-12.3.0/bin/cmake -E cmake_link_script CMakeFiles/llvm-omp-device-info.dir/link.txt --verbose=1
/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang++ --target=x86_64-unknown-linux-gnu --sysroot=/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64 -O2 -ftree-vectorize -march=native -fno-math-errno -Wno-unused-command-line-argument -Wno-unused-command-line-argument -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -std=c++17 -O3 -DNDEBUG -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Z3/4.12.2-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Z3/4.12.2-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GMP/6.2.1-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GMP/6.2.1-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libxml2/2.11.4-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/libxml2/2.11.4-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hwloc/2.9.1-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hwloc/2.9.1-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Perl/5.36.1-GCCcore-12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Perl/5.36.1-GCCcore-12.3.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GCCcore/12.3.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/GCCcore/12.3.0/lib  -Wl,-rpath-link,/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./lib  -Wl,--gc-sections "CMakeFiles/llvm-omp-device-info.dir/llvm-omp-device-info.cpp.o" -o /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/bin/llvm-omp-device-info  -Wl,-rpath,"\$ORIGIN/../lib:/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./lib:/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/runtimes/runtimes-bins/openmp/runtime/src:/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/lib:" ../../../runtime/src/libomp.so /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/lib/libomptarget.so.16 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hwloc/2.9.1-GCCcore-12.3.0/lib/libhwloc.so -lm -ldl

Don't be fooled by the -Wl,rpath, that is set by the build system. What we should have had here was an invocation of the clang++ wrapper. Instead, it invoked the full path to the clang++ compiler /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang++. That is wrong, we need to wrapper to add every -L argument to the RPATH.

The question is: why is it not calling the wrapper? If we know that, we can fix this.

@casparvl
Copy link
Collaborator Author

Not sure why this 'Re-run cmake no build system arguments' is happening, but this seems to be picking up on the full compiler path:

[ 88%] Performing configure step for 'runtimes'
cd /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/runtimes/runtimes-bins && /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CMake/3.26.3-GCCcore-12.3.0/bin/cmake -DCMAKE_C_COMPILER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/l
lvm.obj.3/./bin/clang -DCMAKE_CXX_COMPILER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang++ -DCMAKE_ASM_COMPILER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang -DCMAKE_LINKER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/
llvm.obj.3/./bin/ld.lld -DCMAKE_AR=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/llvm-ar -DCMAKE_RANLIB=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/llvm-ranlib -DCMAKE_NM=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/.
/bin/llvm-nm -DCMAKE_OBJDUMP=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/llvm-objdump -DCMAKE_OBJCOPY=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/llvm-objcopy -DCMAKE_STRIP=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj
.3/./bin/llvm-strip -DCMAKE_READELF=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Clang/16.0.6-GCCcore-12.3.0 -DCMAKE_SYSROOT=/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64 -DLLVM_BINARY_DIR=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3 -DLLVM_CONFIG_PATH=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-sv -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=16.0.6 -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/gmake -DCMAKE_C_COMPILER_LAUNCHER= -DCMAKE_CXX_COMPILER_LAUNCHER= -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON "-DLLVM_ENABLE_RUNTIMES=libunwind;libcxx;libcxxabi;compiler-rt;openmp" -DLIBOMPTARGET_NVPTX_BC_LINKER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.2/bin/llvm-link -DLIBOMP_HWLOC_INSTALL_DIR=/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hwloc/2.9.1-GCCcore-12.3.0 -DLIBOMP_USE_HWLOC=ON -DLIBOMPTARGET_NVPTX_BC_LINKER=/tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.2/bin/llvm-link "-GUnix Makefiles" -S /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm-project-16.0.6.src/llvm/runtimes/../../runtimes -B /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/runtimes/runtimes-bins
Re-run cmake no build system arguments
-- The C compiler identification is Clang 16.0.6
-- The CXX compiler identification is Clang 16.0.6
-- The ASM compiler identification is GNU
-- Found assembler: /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/bot/easybuild/build/Clang/16.0.6/GCCcore-12.3.0/llvm.obj.3/./bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test Terminfo_LINKABLE
-- Performing Test Terminfo_LINKABLE - Success
-- Found Terminfo: /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libtinfo.so
-- Found ZLIB: /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libz.so (found version "1.2.13")
-- Found zstd: /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libzstd.so
-- Found LibXml2: /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libxml2.so (found version "2.11.4")

That is probably, subsequently, used. I'm not quite sure why it would be different on our local system... Note that the -DCMAKE-CXX_COMPILER passed here is already the full path to the stage3 compiler (and thus wrong, since it is not wrapped).

@casparvl
Copy link
Collaborator Author

Very strange: I cannot reproduce this on our local system when building with EESSI-extend. I see the exact same link line for llvm-omp-device-info. Also calling the g++ by full path. So I would expect it to not set an RPATH. Yet... somehow... the RPATH sanity check shows that the correct RPATH is set. This is puzzling, how is that set if the compiler wrappers are not used?!

@casparvl
Copy link
Collaborator Author

casparvl commented Sep 25, 2024

I also see (on the local build and the one on AWS):

-- Installing: /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info
-- Set runtime path of "/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Clang/16.0.6-GCCcore-12.3.0/bin/llvm-omp-device-info" to "$ORIGIN/../lib"

On the local system, it does not seem to overwrite the RPATH that was already set. Also, there is definitely no RUNPATH in that file (I don't think you can have both, can you?). Things just keep getting stranger... :|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant