[2023.06]{foss/2021b} SciPy-bundle v2021.10 #306

laraPPr · 2023-07-14T14:13:58Z

No description provided.

eessi-bot · 2023-07-14T14:14:03Z

Instance eessi-bot-citc-aws is configured to build:

arch x86_64/generic for repo eessi-2021.12
arch x86_64/generic for repo eessi-2023.06-compat
arch x86_64/generic for repo eessi-2023.06-software
arch x86_64/intel/haswell for repo eessi-2021.12
arch x86_64/intel/haswell for repo eessi-2023.06-compat
arch x86_64/intel/haswell for repo eessi-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-2021.12
arch x86_64/intel/skylake_avx512 for repo eessi-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-2023.06-software
arch x86_64/amd/zen2 for repo eessi-2021.12
arch x86_64/amd/zen2 for repo eessi-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-2023.06-software
arch x86_64/amd/zen3 for repo eessi-2021.12
arch x86_64/amd/zen3 for repo eessi-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-2023.06-software
arch aarch64/generic for repo eessi-2021.12
arch aarch64/generic for repo eessi-2023.06-compat
arch aarch64/generic for repo eessi-2023.06-software
arch aarch64/neoverse_n1 for repo eessi-2021.12
arch aarch64/neoverse_n1 for repo eessi-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi-2023.06-software
arch aarch64/neoverse_v1 for repo eessi-2021.12
arch aarch64/neoverse_v1 for repo eessi-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi-2023.06-software

laraPPr · 2023-07-14T14:19:50Z

bot: build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software

eessi-bot · 2023-07-14T14:19:53Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software from laraPPr
- expanded format: build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software
handling command build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software resulted in:
- submitted job 5810, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)

eessi-bot · 2023-07-14T14:20:00Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_v1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.07/pr_306/5810

date	job status	comment
Jul 14 14:19:59 UTC 2023	submitted	job id `5810` awaits release by job manager
Jul 14 14:20:15 UTC 2023	released	job awaits launch by Slurm scheduler
Jul 14 14:24:19 UTC 2023	running	job `5810` is running
Jul 14 14:55:53 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-5810.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1689346500.tar.gz` size: 2 MiB (2478219 bytes) entries: 959 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` software under 2023.06/software/linux/aarch64/neoverse_v1/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`

casparvl · 2023-07-18T14:58:43Z

@laraPPr do you have access to the AWS cluster? It would probably be faster to try and do this build yourself, rather than let the bot do it. It'll allow you to go through 'regular' EasyBuild debugging procedures (e.g. get the build environment with --dump-env-script and then try to rerun the test suite manually). If that reproduces your issue, it's much easier to debug, as you can probably start modifying things in the test suite, make it more verbose, increase tolerance, etc, to see where it fails.

I have seen numpy failures before that were related to the version of OpenBLAS underneath. I think we also see more test failures for OpenBLAS on ARM, right? I think we ignore those, since they are 'only' tolerance failures. But then any test on top (like a numpy test, which calls OpenBLAS) will likely also fail.

The real solution here is to address it with the OpenBLAS devs and get to the bottom of it. If it is expected / 'ok' that these tolerances are exceeded. If so, tests that are written on top will also have to be adjusted to account for these increased tolerances. In any case, you could address this in an issue to the numpy devs as well. It's their test after all, would be interesting to know if they ever tested on aarch64.

laraPPr · 2023-07-24T08:29:12Z

@laraPPr do you have access to the AWS cluster? It would probably be faster to try and do this build yourself, rather than let the bot do it. It'll allow you to go through 'regular' EasyBuild debugging procedures (e.g. get the build environment with --dump-env-script and then try to rerun the test suite manually). If that reproduces your issue, it's much easier to debug, as you can probably start modifying things in the test suite, make it more verbose, increase tolerance, etc, to see where it fails.

I do not think I have enough access to the AWS cluster to build it myself. I've also never done 'regular' EasyBuild debugging procedures but I am willing to try.

laraPPr · 2023-07-24T08:39:22Z

I have seen numpy failures before that were related to the version of OpenBLAS underneath. I think we also see more test failures for OpenBLAS on ARM, right? I think we ignore those, since they are 'only' tolerance failures. But then any test on top (like a numpy test, which calls OpenBLAS) will likely also fail.

Yes there was a hook set for the installation of OpenBLAS on aarch64 that relaxed the allowed number of failing numerical LAPACK tests.

boegel · 2023-08-02T13:54:34Z

In EESSI pilot 2021.12, we installed SciPy-bundle-2020.03-foss-2020a-Python-3.8.2.eb with --skip-test-step for aarch64 to dance around failing tests, see https://github.com/EESSI/software-layer/blob/2f6709f019d9b4a0d7852883842d06da46b961fc/EESSI-pilot-install-software.sh#L220C25-L220C41
We did not have to do this for SciPy-bundle-2021.05-foss-2021a.eb however.

We have run into issues like this before in EasyBuild, see easybuilders/easybuild-easyconfigs#11959

I think we should try and asses how "serious" these test failures are, but most likely we'll need to move forward with ignoring the failing tests, at least for now, since fixing may require significant effort...
It's not like we guarantee for other software (where we may not be running tests at all) that it's 100% bugfree, so the effort we do has to be within reason imho: there's no way we can guarantee that all software we include in EESSI doesn't have any bugs at all, we have to basically provide it as-is (with some exceptions where patching is obvious/easy).

boegel · 2023-08-05T15:06:28Z

bot: build repo:eessi-2023.06-software

eessi-bot · 2023-08-05T15:06:31Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build repo:eessi-2023.06-software from boegel
- expanded format: build repository:eessi-2023.06-software
handling command build repository:eessi-2023.06-software resulted in:
- submitted job 6225, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6226, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6227, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6228, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6229, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6230, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6231, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)
- submitted job 6232, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)

eessi-bot · 2023-08-05T15:07:00Z

New job on instance eessi-bot-citc-aws for architecture x86_64-generic for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6225

date	job status	comment
Aug 05 15:06:59 UTC 2023	submitted	job id `6225` awaits release by job manager
Aug 05 15:07:54 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:10 UTC 2023	running	job `6225` is running
Aug 05 16:04:24 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6225.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1691251361.tar.gz` size: 129 MiB (135708614 bytes) entries: 12339 modules under 2023.06/software/linux/x86_64/generic/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/x86_64/generic/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/x86_64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:58 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1691251361.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:01Z

New job on instance eessi-bot-citc-aws for architecture x86_64-intel-haswell for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6226

date	job status	comment
Aug 05 15:07:00 UTC 2023	submitted	job id `6226` awaits release by job manager
Aug 05 15:07:52 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:08 UTC 2023	running	job `6226` is running
Aug 05 16:04:22 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6226.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-haswell-1691251418.tar.gz` size: 130 MiB (136488715 bytes) entries: 12339 modules under 2023.06/software/linux/x86_64/intel/haswell/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/x86_64/intel/haswell/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/x86_64/intel/haswell `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:05 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-haswell-1691251418.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:03Z

New job on instance eessi-bot-citc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6227

date	job status	comment
Aug 05 15:07:02 UTC 2023	submitted	job id `6227` awaits release by job manager
Aug 05 15:07:50 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:06 UTC 2023	running	job `6227` is running
Aug 05 15:59:00 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6227.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1691251075.tar.gz` size: 129 MiB (135825699 bytes) entries: 12339 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:31 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1691251075.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:04Z

New job on instance eessi-bot-citc-aws for architecture x86_64-amd-zen2 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6228

date	job status	comment
Aug 05 15:07:04 UTC 2023	submitted	job id `6228` awaits release by job manager
Aug 05 15:07:48 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:04 UTC 2023	running	job `6228` is running
Aug 05 16:01:11 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6228.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1691251201.tar.gz` size: 130 MiB (136431624 bytes) entries: 12339 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:22 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1691251201.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:06Z

New job on instance eessi-bot-citc-aws for architecture x86_64-amd-zen3 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6229

date	job status	comment
Aug 05 15:07:05 UTC 2023	submitted	job id `6229` awaits release by job manager
Aug 05 15:07:46 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:02 UTC 2023	running	job `6229` is running
Aug 05 15:52:27 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6229.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1691250665.tar.gz` size: 130 MiB (136447329 bytes) entries: 12339 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/x86_64/amd/zen3/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/x86_64/amd/zen3 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:40 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen3-1691250665.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:07Z

New job on instance eessi-bot-citc-aws for architecture aarch64-generic for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6230

date	job status	comment
Aug 05 15:07:07 UTC 2023	submitted	job id `6230` awaits release by job manager
Aug 05 15:07:44 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:11:00 UTC 2023	running	job `6230` is running
Aug 05 16:05:27 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6230.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1691251476.tar.gz` size: 125 MiB (131889732 bytes) entries: 12339 modules under 2023.06/software/linux/aarch64/generic/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/aarch64/generic/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/aarch64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:49 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1691251476.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:09Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_n1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6231

date	job status	comment
Aug 05 15:07:08 UTC 2023	submitted	job id `6231` awaits release by job manager
Aug 05 15:07:42 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:10:58 UTC 2023	running	job `6231` is running
Aug 05 16:17:41 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6231.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_n1-1691252183.tar.gz` size: 125 MiB (131845623 bytes) entries: 12339 modules under 2023.06/software/linux/aarch64/neoverse_n1/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/aarch64/neoverse_n1/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/aarch64/neoverse_n1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:05:14 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-neoverse_n1-1691252183.tar.gz` to S3 bucket succeeded

eessi-bot · 2023-08-05T15:07:11Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_v1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6232

date	job status	comment
Aug 05 15:07:10 UTC 2023	submitted	job id `6232` awaits release by job manager
Aug 05 15:07:40 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 05 15:10:57 UTC 2023	running	job `6232` is running
Aug 05 15:29:09 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-6232.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1691249284.tar.gz` size: 2 MiB (2493350 bytes) entries: 959 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` software under 2023.06/software/linux/aarch64/neoverse_v1/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`

boegel · 2023-08-11T12:23:09Z

@laraPPr To ignore the failing numpy tests for now, maybe we can try simply removing the corresponding test modules (fft/tests/test_pocketfft.py and polynomial/tests/test_classes.py) right before running the tests, via pretestopts for example (controlled via a hook).
Maybe we'll need to run python -c 'import fft.tests; print(fft.tests.__file__)' to figure out the right place.

laraPPr · 2023-08-16T11:03:14Z

bot: build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software

eessi-bot · 2023-08-16T11:03:17Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software from laraPPr
- expanded format: build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software
handling command build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software resulted in:
- submitted job 6508, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)

eessi-bot · 2023-08-16T11:03:24Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_v1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6508

date	job status	comment
Aug 16 11:03:23 UTC 2023	submitted	job id `6508` awaits release by job manager
Aug 16 11:03:48 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 16 11:06:50 UTC 2023	running	job `6508` is running
Aug 16 11:24:09 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-6508.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1692185020.tar.gz` size: 2 MiB (2496069 bytes) entries: 959 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` software under 2023.06/software/linux/aarch64/neoverse_v1/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`

boegel

@laraPPr You will also need to define pre_test_hook that picks up on PRE_TEST_HOOKS, similar to the pre_configure_hook that's already there.

eb_hooks.py

laraPPr · 2023-08-21T09:30:21Z

bot: build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software

eessi-bot · 2023-08-21T09:30:24Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software from laraPPr
- expanded format: build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software
handling command build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software resulted in:
- submitted job 6642, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)

eessi-bot · 2023-08-21T09:30:30Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_v1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6642

date	job status	comment
Aug 21 09:30:29 UTC 2023	submitted	job id `6642` awaits release by job manager
Aug 21 09:31:06 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 21 09:35:09 UTC 2023	running	job `6642` is running
Aug 21 09:42:17 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-6642.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1692610886.tar.gz` size: 0 MiB (89775 bytes) entries: 3 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_v1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`

Fix indentation line 267

laraPPr · 2023-08-22T09:59:33Z

bot: build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software

eessi-bot · 2023-08-22T09:59:36Z

Updates by the bot instance eessi-bot-citc-aws (click for details)

received bot command build arch:aarch64/neoverse_v1 repo:eessi-2023.06-software from laraPPr
- expanded format: build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software
handling command build architecture:aarch64/neoverse_v1 repository:eessi-2023.06-software resulted in:
- submitted job 6677, for details & status see [2023.06]{foss/2021b} SciPy-bundle v2021.10 #306 (comment)

eessi-bot · 2023-08-22T09:59:43Z

New job on instance eessi-bot-citc-aws for architecture aarch64-neoverse_v1 for repository eessi-2023.06-software in job dir /mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.08/pr_306/6677

date	job status	comment
Aug 22 09:59:42 UTC 2023	submitted	job id `6677` awaits release by job manager
Aug 22 09:59:52 UTC 2023	released	job awaits launch by Slurm scheduler
Aug 22 10:03:55 UTC 2023	running	job `6677` is running
Aug 22 10:59:01 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-6677.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1692701851.tar.gz` size: 125 MiB (131383215 bytes) entries: 12339 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all `Eigen/3.3.9-GCCcore-11.2.0.lua` `hypothesis/6.14.6-GCCcore-11.2.0.lua` `pybind11/2.7.1-GCCcore-11.2.0.lua` `SciPy-bundle/2021.10-foss-2021b.lua` software under 2023.06/software/linux/aarch64/neoverse_v1/software `Eigen/3.3.9-GCCcore-11.2.0` `hypothesis/6.14.6-GCCcore-11.2.0` `pybind11/2.7.1-GCCcore-11.2.0` `SciPy-bundle/2021.10-foss-2021b` other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Aug 22 14:04:56 UTC 2023	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-neoverse_v1-1692701851.tar.gz` to S3 bucket succeeded

boegel · 2023-08-22T14:00:33Z

@laraPPr Hook looks good to me now, let's make sure builds are done for all CPU targets so we can deploy this?

We should also open an internal issue with details on the failing tests, and start looking into reporting the problems upstream. Let's do at least the internal issue before merging this PR (to avoid forgetting about it).

laraPPr · 2023-08-22T14:06:00Z

@laraPPr Hook looks good to me now, let's make sure builds are done for all CPU targets so we can deploy this?

We should also open an internal issue with details on the failing tests, and start looking into reporting the problems upstream. Let's do at least the internal issue before merging this PR (to avoid forgetting about it).

Should I than also look at other tests that might have failed and were skipped?

boegel · 2023-08-22T17:17:37Z

@laraPPr Hook looks good to me now, let's make sure builds are done for all CPU targets so we can deploy this?
We should also open an internal issue with details on the failing tests, and start looking into reporting the problems upstream. Let's do at least the internal issue before merging this PR (to avoid forgetting about it).

Should I than also look at other tests that might have failed and were skipped?

The issue should cover all failing tests in this SciPy-bundle, yes, so perhaps also scipy (if there are failing tests there).

You should be able to spot if there's additional failing test suite easily by looking for "Test failure ignored" in the EasyBuild log that is included with the installation.

boegel · 2023-08-22T17:18:12Z

All staging PRs merged, so deployment is being done right now, should be fully in place in about an hour or so.

boegel · 2023-08-22T17:58:11Z

@laraPPr Sorry, my mistake, the way to check for failing test steps is to check for lines that start with "``", since that's the message emitted by the echo that the hook injects.
I only see this for the numpy test suite in `/cvmfs/pilot.eessi-hpc.org/versions/2023.06/software/linux/aarch64/neoverse_v1/software/SciPy-bundle/2021.10-foss-2021b/easybuild/easybuild-SciPy-bundle-2021.10-20230822.105656.log.bz2`:

=========================== short test summary info ============================
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_identity - AssertionError:
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifft[None] - AssertionErr...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifft[backward] - Assertio...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifft[ortho] - AssertionEr...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifft[forward] - Assertion...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifft2 - AssertionError:
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_ifftn - AssertionError:
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_irfft2 - AssertionError:
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_irfftn - AssertionError:
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_all_1d_norm_preserving - ...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_dtypes[float16] - Asserti...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_dtypes[float32] - Asserti...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_dtypes[float64] - Asserti...
FAILED fft/tests/test_pocketfft.py::TestFFT1D::test_dtypes[float128] - Assert...
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[Polynomial]
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[Legendre]
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[Chebyshev]
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[Laguerre]
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[Hermite] - ...
FAILED polynomial/tests/test_classes.py::test_bad_conditioned_fit[HermiteE]
= 20 failed, 14429 passed, 81 skipped, 1253 deselected, 22 xfailed, 79 warnings in 171.82s (0:02:51) =
ignoring failing tests

laraPPr · 2023-08-23T08:29:41Z

created issue for the numy test failures with SciPy-bundle:

#318

… into scipy-bundl-2021b

split Lmod hooks merge ok despite some stupidly failing CI workflow

[2023.06]{foss/2021b} SciPy-bundle neoverse_v1

1adfe5e

laraPPr mentioned this pull request Jul 14, 2023

{2023.06}[foss/2021b] GROMACS v2021.5 #304

Merged

laraPPr added pilot-2021.06 pilot-2023.06 labels Jul 14, 2023

casparvl mentioned this pull request Jul 20, 2023

{2023.06} foss/2022b #309

Merged

laraPPr removed the pilot-2021.06 label Jul 24, 2023

boegel changed the title ~~[2023.06]{foss/2021b} SciPy-bundle neoverse_v1~~ [2023.06]{foss/2021b} SciPy-bundle v2021.01 Aug 3, 2023

boegel changed the title ~~[2023.06]{foss/2021b} SciPy-bundle v2021.01~~ [2023.06]{foss/2021b} SciPy-bundle v2021.10 Aug 3, 2023

Update eb_hooks.py

4f54fd6

boegel requested changes Aug 16, 2023

View reviewed changes

eb_hooks.py Outdated Show resolved Hide resolved

Update eb_hooks.py

036a6dd

Update eb_hooks.py

44f7ff5

Fix indentation line 267

laraPPr added the ready-to-deploy Mark a PR as ready to deploy label Aug 22, 2023

boegel added bot:deploy Ask bot to deploy missing software installations to EESSI and removed ready-to-deploy Mark a PR as ready to deploy labels Aug 22, 2023

laraPPr mentioned this pull request Aug 23, 2023

numpy/scipy test suite failures on aarch64/neoverse_v1 (SciPy-bundle) #318

Open

lara added 3 commits August 23, 2023 10:35

Trigger CI/CD pipeline

f34854c

Merge branch 'scipy-bundl-2021b' of github.com:laraPPr/software-layer…

d3d5f5f

… into scipy-bundl-2021b

Trigger CI/CD pipeline

938cf06

boegel approved these changes Aug 23, 2023

View reviewed changes

boegel merged commit 468d2c2 into EESSI:2023.06 Aug 23, 2023
65 checks passed

boegel mentioned this pull request Sep 1, 2023

{2023.06} foss/2022a #310

Merged

trz42 pushed a commit to trz42/software-layer that referenced this pull request Apr 7, 2024

Merge pull request EESSI#306 from trz42/split_lmod_hooks

22e87e7

split Lmod hooks merge ok despite some stupidly failing CI workflow

[2023.06]{foss/2021b} SciPy-bundle v2021.10 #306

[2023.06]{foss/2021b} SciPy-bundle v2021.10 #306

Conversation

laraPPr commented Jul 14, 2023

eessi-bot bot commented Jul 14, 2023

laraPPr commented Jul 14, 2023

eessi-bot bot commented Jul 14, 2023 • edited Loading

eessi-bot bot commented Jul 14, 2023 • edited Loading

casparvl commented Jul 18, 2023

laraPPr commented Jul 24, 2023

laraPPr commented Jul 24, 2023

boegel commented Aug 2, 2023

boegel commented Aug 5, 2023

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

eessi-bot bot commented Aug 5, 2023 • edited Loading

boegel commented Aug 11, 2023

laraPPr commented Aug 16, 2023

eessi-bot bot commented Aug 16, 2023 • edited Loading

eessi-bot bot commented Aug 16, 2023 • edited Loading

boegel left a comment

Choose a reason for hiding this comment

laraPPr commented Aug 21, 2023

eessi-bot bot commented Aug 21, 2023 • edited Loading

eessi-bot bot commented Aug 21, 2023 • edited Loading

laraPPr commented Aug 22, 2023

eessi-bot bot commented Aug 22, 2023 • edited Loading

eessi-bot bot commented Aug 22, 2023 • edited Loading

boegel commented Aug 22, 2023

laraPPr commented Aug 22, 2023 • edited Loading

boegel commented Aug 22, 2023

boegel commented Aug 22, 2023

boegel commented Aug 22, 2023

laraPPr commented Aug 23, 2023

eessi-bot bot commented Jul 14, 2023 •

edited

Loading

eessi-bot bot commented Jul 14, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 5, 2023 •

edited

Loading

eessi-bot bot commented Aug 16, 2023 •

edited

Loading

eessi-bot bot commented Aug 16, 2023 •

edited

Loading

eessi-bot bot commented Aug 21, 2023 •

edited

Loading

eessi-bot bot commented Aug 21, 2023 •

edited

Loading

eessi-bot bot commented Aug 22, 2023 •

edited

Loading

eessi-bot bot commented Aug 22, 2023 •

edited

Loading

laraPPr commented Aug 22, 2023 •

edited

Loading