[WIP] Project subdirectories in dev.eessi.io install path and EESSI-extend integration

[Neves-P]

Please don't merge for now

Previous installations to dev.eessi.io are landing in /cvmfs/dev.eessi.io/versions/2023.06/ but should also include a project specific subdirectory such as /cvmfs/dev.eessi.io/versions/2023.06/ESPResSo/. Right now, this should be done by setting the environment variable $EESSI_DEV_PROJECT in the dev.eessi.io bot build script used in each of the projects (example: https://github.com/EESSI/dev.eessi.io-scripts/blob/main/bot/bot-build-dev.eessi.io.slurm), but maybe this can be improved by grabbing the information in the job.cfg file which includes a field like this:

[repository]
...
repo_id: repo_name
...

For now, however, I kept it simple to reduce points of failure until we are sure the installations land in the right place.

EESSI-extend

An issue why this PR shouldn't be merged right now: EasyBuild is ignoring the cvmfs repository and installpath overrides and trying to write to software.eessi.io (and failing, because we mount this as read-only). Because this was working earlier I think that some wires are getting crossed when we switched to EESSI-extend in EESSI-install-software.sh (see #790 ) and similarly to what is described at #802. I've not managed to figure out where things are going wrong.

Relevant error from log:

== building and installing ESPResSo/4.2.2-foss-2023a-1f672a00e22a9bc5fec1f7070bd28ff7718a706e...
  >> installation prefix: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/ESPResSo/4.2.2-foss-2023a-1f672a00e22a9bc5fec1f7070bd28ff7718a706e
== FAILED: Installation ended unsuccessfully (build directory: /tmp/bot/easybuild/build/ESPResSo/4.2.2/foss-2023a-1f672a00e22a9bc5fec1f7070bd28ff7718a706e): build failed (first 300 chars): Failed to create lock /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/.locks/_cvmfs_software.eessi.io_versions_2023.06_software_linux_x86_64_amd_zen2_software_ESPResSo_4.2.2-foss-2023a-1f672a00e22a9bc5fec1f7070bd28ff7718a706e.lock: [Errno 38] Function not implemente (took 0 secs)

I've tried changing the installpath without setting the EESSI-extend envvars like $EESSI_SITE_INSTALL and $EESSI_USER_INSTALL which might work, but I think users or sites should be able to change these to their needs when building for dev.eessi.io.

To try to figure this out I also added a bunch of extra output to the logs that should be removed before this PR is merged. I will do so before making this as ready for review.

I've tried a few approaches already and I'm out of ideas how to correct the installpath...

TODO after this PR

Address #799

[eessi-bot]

Instance eessi-bot-mc-aws is configured to build for:

• architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
• repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

[riscv-eessi-io-bot]

Instance eessi-bot-riscv is configured to build for:

• architectures: riscv64/generic
• repositories: riscv.eessi.io-20240402

[eessi-bot]

Instance eessi-bot-mc-azure is configured to build for:

• architectures: x86_64/amd/zen4
• repositories: eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi.io-2023.06-compat

[Neves-P]

Seems like the last attempt 821455b got a bit further!

Now the installation in pointing to dev.eessi.io. However, this is a pretty dirty hack imo. The issue was that $EESSI_CVMFS_REPO was not being overwritten to dev.eessi.io so I just replaced the check with one that see if $EESSI_DEV_PROJECT is defined (which indirectly means that we're building for dev.eessi.io). Although it is still weird that $EESSI_CVMFS_REPO is not /cvmfs/dev.eessi.io as it should be, which is likely still problematic.

Edit: to clarify, installation starts but the build fails, which is probably happening because I just the first recent commit ID I could find, no guarantees that it would build in the first place.