Build bioconda-utils-build-env images consistently

[mbargull]

bioconda/bioconda-recipes#40550 (comment) :

1. Build images consistently

Looking into this some more, the containers are built so differently that I think it will take a substantial documentation and cleanup effort to unify them. They work fine now, it's just convoluted and hard to follow/troubleshoot. Since this is not required for ARM builds I propose we leave this the way it is for now.

---

We build bioconda/bioconda-utils-build-env-cos7 in the bioconda-utils repository, whereas other images are built here.

1. We should either build the build-env container here too or at least re-use the same base workflow to reduce maintenance burden.

2. bioconda-utils-build-env-cos7 is the x86_64 image an bioconda-utils-build-env-cos7-aarch64 the aarch64 one.
   We should turn bioconda-utils-build-env-cos7 into a multi-arch manifest so we don't have the naming inconsistency.
   (If needed, we can retain arch-specific ones, i.e., add bioconda-utils-build-env-cos7-x86_64.)

[martin-g]

I'll take a look at this !

[martin-g]

As far as I can see the reason to have both bioconda-utils-build-env-cos7-x86_64 and bioconda-utils-build-env-cos7-aarch64 is that they depend on different base images:
https://github.com/bioconda/bioconda-utils/blob/ba9cb8d5173cded978626d981fa586bbf2778faf/Dockerfile#L2-L4

I don't see a way to pass dynamic value for build-arg depending on the arch of the currently built image/manifest.

[martin-g]

Related-to: conda-forge/docker-images#102 (comment)

[martin-g]

Today I learned about docker-manifest command - conda-forge/docker-images#102 (comment)!

With this command it is possible to create a new Docker image/manifest that combines several other (existing) images into one: https://github.com/martin-g/docker-images/pull/3/files#diff-0201eb83bcbafe2c370209f244a029a0789c6d696ef8e9989f291a64252db649R24-R29

[daler]

Great! To make sure I understand, the idea would be to first use manifest to get multi-arch conda-forge images available upstream, which then makes it easier here to use multi-arch?

[mbargull]

We could do that, yes. But we could also just do a matrix build which add the images to the manifest once build (we already do that in

bioconda-containers/.github/workflows/base-glibc-busybox-bash.yaml

Lines 98 to 100 in 8526637

	buildah manifest add \
	"${image_name}:${tag}" \
	"${image_id}"

).

[daler]

That seems more straightforward, and (if I understand correctly) would work independent of what happens upstream with conda-froge.

[martin-g]

But we could also just do a matrix build which add the images to the manifest once build

There is a small problem with this approach: by using a matrix Github Actions will use different VMs for the matrix jobs, thus the built Docker images won't be available locally for the build of the manifest. The images should be pushed to a registry first and then pulled into a new CI VM (a job that needs/depends-on the earlier build job) to build the manifest and push it to the registry too.
This is how I did it for conda-forge - conda-forge/docker-images#249
I am not sure how long it will take to review and merge conda-forge/docker-images#249, so I will prepare a PR for the build-env containers (that might be closed later, but this is fine!).

[martin-g]

Question:
Should I merge https://github.com/bioconda/bioconda-utils/blob/master/.github/workflows/release-please.yml into https://github.com/bioconda/bioconda-utils/blob/master/.github/workflows/build-image.yml ?
And execute the release jobs only for push events to main branch.

[martin-g]

1. We should either build the build-env container here too or at least re-use the same base workflow to reduce maintenance burden.

Moving the Docker build in this repo fails with:

[2/3] STEP 5/5: RUN . /opt/conda/etc/profile.d/conda.sh  && conda activate base &&     pip wheel . &&     mkdir - /opt/bioconda-utils &&     cp ./bioconda_utils-*.whl     ./bioconda_utils/bioconda_utils-requirements.txt     /opt/bioconda-utils/     &&     chgrp -R lucky /opt/bioconda-utils &&     chmod -R g=u /opt/bioconda-utils
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
error building at STEP "RUN . /opt/conda/etc/profile.d/conda.sh  && conda activate base &&     pip wheel . &&     mkdir - /opt/bioconda-utils &&     cp ./bioconda_utils-*.whl     ./bioconda_utils/bioconda_utils-requirements.txt     /opt/bioconda-utils/     &&     chgrp -R lucky /opt/bioconda-utils &&     chmod -R g=u /opt/bioconda-utils": error while running runtime: exit status 1
time="2023-11-30T13:24:02Z" level=error msg="exit status 1"
Error: Error: buildah exited with code 1

i.e. it needs to build the Python project which is still in bioconda-utils repo. IMO the Docker related code should stay at bioconda-utils repo.

If you agree then I will do the following in bioconda-utils repo:

2. bioconda-utils-build-env-cos7 is the x86_64 image an bioconda-utils-build-env-cos7-aarch64 the aarch64 one.
We should turn bioconda-utils-build-env-cos7 into a multi-arch manifest so we don't have the naming inconsistency.
(If needed, we can retain arch-specific ones, i.e., add bioconda-utils-build-env-cos7-x86_64.)

[martin-g]

Actually I was able to build the Docker images and the new manifest in bioconda-containers by using a second actions/checkout to fetch the bioconda-utils sources:

• PR in my fork: https://github.com/martin-g/bioconda-containers/pull/3/files
• Fully green CI build that uploads the images and the manifest to my Quay.io account (https://quay.io/repository/mgrigorov/bioconda-utils-build-env-cos7?tab=tags)