Cross building in podman/docker with qemu fails

[alehed]

Bug Report

Environment

What operating system is being used to run coreos-assembler?

Ubuntu 24.04 running in WSL2.

What operating system is being assembled?

Fedora CoreOS

Is coreos-assembler running in Podman or Docker?

Podman, but the same thing happens with docker.

If Podman, is coreos-assembler running privileged or unprivileged?

Privileged (although the same happens when running without sudo, just that somehow KVM doesn't work then).

Expected Behavior

I can sucessfully build Fedora CoreOS for aarch64.

Actual Behavior

The rpm-ostree step of the build fails with

Running: rpm-ostree compose tree --touch-if-changed /srv/tmp/treecompose.changed --cachedir=/srv/cache --unified-core /srv/tmp/override/coreos-assembler-override-manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --ex-write-lockfile-to /srv/tmp/repo/tmp/manifest-lock.generated.aarch64.json.tmp --ex-lockfile=/srv/src/config/manifest-lock.aarch64.json --ex-lockfile=/srv/src/config/manifest-lock.overrides.yaml --no-parent
bwrap: Creating new namespace failed: Invalid argument
error: bwrap test failed, see <https://github.com/coreos/rpm-ostree/pull/429>: bwrap(true): Child process killed by signal 1
failed to execute cmd-build: exit status 1

Reproduction Steps

1. Prepare for building with

   mkdir tmp && cd tmp
   sudo podman run --privileged --rm docker.io/tonistiigi/binfmt --install all
   

2. Init coreos config

   sudo podman run \
       --rm \
       --privileged \
       --platform=linux/arm64 \
       --security-opt="label=disable" \
       --volume=${PWD}:/srv/ \
       --volume=/var/tmp:/var/tmp \
       --tmpfs=/tmp \
       --device=/dev/kvm \
       --device=/dev/fuse \
       quay.io/coreos-assembler/coreos-assembler:latest \
       init https://github.com/coreos/fedora-coreos-config
   

3. Check that we are building for aarch64

   sudo podman run \
       --rm \
       --privileged \
       --platform=linux/arm64 \
       --security-opt="label=disable" \
       --volume=${PWD}:/srv/ \
       --volume=/var/tmp:/var/tmp \
       --tmpfs=/tmp \
       --device=/dev/kvm \
       --device=/dev/fuse \
       quay.io/coreos-assembler/coreos-assembler:latest \
       basearch

4. Run fetch

   sudo podman run \
       --rm \
       --privileged \
       --platform=linux/arm64 \
       --security-opt="label=disable" \
       --volume=${PWD}:/srv/ \
       --volume=/var/tmp:/var/tmp \
       --tmpfs=/tmp \
       --device=/dev/kvm \
       --device=/dev/fuse \
       quay.io/coreos-assembler/coreos-assembler:latest \
       fetch --strict

5. Run simple build

   sudo podman run \
       --rm \
       --privileged \
       --platform=linux/arm64 \
       --security-opt="label=disable" \
       --volume=${PWD}:/srv/ \
       --volume=/var/tmp:/var/tmp \
       --tmpfs=/tmp \
       --device=/dev/kvm \
       --device=/dev/fuse \
       quay.io/coreos-assembler/coreos-assembler:latest \
       build container

Other Information

I am not sure if this really a bug or just something that is currently unsupported. I am aware of #2248, but the doc link in the answer to that issue doesn't exist any more, so thought I would re-raise what the supported way is to cross-build images using coreos assembler (if there is any).

My final goal is not to build Fedora CoreOS, but some custom Fedora-based OS for aarch64 using CoreOS Assembler (the non-cross building part already works), but if cross building is not supported, I would have to either use a different tool or entirely overthink my OS choice.

[dustymabe]

I am not sure if this really a bug or just something that is currently unsupported. I am aware of #2248, but the doc link in the answer to that issue doesn't exist any more, so thought I would re-raise what the supported way is to cross-build images using coreos assembler (if there is any).

I'm not saying cross building won't work, but I don't know of anyone who does it.

My final goal is not to build Fedora CoreOS, but some custom Fedora-based OS for aarch64 using CoreOS Assembler (the non-cross building part already works), but if cross building is not supported, I would have to either use a different tool or entirely overthink my OS choice.

How custom are you looking to go? If you can derive from Fedora CoreOS and just make tweaks to it then I would just do a container build and then rebase an install of FCOS to it.

There's also bootc if you want something more custom and less like CoreOS.

[alehed]

How custom are you looking to go? If you can derive from Fedora CoreOS and just make tweaks to it then I would just do a container build and then rebase an install of FCOS to it.

My original idea was to take Fedora IOT (better Raspberry PI support and more minimal than CoreOS) and built it using CoreOSAssembler, adding some custom software to it.

Container builds look like an interesting approach that I was not aware of. I think this would also make distribution easier (hosting containers is more widely supported than ostree repos). I'm not sure how this affects diffs between os images though, I assume that the caching would be at the layer level, so updating the base layer is probably a bigger operation.