[rawhide] broken bootc storage symlink causes ext.config.files.validate-symlinks to fail

[marmijo]

In rawhide, tracking Fedora 42, with bootc-0.1.15-1.fc42 installed, there is a broken symbolic link in the /usr/lib/bootc directory causing the validate-symlinks kola test to fail.

The symlink /usr/lib/bootc/storage -> /sysroot/ostree/bootc/storage is created, but it seems the /sysroot/ostree/bootc directory doesn't exist at all.

[core@cosa-devsh ~]$ ll /usr/lib/bootc
total 0
drwxr-xr-x. 2 root root  6 Jan  1  1970 bound-images.d
drwxr-xr-x. 2 root root  6 Jan  1  1970 install
drwxr-xr-x. 2 root root  6 Jan  1  1970 kargs.d
lrwxrwxrwx. 2 root root 29 Aug  1  2022 storage -> /sysroot/ostree/bootc/storage (broken)

[core@cosa-devsh ~]$ ll /sysroot/ostree
total 0
lrwxrwxrwx. 1 root root   8 Aug  1  2022 boot.1 -> boot.1.1
drwxr-xr-x. 3 root root  27 Aug  1  2022 boot.1.1
drwxr-xr-x. 3 root root  27 Aug  1  2022 deploy
drwxr-xr-x. 7 root root 102 Aug  1  2022 repo

ext.config.files.validate-symlinks failure:

Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\m\t\a\b* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\m\t\a\b ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\s\s\l\/* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\s\s\l\/ ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\s\w\i\d\/\s\w\i\d\t\a\g\s\.\d\/\f\e\d\o\r\a\p\r\o\j\e\c\t\.\o\r\g* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\s\w\i\d\/\s\w\i\d\t\a\g\s\.\d\/\f\e\d\o\r\a\p\r\o\j\e\c\t\.\o\r\g ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\x\d\g\/\s\y\s\t\e\m\d\/\u\s\e\r* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\e\t\c\/\x\d\g\/\s\y\s\t\e\m\d\/\u\s\e\r ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\.\b\u\i\l\d\-\i\d\/* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\.\b\u\i\l\d\-\i\d\/ ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\f\i\r\m\w\a\r\e* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\f\i\r\m\w\a\r\e ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\m\o\d\u\l\e\s\/* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\l\i\b\/\m\o\d\u\l\e\s\/ ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\e\t\c\-\p\k\i\-\e\n\t\i\t\l\e\m\e\n\t* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\e\t\c\-\p\k\i\-\e\n\t\i\t\l\e\m\e\n\t ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\e\d\h\a\t\.\r\e\p\o* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\e\d\h\a\t\.\r\e\p\o ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + for search_element in "${list_broken_symlinks_skip[@]}"
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\h\s\m* ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ /usr/lib/bootc/storage == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\h\s\m ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + [[ false == \f\a\l\s\e ]]
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + fatal 'Error: /usr/lib/bootc/storage symlink to /sysroot/ostree/bootc/storage which does not exist'
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + echo 'Error: /usr/lib/bootc/storage symlink to /sysroot/ostree/bootc/storage which does not exist'
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: Error: /usr/lib/bootc/storage symlink to /sysroot/ostree/bootc/storage which does not exist
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2228]: + exit 1
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2227]: find: ‘standard output’: Broken pipe
Aug 19 20:54:43 qemu0 kola-runext-validate-symlinks[2227]: find: write error
Aug 19 20:54:43 qemu0 systemd[1]: kola-runext-37.service: Main process exited, code=exited, status=1/FAILURE
Aug 19 20:54:43 qemu0 systemd[1]: kola-runext-37.service: Failed with result 'exit-code'.

Additional Information

This isn't isolated to Fedora CoreOS. I saw this using quay.io/fedora/fedora:rawhide as well.

[root@323b8e74acd5 /]# ll /usr/lib/bootc
total 4
drwxr-xr-x. 1 root root  0 Aug 16 00:00 bound-images.d
drwxr-xr-x. 1 root root  0 Aug 16 00:00 install
drwxr-xr-x. 1 root root  0 Aug 16 00:00 kargs.d
lrwxrwxrwx. 1 root root 29 Aug 16 00:00 storage -> /sysroot/ostree/bootc/storage (broken)

[cgwalters]

This is part of containers/bootc#20 and came from containers/bootc@0527ca9

The symlink isn't actually broken on a system deployed via bootc.

I dunno...I think I'd just add this to an allowlist.

[marmijo]

Ah, sounds good. I can add it to the allowlist for the test here: https://github.com/coreos/fedora-coreos-config/blob/fb0357bbd8b39de5eb1a3b28dd50cda0dacf78f6/tests/kola/files/validate-symlinks#L19-L30.

Thanks for the info @cgwalters

[marmijo]

Opened coreos/fedora-coreos-config#3104