Rethink matrix-based hardening testing

[comps]

We have a lot of combinations to test, consider these axes:

• profile (15-ish)
• hardening technologies, oscap/ansible/imagebuilder/anaconda (4)
• without / with-gui (2)
• bios / uefi (2) - Test on UEFI/non-UEFI #19
• hardening using shipped / generated content (2)
• hardening on VMs / on bare metal (2)
• architecture, for host-os (4)
• probably more

That creates a huge amount of combinations / tests to run.

Consider not just specifying per-profile tests for each test type, but parametrize everything externally, ie. via a TMT plan. Because some axes (uefi) will be just an env variable for library code, with no test change needed. It would be therefore nice to treat profiles the same way.

Ie.

PROFILE=ospp BOOT=uefi PACKAGES=with-gui

These matrix combinations could be handled externally (by runcontest2) and/or via TMT plans. We could even have TMT plan good-enough defaults with some minimal test sets, and do runcontest2 to run everything.

By "run everything", I mean to run smart combinations of variables to give us beneficial results (Orthogonal Arrays), not a*b*c*d... "everything".
Ie. from

* VM + bios + no-gui
* bare + bios + with-gui
* bare + uefi + no-gui
* VM + uefi + with-gui

we effectively test everything with everything else, with 4 instead of 9 combinations.

[comps]

Hardening tests (files) could be all inside /hardening as python modules (like they're now), and a central test.py would execute them based on env variables. Ie.

hardening/
├── anaconda.py
├── ansible.py
├── image-builder.py
├── main.fmf
├── oscap.py
└── test.py

TMT/FMF would see just one "test", that being: /hardening, or perhaps /hardening/$profilename.

Results would be reported directly under that, ie.

/hardening/pci-dss/some_rule_name
/hardening/ospp/dracut_fips_enabled

Not sure whether all matrix combinations should be encoded in the name, the note, or additional TMT metadata fields, but it needs to be somehow possible and easy to distinguish ie. bare metal oscap hardening test results from ansible VM hardening results.

Waivers could fortunately use extra python variables, opening up possibilities like

# applies to all hardening methods
/hardening/pci-dss/some_rule_name
    True

# only Ansible, no matter if host-os, VM or elsewhere
/hardening/ospp/setup_fips_enabled
    remediation == 'ansible'

# fails on UEFI on bare metal, but not in VMs
/hardening/ospp/some_grub2_thing
    machine == 'bare' and boot == 'uefi' and arch == 'x86_64'   # not on ppc64le UEFI

Test execution itself would be done via documented env variables,

$ tmt run -e CONTEST_MACHINE=vm -e CONTEST_BOOT=bios ...

and the defaults, or the "just run all tests" version would be done via TMT plans, which would pre-configure a test set according to some matrix combinations (see first post here), so productization and similar testing could just

$ tmt run ... plans -n /plans/run-all

without enumerating the env variables. We could use it manually like that too, with -e overrides when necessary.

---

So I guess the biggest hurdle is representing results in some meaningful way in HTML reports and on the console, because

/hardening/profile-name/rule-name

is just not descriptive enough, and if we were to extend it to include ie.

/hardening/ospp/setup_fips_enabled/oscap,x86_64,bare,uefi,with-gui,generated_fix

then getting rid of the extra info when writing waiver rules would be too arbitrary (apply only to hardening tests) and if we don't remove it, we'd have to write .* at the end of all rules. And HTML reports and console would be really noisy.

[comps]

I guess one way of reducing

/hardening/ospp/setup_fips_enabled/oscap,x86_64,bare,uefi,with-gui,generated_fix

would be to consider all possible combinations of each dimension for the current run, and if one dimension never changes, just don't include it in the result name.

For example, if all tests in a single run (tmt run) would only run on VMs, it makes no sense to print out vm / bare. The architecture would also be always x86_64, so no point in printing it.

This way, the result test names would only contain dimensions relevant to that run (dimensions which have at least 2 possible values).

---

Waiver rules and their python code would use some concept of "dimensions", python objects that support attribute access - the point is to not use strings (which may have typos), so ie.

# fails on UEFI on bare metal, but not in VMs
/hardening/ospp/some_grub2_thing
    status.error and machine.bare and boot.uefi and arch.x86_64   # not on ppc64le UEFI

would trigger an error if a user wrote machine.bear.

I imagine these could also support key (machine['bare']) and call (machine('bare')) access, if needed.