fix(validate): support returning empty resources object/slice

[brandtkeller]

BLUF

Domains are responsible for the collection of data that will be processed by a provider. This PR makes a stance that domains should:

• Always perform the full collection of data possible(do not exit on error before completion)
• Return an empty object when an error occurs

This sets a standard where Lula always attempts to retrieve the full extent of evidence possible. When an error occurs - reviewers/auditors should want to have a full view of what worked and what did not with regards to domain resource collection.

Description

Looking to setup the changes required to test impact to our current validations on allowing domains to return a map item that contains an empty payload. Most applicable to our k8s domain but still relevant elsewhere.

Update: We will need to rewrite some of our existing policies to account for an edge case where no resources were collected. This is more of an edge case in my opinion as the stability of the current validations has been good.

Context primarily captured in the issue below - but the primary goal is that providers should be gating whether the data allows for policy adherence and not a domain. A domain should be objective in that - if there was no errors during the collection of the data - it should continue to progress.

Removed a return statement in QueryCluster and instead aggregate the errors into a single string - Given the ability to continue writing resources if one of the resource items fails - this would allow for the validation to still collect evidence for later review - I believe this is a fundamental part of how Lula should operate - always being in a position to provide as much evidence as possible to enable less guessing in the event a validation or environment needs correction.

Related Issue

Fixes #589

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Checklist before merging

• Test, docs, adr added or updated as needed
• Schema Updates applied
• Contributor Guide Steps followed

[mildwonkey]

I left some comments, but none of them have to block merging this as-is from my POV.

[mildwonkey]

I have some feedback but I don't think it's anything that has to block merging this as-is. I'm a bit concerned that changing this behavior didn't necessitate test changes; we should consider adding some tests that validate the partial response on error.

[meganwolf0]

I think this looks good, possibly agree with Kristin's comment above. Also wondering if you did any current validation tests and measured the outcomes against this branch v main (just to make sure partial resource return isn't flipping satisfaction on any existing validations, particularly those in UDS Core)?

I feel like this is a good test case for figuring out how to really instrument a proper testbed for that sort of thing - obviously we cant test every validation, but might be nice to test a large swath - or even have an action that we support to test old v new Lula version that we could provide people to run in their own environments to ensure that their validations are still responding as expected with Lula version upgrades...

[brandtkeller]

I think this looks good, possibly agree with Kristin's comment above. Also wondering if you did any current validation tests and measured the outcomes against this branch v main (just to make sure partial resource return isn't flipping satisfaction on any existing validations, particularly those in UDS Core)?

Tested against uds-core and did not reduce counts of satisfied controls.

I feel like this is a good test case for figuring out how to really instrument a proper testbed for that sort of thing - obviously we cant test every validation, but might be nice to test a large swath - or even have an action that we support to test old v new Lula version that we could provide people to run in their own environments to ensure that their validations are still responding as expected with Lula version upgrades...

I like that idea!