-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(validation)!: validation testing framework #667
Open
meganwolf0
wants to merge
18
commits into
main
Choose a base branch
from
460-validation-review-fields-process
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
b1e464f
feat: inject internal package
meganwolf0 f10a61d
feat: updated inject functionality for complex filters
meganwolf0 1b2d79f
feat: inject testing
meganwolf0 53c1775
feat(validation): add validation testing skeleton
meganwolf0 4b41d2b
Merge remote-tracking branch 'origin/main' into 460-validation-review…
meganwolf0 33f8d14
feat(validation-tests): refactor of inject->transform, tests, docs
meganwolf0 80393e4
Merge remote-tracking branch 'origin/main' into 460-validation-review…
meganwolf0 1e5cb90
fix: updated deps, fixes per lint checks
meganwolf0 ea310a9
feat(validation-tests): additional testing
meganwolf0 dc8e464
feat(dev): updated dev validate
meganwolf0 4ab6611
docs: updated test docs, small cli print mods
meganwolf0 b2dba9c
fix: handle path indicies, docs updates
meganwolf0 c3bd87f
fix: rm unused test files
meganwolf0 de00764
Update docs/reference/testing.md
meganwolf0 d8ea783
feat(dev): updated tests printing
meganwolf0 233fd60
Merge branch 'main' into 460-validation-review-fields-process
meganwolf0 e82a09f
fix: run tests test
meganwolf0 c07801e
Merge remote-tracking branch 'refs/remotes/origin/460-validation-revi…
meganwolf0 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| # Testing | ||
|
|
||
| Testing is a key part of Lula Validation development. Since the results of the Lula Validations are determined by the policy set by the `provider`, those policies must be tested to ensure they are working as expected. | ||
|
|
||
| ## Validation Testing | ||
|
|
||
| In the Lula Validation, a `tests` property is used to specify each test that should be performed against the validation. Each test is a map of the following properties: | ||
|
|
||
| - `name`: The name of the test | ||
| - `changes`: An array of changes or transformations to be applied to the resources used in the test validation | ||
| - `expected-result`: The expected result of the test - satisfied or not-satisfied | ||
|
|
||
| A change is a map of the following properties: | ||
|
|
||
| - `path`: The path to the resource to be modified. The path syntax is described below. | ||
| - `type`: The type of operation to be performed on the resource | ||
| - `update`: (default) updates the resource with the specified value | ||
| - `delete`: deletes the field specified | ||
| - `add`: adds the specified value | ||
| - `value`: The value to be used for the operation (string) | ||
| - `value-map`: The value to be used for the operation (map[string]interface{}) | ||
|
|
||
| An example of a test added to a validation is: | ||
|
|
||
| ```yaml | ||
| domain: | ||
| type: kubernetes | ||
| kubernetes-spec: | ||
| resources: | ||
| - name: podsvt | ||
| resource-rule: | ||
| version: v1 | ||
| resource: pods | ||
| namespaces: [validation-test] | ||
| provider: | ||
| type: opa | ||
| opa-spec: | ||
| rego: | | ||
| package validate | ||
|
|
||
| import future.keywords.every | ||
|
|
||
| validate { | ||
| every pod in input.podsvt { | ||
| podLabel := pod.metadata.labels.foo | ||
| podLabel == "bar" | ||
| } | ||
| } | ||
| tests: | ||
| - name: modify-pod-label-not-satisfied | ||
| expected-result: not-satisfied | ||
| changes: | ||
| - path: podsvt.[metadata.namespace=validation-test].metadata.labels.foo | ||
| type: update | ||
| value: baz | ||
| - name: delete-pod-label-not-satisfied | ||
| expected-result: not-satisfied | ||
| changes: | ||
| - path: podsvt.[metadata.namespace=validation-test].metadata.labels.foo | ||
| type: delete | ||
| ``` | ||
|
|
||
| There are two tests here: | ||
| * The first test will locate the first pod in the `validation-test` namespace and update the label `foo` to `baz`. Then a `validate` will be executed against the modified resources. The expected result of this is that the validation will fail, i.e., will be `not-satisfied`, which would result in a successful test. | ||
| * The second test will locate the first pod in the `validation-test` namespace and delete the label `foo`, then proceed to validate the modified resources and compare to the expected result. | ||
|
|
||
| ### Path Syntax | ||
|
|
||
| This feature uses the kyaml library to inject data into the resources, so the path syntax is based on this library. | ||
|
|
||
| The path should be a "." delimited string that specifies the keys along the path to the resource seeking to be modified. In addition to keys, a list item can be specified by using the “[some-key=value]” syntax. For example, the following path: | ||
|
|
||
| ``` | ||
| pods.[metadata.namespace=grafana].spec.containers.[name=istio-proxy] | ||
| ``` | ||
|
|
||
| Will start at the pods key, then since the next item is a [*=*] it assumes pods is a list, and will iterate over each item in the list to find where the key `metadata.namespace` is equal to `grafana`. It will then find the `containers` list item in `spec`, and iterate over each item in the list to find where the key `name` is equal to `istio-proxy`. | ||
|
|
||
| Multiple filters can be added for a list, for example the above example could be modified to filter both by namespace and pod name: | ||
|
|
||
| ``` | ||
| pods.[metadata.namespace=grafana,metadata.name=operator].spec.containers.[name=istio-proxy] | ||
| ``` | ||
|
|
||
| To support map keys containing ".", [] syntax will also be used, e.g., | ||
|
|
||
| ``` | ||
| namespaces.[metadata.namespace=grafana].metadata.labels.["some.key/label"] | ||
| ``` | ||
|
|
||
| Additionally, individual list items can be found via their index, e.g., | ||
|
|
||
| ``` | ||
| namespaces.[0].metadata.labels | ||
| ``` | ||
|
|
||
| Which will point to the labels key of the first namespace. Additionally, a `[-]` can be used to specify the last item in the list. | ||
|
|
||
| >[!IMPORTANT] | ||
| > The path will return only one item, the first item that matches the filters along the path. If no items match the filters, the path will return an empty map. | ||
|
|
||
| ### Change Type Behavior | ||
|
|
||
| **Add** | ||
| * All keys in the path must exist, except for the last key. If you are trying to add a map, then use `value-map` and specify the existing root key. | ||
| * If a sequence is "added" to, then the value items will be appended to the sequence. | ||
|
|
||
| **Update** | ||
| * If a sequence is "updated", then the entire sequence will be replaced. | ||
|
|
||
| **Delete** | ||
| * Currently only supports deleting a key, error will be returned if the last item in the path resolves to a sequence. | ||
| * No values should be specified for delete. | ||
|
|
||
| A note about replacing a key with an empty map - due to the way the `kyaml` library works, simply trying to overwrite an existing key with an empty map will not yield a removal of all the existing data of the map, it will just try and merge the differences, which is possibly not the desired outcome. To replace a map with an empty map, you must combine `delete` a change type and `add` a change type, e.g., | ||
|
|
||
| ```yaml | ||
| changes: | ||
| - path: pods.[metadata.namespace=grafana].metadata.labels | ||
| type: delete | ||
| - path: pods.[metadata.namespace=grafana].metadata | ||
| type: add | ||
| value-map: | ||
| labels: {} | ||
| ``` | ||
|
|
||
| Which will delete the existing labels map and then add an empty map, such that the "labels" key will still exist but will be an empty map. | ||
|
|
||
| ## Executing Tests | ||
|
|
||
| Tests can be executed by specifying the `--run-tests` flag when running `lula dev validate`. E.g., | ||
|
|
||
| ```sh | ||
| lula dev validate -f ./validation.yaml --run-tests | ||
| ``` | ||
|
|
||
| This will execute the tests and print the test results to the console. | ||
|
|
||
| To aid in debugging, the `--print-test-resources` flag can be used to print the resources used for each test to the validation directory, the filenames will be `<test-name>.json`.. E.g., | ||
|
|
||
| ```sh | ||
| lula dev validate -f ./validation.yaml --run-tests --print-test-resources | ||
| ``` | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an interesting point as we look to clear a path for validations used outside of OSCAL. We map the result of a policy (pass/fail) to align to findings/observations using the OSCAL terminology (satisfied/not-satisfied).
We might look at what terminology OPA/Kyverno use and/or decide on what Lulas will be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's fair - I initially was thinking like pass/fail - that also sort of conflates with the overall test result (which is pass/fail) so walked back the terminology. I mean it could be more like policy/provider-result -> true/false? Or accept/reject?