ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace
string
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate
policy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullName
bool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex
string
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex
string
IncludeTestRegex is used to include tests based on a regular expression.
repeatCount
int
RepeatCount indicates how many times the tests should be executed.
testFile
string
TestFile is the name of the file containing the test to run.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field
Type
Required
Inline
Description
namespace
string
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name
string
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
Operation defines a single operation, only one action is permitted for a given operation.
Field
Type
Required
Inline
Description
description
string
Description contains a description of the operation.
continueOnError
bool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
Timeout for the operation. Overrides the global timeout set in the Configuration.
namespace
string
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name
string
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector
string
Selector defines labels selector.
container
string
Container in pod to get logs from else --all-containers is used.
tail
int
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace
string
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate
policy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullName
bool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex
string
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex
string
IncludeTestRegex is used to include tests based on a regular expression.
repeatCount
int
RepeatCount indicates how many times the tests should be executed.
testFile
string
TestFile is the name of the file containing the test to run.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field
Type
Required
Inline
Description
namespace
string
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name
string
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
Operation defines a single operation, only one action is permitted for a given operation.
Field
Type
Required
Inline
Description
description
string
Description contains a description of the operation.
continueOnError
bool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
Timeout for the operation. Overrides the global timeout set in the Configuration.
namespace
string
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name
string
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector
string
Selector defines labels selector.
container
string
Container in pod to get logs from else --all-containers is used.
tail
int
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
\ No newline at end of file
diff --git a/main/search/search_index.json b/main/search/search_index.json
index b067933b7..16c2610c6 100644
--- a/main/search/search_index.json
+++ b/main/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"community/","title":"Community","text":"
Chainsaw has a growing community and we would definitely love to see you join and contribute.
Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.
Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters
To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.
This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.
If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces
If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.
Input Description releasechainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"install/","title":"Install","text":"
You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.
We also provide a GitHub action to easily install Chainsaw in your workflows.
"},{"location":"install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"install/#homebrew-tap","title":"Homebrew tap","text":"
add tap:
$ brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n
install chainsaw:
$ brew install kyverno/chainsaw/chainsaw\n
Don't forget to specify the tap name
Homebrew core already has a tool named chainsaw.
Be sure that you specify the tap name when installing to install the right tool.
Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.
Chainsaw is a tool developed to run end to end tests in Kubernetes clusters.
It is meant to test Kubernetes operators work as expected by running a sequence of test steps for:
Creating resources
Asserting operators react (or not) the way they should
"},{"location":"intro/#why-we-made-it","title":"Why we made it ?","text":"
While developing Kyverno we need to run end to end tests to make sure our admission controller works as expected.
A typical Kyverno end to end test
Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:
Create a policy
Create a resource
Check that Kyverno acted as expected
Cleanup and move to the next test
"},{"location":"intro/#from-kuttl-to-chainsaw","title":"From KUTTL to Chainsaw","text":"
We started with another tool called KUTTL.
While KUTTL was a great tool to start with, we quickly identified some limitations and forked it to add the features we needed.
In the end we needed more flexibility than what KUTTL could offer and we started designing a new assertion model and at this point it was simpler to start a new tool from scratch than continuing making changes in our KUTTL fork.
The changes we were making was simply too large to have a chance to be incorporated upstream.
A Chainsaw test is made of YAML files in a folder.
YAML files can contain raw manifests with a special file naming convention to identify the step operations. This is useful to create tests quickly but doesn't allow great flexibility.
Another option is to have a chainsaw-test.yaml file containing a Test resource. While more verbose, this offers full flexibility over the test and test steps configuration.
"},{"location":"quick-start/#create-the-test","title":"Create the test","text":"
By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.
# create test file\n$ cat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n - try:\n # first operation: create the config map\n - apply:\n # file is relative to the test folder\n file: configmap.yaml\n # second operation: verify the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n file: configmap.yaml\nEOF\n
Resources, blog posts and videos talking about Chainsaw:
Kyverno Chainsaw - The ultimate end-to-end testing tool!
Kyverno Chainsaw - Exploring the Power of Assertion Trees!
Nirmata Office Hours for Kyverno- Episode 9- Demonstrate Kyverno Chainsaw
Kubebuilder Community Meeting - February 1, 2024
"},{"location":"writing-tests/","title":"What is Chainsaw","text":""},{"location":"writing-tests/#overview","title":"Overview","text":"
Chainsaw provides a declarative approach to testing production-grade Kubernetes operators and controllers.
It provides a way to inject an operator (subject under test) during the setup and allows tests to be standard YAML files.
In Chainsaw everything is YAML. We use YAML to manipulate resources, define cluster state assertions, describe commands to run, etc... Everything is declarative.
ConfigurationSpec contains the configuration used to run tests.
Field Type Required Inline Description timeoutsTimeouts
Global timeouts configuration. Applies to all tests/test steps if not overridden.
skipDeletebool
If set, do not delete the resources after running the tests (implies SkipClusterDelete).
templatebool
Template determines whether resources should be considered for templating.
failFastbool
FailFast determines whether the test should stop upon encountering the first failure.
parallelint
The maximum number of tests to run at once.
reportFormatReportFormatType
ReportFormat determines test report format (JSON reportNamestring
ReportName defines the name of report to create. It defaults to \"chainsaw-report\".
namespacestring
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplatepolicy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullNamebool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegexstring
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegexstring
IncludeTestRegex is used to include tests based on a regular expression.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
testFilestring
TestFile is the name of the file containing the test to run.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description timeoutmeta/v1.Duration
Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheckFileRefOrCheck
FileRefOrAssert provides a reference to the expected error.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectSelectorObjectSelector
ObjectSelector determines the selection process of referenced objects.
apiVersionstring
API version of the referent.
kindstring
Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
Operation defines a single operation, only one action is permitted for a given operation.
Field Type Required Inline Description descriptionstring
Description contains a description of the operation.
continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a creation operation.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
Field Type Required Inline Description timeoutmeta/v1.Duration
Timeout for the operation. Overrides the global timeout set in the Configuration.
namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selectorstring
Selector defines labels selector.
containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Field Type Required Inline Description applymeta/v1.Duration
Apply defines the timeout for the apply operation
assertmeta/v1.Duration
Assert defines the timeout for the assert operation
cleanupmeta/v1.Duration
Cleanup defines the timeout for the cleanup operation
deletemeta/v1.Duration
Delete defines the timeout for the delete operation
errormeta/v1.Duration
Error defines the timeout for the error operation
execmeta/v1.Duration
Exec defines the timeout for exec operations
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/","title":"How to Perform Efficient E2E Testing with Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#introduction","title":"Introduction","text":"
Chainsaw is a powerful tool that enables efficient end-to-end (E2E) testing, making it an essential asset for software developers. Built to be stronger and more cohesive than KUTTL, Chainsaw offers a superior suite of features for Kubernetes (k8s) operators and objects. In this blog post, we will explore how to leverage Chainsaw for efficient E2E testing and enhance your overall testing workflow. Let's dive in!
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#table-of-contents","title":"Table of Contents","text":"
How to Perform Efficient E2E Testing with Chainsaw
To get started with Chainsaw, you'll first need to download the appropriate release for your operating system and architecture. The releases are available on the official GitHub page at the following link: Release
Here's a breakdown of the available releases:
For macOS (Darwin) Users:
Intel-based Macs: chainsaw_darwin_amd64.tar.gz
Apple Silicon Macs: chainsaw_darwin_arm64.tar.gz
For Linux Users:
32-bit: chainsaw_linux_386.tar.gz
64-bit (AMD): chainsaw_linux_amd64.tar.gz
64-bit (ARM): chainsaw_linux_arm64.tar.gz
Each release also comes with associated .pem, .sbom, .sbom.pem, and .sig files. These are used for verification and security purposes. If you're interested in verifying the integrity of your download, you can use these files. However, for the purpose of this guide, we'll focus on the main .tar.gz files.
Chainsaw is a robust tool for end-to-end (e2e) testing in Kubernetes environments. It offers a flexible configuration system that allows users to define their testing parameters and conditions. The configuration can be set up in two primary ways:
Configuration File: This is a structured file where you can define all your testing parameters.
Command-line Flags: These are options you can pass directly when running Chainsaw commands. If both a configuration file and command-line flags are provided, the flags will override the settings in the configuration file.
Chainsaw follows a hierarchy in loading its configurations:
User-specified Configuration: If you provide a configuration file explicitly using a command-line flag.
Default Configuration File: If no configuration is specified, Chainsaw will search for a default file named .chainsaw.yaml in the current directory.
Internal Default Configuration: If neither of the above is available, Chainsaw will resort to a default configuration embedded within its binary.
To specify a custom configuration, use the following command:
In Chainsaw, a test is essentially an ordered sequence of test steps. These steps are executed sequentially, and if any step fails, the entire test is considered failed. Each test step can consist of multiple operations, such as creating, updating, or deleting resources in a Kubernetes cluster, or asserting that certain conditions are met.
Chainsaw supports three primary test definition mechanisms:
Manifests based syntax: This is a straightforward method where you provide Kubernetes resource manifests. Chainsaw uses these manifests to create, update, or assert expectations against a cluster.
TestSteps based syntax: A more verbose method that offers flexibility in defining each test step. It allows for additional configurations and can be combined with the Manifests based syntax.
Test based syntax: The most explicit method that provides a comprehensive overview of the test. It doesn't rely on file naming conventions and offers flexibility at both the test and test step levels.
Creating tests in Chainsaw requires a clear understanding of the test definition mechanisms. Here's a detailed guide for each syntax:
Using Manifests based syntax
File Naming: Files should follow the convention <step index>-<name|assert|error>.yaml. For instance, 00-configmap.yaml for resource creation, 01-assert.yaml for assertions, and 02-error.yaml for expected errors.
You can combine TestStep resources with raw Kubernetes manifests. For instance, a TestStep might apply a resource, while a separate manifest file makes assertions about that resource.
Chainsaw processes the test by executing each step in sequence, ensuring that your Kubernetes environment meets the defined conditions.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-4-running-tests","title":"Section 4: Running Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#41-preparing-for-test-execution","title":"4.1 Preparing for Test Execution","text":"
Before executing your tests, ensure that:
Kubernetes Cluster: You have access to a Kubernetes cluster where the tests will run. This could be a local cluster (like Minikube or kind) or a remote one.
Kubeconfig: Your kubeconfig is correctly set up to point to the desired cluster. Chainsaw will use the current context from your kubeconfig to interact with the cluster.
Test Files: All your test files, whether they are written using Manifests based, TestSteps based, or Test based syntax, are organized and accessible.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#42-executing-tests-in-chainsaw","title":"4.2 Executing Tests in Chainsaw","text":"
Once you're set up, running tests in Chainsaw is straightforward:
Navigate to the Test Directory:
cd path/to/your/test/directory\n
Run the Tests:
chainsaw test\n
This command will execute all tests in the current directory.
View the Results: Chainsaw will display the results in the terminal. It provides a summary of passed, failed, and skipped tests. For any failed tests, Chainsaw will display detailed error messages to help diagnose the issue.
Chainsaw provides a robust and flexible framework for end-to-end testing in Kubernetes environments. By understanding its configuration and test definition mechanisms, developers can create comprehensive test suites that ensure their Kubernetes applications and configurations are functioning as expected.
"},{"location":"blog/first-test/","title":"First working test !","text":"
First test run !
The quick start test is passing
# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n\n# create test folder\nmkdir quick-start\n\n# enter test folder\ncd quick-start\n\n# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: chainsaw-quick-start\ndata:\n foo: bar\nEOF\n\n# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n # first step applies the config map\n - apply:\n # file is relative to the test folder\n - file: configmap.yaml\n # second step verifies the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n - file: configmap.yaml\nEOF\n\n../chainsaw test --test-dir .\n\nRunning without configuration\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN chainsaw\n=== RUN chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT chainsaw/quick-start\n runner.go:35: step-1\n runner.go:35: apply chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n runner.go:35: step-2\n runner.go:35: assert chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n runner.go:68: cleanup namespace: chainsaw-polite-chamois\n--- PASS: chainsaw (0.00s)\n --- PASS: chainsaw/quick-start (5.09s)\nPASS\nDone.\n
The purpose of collectors is to collect certain information about the outcome of a step should it fail (in the case of catch) or at the end of the step (in the case of finally).
The ultimate goal of collectors is to gather information about the failure of a step and therefore help understand what caused it to fail.
A test step can have an arbitrary number of collectors.
-h, --help help for assert\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use (default \"default\")\n --no-color Removes output colors\n --timeout duration The assert timeout to use (default 30s)\n
--catalog string Path to the built test catalog file\n -h, --help help for docs\n --readme-file string Name of the built docs file (default \"README.md\")\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test.yaml\")\n
--description If set, adds description when applicable (default true)\n --force If set, existing test will be deleted if needed\n -h, --help help for test\n --save If set, created test will be saved\n
--autogenTag Determines if the generated docs should contain a timestamp (default true)\n -h, --help help for docs\n -o, --output string Output path (default \".\")\n --website Website version\n
--apply-timeout duration The apply timeout to use as default for configuration (default 5s)\n --assert-timeout duration The assert timeout to use as default for configuration (default 30s)\n --cleanup-delay duration Adds a delay between the time a test ends and the time cleanup starts\n --cleanup-timeout duration The cleanup timeout to use as default for configuration (default 30s)\n --config string Chainsaw configuration file\n --delete-timeout duration The delete timeout to use as default for configuration (default 15s)\n --error-timeout duration The error timeout to use as default for configuration (default 30s)\n --exclude-test-regex string Regular expression to exclude tests\n --exec-timeout duration The exec timeout to use as default for configuration (default 5s)\n --fail-fast Stop the test upon encountering the first failure\n --force-termination-grace-period duration If specified, overrides termination grace periods in applicable resources\n --full-name Use full test case folder path instead of folder name\n -h, --help help for test\n --include-test-regex string Regular expression to include tests\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use for tests\n --no-cluster Runs without cluster\n --no-color Removes output colors\n --parallel int The maximum number of tests to run at once\n --repeat-count int Number of times to repeat each test (default 1)\n --report-format string Test report format (JSON|XML|nil)\n --report-name string The name of the report to create (default \"chainsaw-report\")\n --selector strings Selector (label query) to filter on\n --skip-delete If set, do not delete the resources after running the tests\n --template If set, resources will be considered for templating\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test.yaml\")\n --values strings Values passed to the tests\n
Please pay attention to the configuration options below, they may or may not be relevant in your case but can be useful in certain cases:
Timeouts
Termination graceful period
Delay before cleanup
Label selectors
"},{"location":"configuration/cleanup-delay/","title":"Delay before cleanup","text":"
At the end of each test, Chainsaw will delete the resources it created during the test.
When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update the internal state.
For this reason, Chainsaw provides the delayBeforeCleanup configuration option and the corresponding --delay-before-cleanup flag.
In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.
The command below will run tests using the configuration from my-config.yaml, taking tests from /path/to/tests, and running a maximum of 10 tests simultaneously.
$ chainsaw test --config my-config.yaml --test-dir /path/to/tests --parallel 10\n
Some Kubernetes resources can take time before being stopped. For example, deleting a Pod can take time if the underlying container doesn't quit quickly enough.
For this reason, Chainsaw provides the forceTerminationGracePeriod configuration option and the corresponding --force-termination-grace-period flag. If set, Chainsaw will override the terminationGracePeriodSeconds when working with the following resource kinds:
$ chainsaw test --force-termination-grace-period 5s ...\n
"},{"location":"configuration/no-cluster/","title":"Running without a cluster","text":"
Chainsaw can be run without connection to a Kubernetes cluster. In this case, chainsaw will not try to create an ephemeral namespace and all operations requiring a Kubernetes cluster will fail.
To run chainsaw in this mode pass the --no-cluster flag.
# pass object { \"foo\": \"bar\"\u00a0} as values to the executed tests\n# `--values -` means values are read from standard input\n$ echo \"foo: bar\" | chainsaw test --values -\n
Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.
One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.
Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures that the string '1.28' is found\n # in stdout or else fails\n (contains($stdout, '1.28')): true\n
Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.
- script:\n content: kubectl version\n check:\n # This check ensures that the string '1.25' is NOT found\n # in stdout or else fails\n (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"
In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures the contents of stdout are exactly as shown.\n # Any deviations will cause a failure.\n ($stdout): |-\n Client Version: v1.28.2\n Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"
In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.
$ kubectl foo\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n top\n
Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check bad kubectl command\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This check below ensures that the string 'top' is found in stderr or else fails\n (contains($stderr, 'top')): true\n
Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This checks that the output is exactly as intended.\n ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"more/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"more/kuttl-migration/#overview","title":"Overview","text":"
The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.
chainsaw migrate kuttl config
migrates a KUTTL TestSuite to the corresponding Chainsaw Configuration
chainsaw migrate kuttl tests
migrates KUTTL tests to the corresponding Chainsaw Tests
See below for an example test and the corresponding built docs.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: basic\nspec:\n description: This is a very simple test that creates a configmap and checks the content is as expected.\n steps:\n - description: This steps applies the configmap in the cluster and checks the configmap content.\n try:\n - description: Create the configmap.\n apply:\n file: configmap.yaml\n - description: Check the configmap content.\n assert:\n file: configmap-assert.yaml\n
This step applies the configmap in the cluster and checks the configmap content.
"},{"location":"more/test-docs/#try","title":"Try","text":"# Operation Description 1 apply Create the configmap. 2 assert Check the configmap content."},{"location":"operations/","title":"Operations","text":"
While tests are made of test steps, test steps can be considered made of operations.
Every operation in a test step runs sequentially.
Only one action per operation
Every operation consists of a single action. While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.
The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed in case an operation consists of multiple actions. For this reason, operations consisting of multiple actions are disallowed.
# ...\n- apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
With check
# ...\n- apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
The assert operation allows you to specify conditions that should hold true for a successful test.
For example, after applying resources, you might want to ensure that a particular pod is running or a service is accessible.
Assertion trees
Assertions in Chainsaw are based on assertion trees.
Assertion trees are a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Reference documentation
The full structure of the Assert is documented here.
"},{"location":"operations/assert/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using assert in a Test resource.
Considering an operation success or failure is not always as simple as checking an error code.
Sometimes an operation can fail but the failure is what you expected, hence the operation should be reported as successful.
Sometimes an operation can succeed but the result is not what you expected, in this case the operation should be reported as a failure.
To support those kind of use cases, some operations support an additional check field to evaluate the operation result against an assertion tree.
Assertion trees
Assertions in Chainsaw are based on assertion trees.
Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Checked model
Different operation have a different model passed through the assertion tree.
The object passed to the assertion tree is the output object of the operation. Additional data like error or standard logs are passed using bindings ($error, $stdout, $stderr)
"},{"location":"operations/check/#expect-vs-check","title":"Expect vs Check","text":"
While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply and create operations. Those operations can operate on files containing multiple manifests and every manifest can have a different result.
To support more granular checks we use the expect field that contains an array of Expectation. Every expectation is made of an optional match and a check statement.
This way it is possible to control the scope of a check.
Null match
If the match statement is null, the check statement applies to all manifests in the operation.
If no expectation matches a given manifest, the default expectation will be used, checking that no error occured.
apply supports expect and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#command","title":"Command","text":"
command supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string@ Always null"},{"location":"operations/check/#create","title":"Create","text":"
create supports expect and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#delete","title":"Delete","text":"
delete supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#script","title":"Script","text":"
script supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string@ Always null"},{"location":"operations/command/","title":"Command","text":"
The command operation provides a means to execute a specific command during the test step.
Reference documentation
The full structure of the Command is documented here.
"},{"location":"operations/command/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using command in a Test resource.
The create operation lets you define resources that should be created in the Kubernetes cluster during the test step. These can be configurations, deployments, services, or any other Kubernetes resource.
Warning
If the resource to be created already exists in the cluster, the step will fail.
Reference documentation
The full structure of the Create is documented here.
"},{"location":"operations/create/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using create in a Test resource.
# ...\n- apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
With check
# ...\n- apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.
Assertion trees
Errors in Chainsaw are based on assertion trees.
Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Reference documentation
The full structure of the Error is documented here.
"},{"location":"operations/error/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using error in a Test resource.
Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.
"},{"location":"operations/non-resource-assert/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using assert in a Test resource.
# ...\n- script:\n content: |\n echo \"hello chainsaw\"\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
A catch statement is also a sequence of operations or collectors.
Operations and collectors contained in a catch statement will be executed only if the step failed when executing the operations in the step's try statement.
Tip
All operations and collectors of a catch statement will be executed regardless of the success or failure of each of them.
A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.
Tip
All operations and collectors of a finally statement will be executed regardless of the success or failure of each of them.
A try statement is a sequence of operations executed in the same order they are declared. If an operation fails the entire step is considered failed.
The try statement is at the heart of a test step, it represents what the step is supposed to be about.
catch and finally statements should be viewed as complementary to the try statement.
Continue on error
By default a test step stops executing when an operation fails and the following operations are not executed.
This behaviour can be changed using the continueOnError field, if continueOnError is set to true the step will still be considered failed but execution will continue with the next operations.
Chainsaw supports two different test definition syntaxes. Each syntax has pros and cons, see the descriptions below for more details on advantages and pitfalls.
"},{"location":"tests/#manifests-based-syntax","title":"Manifests based syntax","text":"
This is the simplest and less verbose syntax.
You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
While this syntax is extremly simple, not all operations are supported. For example, delete, command, script and sleep operations are not.
Another strong limitation is that it is not possible to specify additional configuration per test, step or operation.
Finally, this syntax relies heavily on file naming conventions, it can be error prone and makes it hard to reuse files across multiple tests.
"},{"location":"tests/#test-based-syntax","title":"Test based syntax","text":"
The Test based syntax is a more verbose and explicit syntax.
It does not rely on file naming conventions to determine test steps order and allows to easily reuse files accross multiple tests.
This syntax also comes with no limitations to provide additional configuration at the test, step or operation level.
Making a choice
Choosing one syntax over the other is not a trivial choice, every one will have its own preference and/or constraints.
It's usually easier to start with the manifests based syntax. However, as test suites grow and tests become more complex, it is often necessary to configure options on a per test, step or operation basis and the Test based syntax becomes necessary.
Fortunately Chainsaw has a command to automatically migrate from manifest based to Test based syntax.
Unless configured differently, by default Chainsaw will automatically cleanup the resources it created after a test finishes. Cleanup happens in reverse order of creation (created last, cleaned up first).
Note that Chainsaw performs a blocking deletion, that is, it will wait the resource is actually not present anymore in the cluster before proceeding with the next resource cleanup.
This is important, especially when the controller being tested makes use of finalizers.
Overriding cleanup timeout
A global cleanup timeout can be defined at the configuration level or using command line flags.
It can also be overriden on a per test or per test step basis but not at the operation level.
"},{"location":"tests/#parallel-execution-of-tests","title":"Parallel Execution of Tests","text":"
While Chainsaw ensures that the steps within a test are executed sequentially, it is designed to run multiple tests in parallel to each other. This parallel execution helps in significantly reducing the overall time required to run an entire suite of tests, making the testing process more efficient, especially in scenarios with a large number of tests.
Parallel execution of tests
By default, Chainsaw will run tests in parallel.
This can be configured at the configuration level or using command line flags. However, individual tests can be configured to run concurrently by setting Concurrent: true in their TestSpec.
All non-concurrent tests are executed first, followed by the concurrent tests in parallel.
"},{"location":"tests/manifests-based/","title":"Manifests based syntax","text":"
This is the simplest and less verbose supported syntax.
You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
While this syntax is simple, it doesn't support deletion operations and doesn't allow specifying additional configuration per test or step.
It also relies a lot on file naming conventions and makes it hard to reuse files across multiple tests.
Manifest files must follow the naming convention <step index>-<name|assert|errors>.yaml.
As an example 00-configmap.yaml, 01-assert.yaml and 02-errors.yaml can all be considered valid file names.
It's also perfectly valid to have multiple files for the same step. Let's imagine we have the following files 00-resources.yaml, 00-more-resources.yaml, 00-assert.yaml and 00-errors.yaml:
00-resources.yaml and 00-more-resources.yaml contain resources that will be applied in step 00
00-assert.yaml contains assert statements in step 00
00-errors.yaml contains error statements in step 00
With the four files above, Chainsaw will assemble a test step made of the combination of all those files.
The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.
The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.
This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the configmap does not contain the lorem: ipsum data.
Assert and error statements are very similar to standard resource definitions, they share the same structure but can be partial. Chainsaw will process only the elements present in the statements against the actual resource definition.
"},{"location":"tests/test-based/","title":"Test based syntax","text":"
The Test based syntax is more verbose than the manifests based syntax but offers more flexibility and features:
Does not rely on file naming conventions for operations ordering
Allows to easily reuse files accross multiple tests
Offers the flexibility to provide additional configuration at the test, step and operation level
Supports all operations and collectors
"},{"location":"tests/test-based/#the-test-resource","title":"The Test resource","text":"
A Test resource, like any Kubernetes resource has an apiVersion, kind and metadata section.
It also comes with a spec section used to declaratively represent the steps of a test and other configuration elements belonging to the test being defined.
Reference documentation
The full structure of the Test resource is documented here.
The manifest below contains a Test in a file called chainsaw-test.yaml.
Chainsaw will load the Test and steps defined in its spec section.
The Test uses a custom timeouts for the whole test. Note that timeouts could have been overridden in specific steps if needed.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n # these timeouts are applied per operation\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step\n # apply a configmap to the cluster\n # the path to the configmap is relative to the folder\n # containing the test, hence allow reusing manifests\n # across multiple tests\n - try:\n - apply:\n file: ../resources/configmap.yaml\n # second step\n # execute assert statements against existing resources\n # in the cluster\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # third step\n # execute error statements against existing resources\n # in the cluster\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#specifying-a-custom-test-file","title":"Specifying a Custom Test File","text":"
If you have your test defined in a different file, you can specify it when running Chainsaw:
$ chainsaw test --test-dir . --test-file=<custom-test-file-name>.yaml\n
Chainsaw now includes the raw resource feature, allowing direct specification of Kubernetes resources within the test definitions.
This feature offers a more streamlined approach for defining resources, especially useful for simpler test scenarios or for cases where resource definitions need to be reused or slightly modified across different tests.
"},{"location":"tests/test-based/#example-raw-resource","title":"Example Raw Resource","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step applies a configmap directly to the cluster\n - try:\n - apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n # second step executes assert statements against existing resources\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # third step executes error statements against existing resources\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#url-support-for-file-references","title":"URL Support for File References","text":"
Chainsaw has support for URLs in file references for assert, apply, error and similar operations.
This feature enhances the reach of Chainsaw by allowing users to reference files hosted on remote locations, such as GitHub raw URLs or other web URLs, directly within their test definitions.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step\n # apply a Kubernetes manifest hosted at a GitHub raw URL\n - try:\n - apply:\n file: https://raw.githubusercontent.com/username/repo/branch/path/to/configmap.yaml\n # second step\n # execute assert statements against existing resources\n # using a file hosted on another web URL\n - try:\n - assert:\n file: https://example.com/path/to/configmap-assert.yaml\n # third step\n # execute error statements against existing resources\n - try:\n - error:\n file: https://mywebsite.com/path/to/configmap-error.yaml\n
"},{"location":"blog/archive/2023/","title":"2023","text":""},{"location":"blog/category/writing-tests/","title":"Writing Tests","text":""},{"location":"blog/category/announcements/","title":"announcements","text":""}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"community/","title":"Community","text":"
Chainsaw has a growing community and we would definitely love to see you join and contribute.
Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.
Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters
To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.
This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.
If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces
If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.
Input Description releasechainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"install/","title":"Install","text":"
You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.
We also provide a GitHub action to easily install Chainsaw in your workflows.
"},{"location":"install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"install/#homebrew-tap","title":"Homebrew tap","text":"
add tap:
$ brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n
install chainsaw:
$ brew install kyverno/chainsaw/chainsaw\n
Don't forget to specify the tap name
Homebrew core already has a tool named chainsaw.
Be sure that you specify the tap name when installing to install the right tool.
Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.
Chainsaw is a tool developed to run end to end tests in Kubernetes clusters.
It is meant to test Kubernetes operators work as expected by running a sequence of test steps for:
Creating resources
Asserting operators react (or not) the way they should
"},{"location":"intro/#why-we-made-it","title":"Why we made it ?","text":"
While developing Kyverno we need to run end to end tests to make sure our admission controller works as expected.
A typical Kyverno end to end test
Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:
Create a policy
Create a resource
Check that Kyverno acted as expected
Cleanup and move to the next test
"},{"location":"intro/#from-kuttl-to-chainsaw","title":"From KUTTL to Chainsaw","text":"
We started with another tool called KUTTL.
While KUTTL was a great tool to start with, we quickly identified some limitations and forked it to add the features we needed.
In the end we needed more flexibility than what KUTTL could offer and we started designing a new assertion model and at this point it was simpler to start a new tool from scratch than continuing making changes in our KUTTL fork.
The changes we were making was simply too large to have a chance to be incorporated upstream.
A Chainsaw test is made of YAML files in a folder.
YAML files can contain raw manifests with a special file naming convention to identify the step operations. This is useful to create tests quickly but doesn't allow great flexibility.
Another option is to have a chainsaw-test.yaml file containing a Test resource. While more verbose, this offers full flexibility over the test and test steps configuration.
"},{"location":"quick-start/#create-the-test","title":"Create the test","text":"
By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.
# create test file\n$ cat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n - try:\n # first operation: create the config map\n - apply:\n # file is relative to the test folder\n file: configmap.yaml\n # second operation: verify the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n file: configmap.yaml\nEOF\n
Resources, blog posts and videos talking about Chainsaw:
Kyverno Chainsaw - The ultimate end-to-end testing tool!
Kyverno Chainsaw - Exploring the Power of Assertion Trees!
Nirmata Office Hours for Kyverno- Episode 9- Demonstrate Kyverno Chainsaw
Kubebuilder Community Meeting - February 1, 2024
"},{"location":"writing-tests/","title":"What is Chainsaw","text":""},{"location":"writing-tests/#overview","title":"Overview","text":"
Chainsaw provides a declarative approach to testing production-grade Kubernetes operators and controllers.
It provides a way to inject an operator (subject under test) during the setup and allows tests to be standard YAML files.
In Chainsaw everything is YAML. We use YAML to manipulate resources, define cluster state assertions, describe commands to run, etc... Everything is declarative.
ConfigurationSpec contains the configuration used to run tests.
Field Type Required Inline Description timeoutsTimeouts
Global timeouts configuration. Applies to all tests/test steps if not overridden.
skipDeletebool
If set, do not delete the resources after running the tests (implies SkipClusterDelete).
templatebool
Template determines whether resources should be considered for templating.
failFastbool
FailFast determines whether the test should stop upon encountering the first failure.
parallelint
The maximum number of tests to run at once.
reportFormatReportFormatType
ReportFormat determines test report format (JSON reportNamestring
ReportName defines the name of report to create. It defaults to \"chainsaw-report\".
namespacestring
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplatepolicy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullNamebool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegexstring
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegexstring
IncludeTestRegex is used to include tests based on a regular expression.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
testFilestring
TestFile is the name of the file containing the test to run.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description timeoutmeta/v1.Duration
Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheckFileRefOrCheck
FileRefOrAssert provides a reference to the expected error.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectSelectorObjectSelector
ObjectSelector determines the selection process of referenced objects.
apiVersionstring
API version of the referent.
kindstring
Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
Operation defines a single operation, only one action is permitted for a given operation.
Field Type Required Inline Description descriptionstring
Description contains a description of the operation.
continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a creation operation.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
Field Type Required Inline Description timeoutmeta/v1.Duration
Timeout for the operation. Overrides the global timeout set in the Configuration.
namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selectorstring
Selector defines labels selector.
containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Field Type Required Inline Description applymeta/v1.Duration
Apply defines the timeout for the apply operation
assertmeta/v1.Duration
Assert defines the timeout for the assert operation
cleanupmeta/v1.Duration
Cleanup defines the timeout for the cleanup operation
deletemeta/v1.Duration
Delete defines the timeout for the delete operation
errormeta/v1.Duration
Error defines the timeout for the error operation
execmeta/v1.Duration
Exec defines the timeout for exec operations
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/","title":"How to Perform Efficient E2E Testing with Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#introduction","title":"Introduction","text":"
Chainsaw is a powerful tool that enables efficient end-to-end (E2E) testing, making it an essential asset for software developers. Built to be stronger and more cohesive than KUTTL, Chainsaw offers a superior suite of features for Kubernetes (k8s) operators and objects. In this blog post, we will explore how to leverage Chainsaw for efficient E2E testing and enhance your overall testing workflow. Let's dive in!
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#table-of-contents","title":"Table of Contents","text":"
How to Perform Efficient E2E Testing with Chainsaw
To get started with Chainsaw, you'll first need to download the appropriate release for your operating system and architecture. The releases are available on the official GitHub page at the following link: Release
Here's a breakdown of the available releases:
For macOS (Darwin) Users:
Intel-based Macs: chainsaw_darwin_amd64.tar.gz
Apple Silicon Macs: chainsaw_darwin_arm64.tar.gz
For Linux Users:
32-bit: chainsaw_linux_386.tar.gz
64-bit (AMD): chainsaw_linux_amd64.tar.gz
64-bit (ARM): chainsaw_linux_arm64.tar.gz
Each release also comes with associated .pem, .sbom, .sbom.pem, and .sig files. These are used for verification and security purposes. If you're interested in verifying the integrity of your download, you can use these files. However, for the purpose of this guide, we'll focus on the main .tar.gz files.
Chainsaw is a robust tool for end-to-end (e2e) testing in Kubernetes environments. It offers a flexible configuration system that allows users to define their testing parameters and conditions. The configuration can be set up in two primary ways:
Configuration File: This is a structured file where you can define all your testing parameters.
Command-line Flags: These are options you can pass directly when running Chainsaw commands. If both a configuration file and command-line flags are provided, the flags will override the settings in the configuration file.
Chainsaw follows a hierarchy in loading its configurations:
User-specified Configuration: If you provide a configuration file explicitly using a command-line flag.
Default Configuration File: If no configuration is specified, Chainsaw will search for a default file named .chainsaw.yaml in the current directory.
Internal Default Configuration: If neither of the above is available, Chainsaw will resort to a default configuration embedded within its binary.
To specify a custom configuration, use the following command:
In Chainsaw, a test is essentially an ordered sequence of test steps. These steps are executed sequentially, and if any step fails, the entire test is considered failed. Each test step can consist of multiple operations, such as creating, updating, or deleting resources in a Kubernetes cluster, or asserting that certain conditions are met.
Chainsaw supports three primary test definition mechanisms:
Manifests based syntax: This is a straightforward method where you provide Kubernetes resource manifests. Chainsaw uses these manifests to create, update, or assert expectations against a cluster.
TestSteps based syntax: A more verbose method that offers flexibility in defining each test step. It allows for additional configurations and can be combined with the Manifests based syntax.
Test based syntax: The most explicit method that provides a comprehensive overview of the test. It doesn't rely on file naming conventions and offers flexibility at both the test and test step levels.
Creating tests in Chainsaw requires a clear understanding of the test definition mechanisms. Here's a detailed guide for each syntax:
Using Manifests based syntax
File Naming: Files should follow the convention <step index>-<name|assert|error>.yaml. For instance, 00-configmap.yaml for resource creation, 01-assert.yaml for assertions, and 02-error.yaml for expected errors.
You can combine TestStep resources with raw Kubernetes manifests. For instance, a TestStep might apply a resource, while a separate manifest file makes assertions about that resource.
Chainsaw processes the test by executing each step in sequence, ensuring that your Kubernetes environment meets the defined conditions.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-4-running-tests","title":"Section 4: Running Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#41-preparing-for-test-execution","title":"4.1 Preparing for Test Execution","text":"
Before executing your tests, ensure that:
Kubernetes Cluster: You have access to a Kubernetes cluster where the tests will run. This could be a local cluster (like Minikube or kind) or a remote one.
Kubeconfig: Your kubeconfig is correctly set up to point to the desired cluster. Chainsaw will use the current context from your kubeconfig to interact with the cluster.
Test Files: All your test files, whether they are written using Manifests based, TestSteps based, or Test based syntax, are organized and accessible.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#42-executing-tests-in-chainsaw","title":"4.2 Executing Tests in Chainsaw","text":"
Once you're set up, running tests in Chainsaw is straightforward:
Navigate to the Test Directory:
cd path/to/your/test/directory\n
Run the Tests:
chainsaw test\n
This command will execute all tests in the current directory.
View the Results: Chainsaw will display the results in the terminal. It provides a summary of passed, failed, and skipped tests. For any failed tests, Chainsaw will display detailed error messages to help diagnose the issue.
Chainsaw provides a robust and flexible framework for end-to-end testing in Kubernetes environments. By understanding its configuration and test definition mechanisms, developers can create comprehensive test suites that ensure their Kubernetes applications and configurations are functioning as expected.
"},{"location":"blog/first-test/","title":"First working test !","text":"
First test run !
The quick start test is passing
# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n\n# create test folder\nmkdir quick-start\n\n# enter test folder\ncd quick-start\n\n# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: chainsaw-quick-start\ndata:\n foo: bar\nEOF\n\n# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n # first step applies the config map\n - apply:\n # file is relative to the test folder\n - file: configmap.yaml\n # second step verifies the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n - file: configmap.yaml\nEOF\n\n../chainsaw test --test-dir .\n\nRunning without configuration\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN chainsaw\n=== RUN chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT chainsaw/quick-start\n runner.go:35: step-1\n runner.go:35: apply chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n runner.go:35: step-2\n runner.go:35: assert chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n runner.go:68: cleanup namespace: chainsaw-polite-chamois\n--- PASS: chainsaw (0.00s)\n --- PASS: chainsaw/quick-start (5.09s)\nPASS\nDone.\n
The purpose of collectors is to collect certain information about the outcome of a step should it fail (in the case of catch) or at the end of the step (in the case of finally).
The ultimate goal of collectors is to gather information about the failure of a step and therefore help understand what caused it to fail.
A test step can have an arbitrary number of collectors.
-h, --help help for assert\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use (default \"default\")\n --no-color Removes output colors\n --timeout duration The assert timeout to use (default 30s)\n
--catalog string Path to the built test catalog file\n -h, --help help for docs\n --readme-file string Name of the built docs file (default \"README.md\")\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test.yaml\")\n
--description If set, adds description when applicable (default true)\n --force If set, existing test will be deleted if needed\n -h, --help help for test\n --save If set, created test will be saved\n
--autogenTag Determines if the generated docs should contain a timestamp (default true)\n -h, --help help for docs\n -o, --output string Output path (default \".\")\n --website Website version\n
--apply-timeout duration The apply timeout to use as default for configuration (default 5s)\n --assert-timeout duration The assert timeout to use as default for configuration (default 30s)\n --cleanup-delay duration Adds a delay between the time a test ends and the time cleanup starts\n --cleanup-timeout duration The cleanup timeout to use as default for configuration (default 30s)\n --config string Chainsaw configuration file\n --delete-timeout duration The delete timeout to use as default for configuration (default 15s)\n --error-timeout duration The error timeout to use as default for configuration (default 30s)\n --exclude-test-regex string Regular expression to exclude tests\n --exec-timeout duration The exec timeout to use as default for configuration (default 5s)\n --fail-fast Stop the test upon encountering the first failure\n --force-termination-grace-period duration If specified, overrides termination grace periods in applicable resources\n --full-name Use full test case folder path instead of folder name\n -h, --help help for test\n --include-test-regex string Regular expression to include tests\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use for tests\n --no-cluster Runs without cluster\n --no-color Removes output colors\n --parallel int The maximum number of tests to run at once\n --repeat-count int Number of times to repeat each test (default 1)\n --report-format string Test report format (JSON|XML|nil)\n --report-name string The name of the report to create (default \"chainsaw-report\")\n --selector strings Selector (label query) to filter on\n --skip-delete If set, do not delete the resources after running the tests\n --template If set, resources will be considered for templating\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test.yaml\")\n --values strings Values passed to the tests\n
Please pay attention to the configuration options below, they may or may not be relevant in your case but can be useful in certain cases:
Timeouts
Termination graceful period
Delay before cleanup
Label selectors
"},{"location":"configuration/cleanup-delay/","title":"Delay before cleanup","text":"
At the end of each test, Chainsaw will delete the resources it created during the test.
When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update the internal state.
For this reason, Chainsaw provides the delayBeforeCleanup configuration option and the corresponding --delay-before-cleanup flag.
In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.
The command below will run tests using the configuration from my-config.yaml, taking tests from /path/to/tests, and running a maximum of 10 tests simultaneously.
$ chainsaw test --config my-config.yaml --test-dir /path/to/tests --parallel 10\n
Some Kubernetes resources can take time before being stopped. For example, deleting a Pod can take time if the underlying container doesn't quit quickly enough.
For this reason, Chainsaw provides the forceTerminationGracePeriod configuration option and the corresponding --force-termination-grace-period flag. If set, Chainsaw will override the terminationGracePeriodSeconds when working with the following resource kinds:
$ chainsaw test --force-termination-grace-period 5s ...\n
"},{"location":"configuration/no-cluster/","title":"Running without a cluster","text":"
Chainsaw can be run without connection to a Kubernetes cluster. In this case, chainsaw will not try to create an ephemeral namespace and all operations requiring a Kubernetes cluster will fail.
To run chainsaw in this mode pass the --no-cluster flag.
# pass object { \"foo\": \"bar\"\u00a0} as values to the executed tests\n# `--values -` means values are read from standard input\n$ echo \"foo: bar\" | chainsaw test --values -\n
Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.
One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.
Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures that the string '1.28' is found\n # in stdout or else fails\n (contains($stdout, '1.28')): true\n
Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.
- script:\n content: kubectl version\n check:\n # This check ensures that the string '1.25' is NOT found\n # in stdout or else fails\n (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"
In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures the contents of stdout are exactly as shown.\n # Any deviations will cause a failure.\n ($stdout): |-\n Client Version: v1.28.2\n Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"
In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.
$ kubectl foo\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n top\n
Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check bad kubectl command\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This check below ensures that the string 'top' is found in stderr or else fails\n (contains($stderr, 'top')): true\n
Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This checks that the output is exactly as intended.\n ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"more/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"more/kuttl-migration/#overview","title":"Overview","text":"
The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.
chainsaw migrate kuttl config
migrates a KUTTL TestSuite to the corresponding Chainsaw Configuration
chainsaw migrate kuttl tests
migrates KUTTL tests to the corresponding Chainsaw Tests
See below for an example test and the corresponding built docs.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: basic\nspec:\n description: This is a very simple test that creates a configmap and checks the content is as expected.\n steps:\n - description: This steps applies the configmap in the cluster and checks the configmap content.\n try:\n - description: Create the configmap.\n apply:\n file: configmap.yaml\n - description: Check the configmap content.\n assert:\n file: configmap-assert.yaml\n
This step applies the configmap in the cluster and checks the configmap content.
"},{"location":"more/test-docs/#try","title":"Try","text":"# Operation Description 1 apply Create the configmap. 2 assert Check the configmap content."},{"location":"operations/","title":"Operations","text":"
While tests are made of test steps, test steps can be considered made of operations.
Every operation in a test step runs sequentially.
Only one action per operation
Every operation consists of a single action. While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.
The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed in case an operation consists of multiple actions. For this reason, operations consisting of multiple actions are disallowed.
# ...\n- apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
With check
# ...\n- apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
The assert operation allows you to specify conditions that should hold true for a successful test.
For example, after applying resources, you might want to ensure that a particular pod is running or a service is accessible.
Assertion trees
Assertions in Chainsaw are based on assertion trees.
Assertion trees are a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Reference documentation
The full structure of the Assert is documented here.
"},{"location":"operations/assert/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using assert in a Test resource.
Considering an operation success or failure is not always as simple as checking an error code.
Sometimes an operation can fail but the failure is what you expected, hence the operation should be reported as successful.
Sometimes an operation can succeed but the result is not what you expected, in this case the operation should be reported as a failure.
To support those kind of use cases, some operations support an additional check field to evaluate the operation result against an assertion tree.
Assertion trees
Assertions in Chainsaw are based on assertion trees.
Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Checked model
Different operation have a different model passed through the assertion tree.
The object passed to the assertion tree is the output object of the operation. Additional data like error or standard logs are passed using bindings ($error, $stdout, $stderr)
"},{"location":"operations/check/#expect-vs-check","title":"Expect vs Check","text":"
While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply and create operations. Those operations can operate on files containing multiple manifests and every manifest can have a different result.
To support more granular checks we use the expect field that contains an array of Expectation. Every expectation is made of an optional match and a check statement.
This way it is possible to control the scope of a check.
Null match
If the match statement is null, the check statement applies to all manifests in the operation.
If no expectation matches a given manifest, the default expectation will be used, checking that no error occured.
apply supports expect and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#command","title":"Command","text":"
command supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string@ Always null"},{"location":"operations/check/#create","title":"Create","text":"
create supports expect and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#delete","title":"Delete","text":"
delete supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string@ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#script","title":"Script","text":"
script supports check and has the following elements to be checked:
Name Purpose Type $values Values provided when invoking chainsaw with --values flag object$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string@ Always null"},{"location":"operations/command/","title":"Command","text":"
The command operation provides a means to execute a specific command during the test step.
Reference documentation
The full structure of the Command is documented here.
"},{"location":"operations/command/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using command in a Test resource.
The create operation lets you define resources that should be created in the Kubernetes cluster during the test step. These can be configurations, deployments, services, or any other Kubernetes resource.
Warning
If the resource to be created already exists in the cluster, the step will fail.
Reference documentation
The full structure of the Create is documented here.
"},{"location":"operations/create/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using create in a Test resource.
# ...\n- apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
With check
# ...\n- apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.
Assertion trees
Errors in Chainsaw are based on assertion trees.
Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.
Reference documentation
The full structure of the Error is documented here.
"},{"location":"operations/error/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using error in a Test resource.
Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.
"},{"location":"operations/non-resource-assert/#usage-in-test","title":"Usage in Test","text":"
Below is an example of using assert in a Test resource.
# ...\n- script:\n content: |\n echo \"hello chainsaw\"\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n# ...\n
A catch statement is also a sequence of operations or collectors.
Operations and collectors contained in a catch statement will be executed only if the step failed when executing the operations in the step's try statement.
Tip
All operations and collectors of a catch statement will be executed regardless of the success or failure of each of them.
A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.
Tip
All operations and collectors of a finally statement will be executed regardless of the success or failure of each of them.
A try statement is a sequence of operations executed in the same order they are declared. If an operation fails the entire step is considered failed.
The try statement is at the heart of a test step, it represents what the step is supposed to be about.
catch and finally statements should be viewed as complementary to the try statement.
Continue on error
By default a test step stops executing when an operation fails and the following operations are not executed.
This behaviour can be changed using the continueOnError field, if continueOnError is set to true the step will still be considered failed but execution will continue with the next operations.
Chainsaw supports two different test definition syntaxes. Each syntax has pros and cons, see the descriptions below for more details on advantages and pitfalls.
"},{"location":"tests/#manifests-based-syntax","title":"Manifests based syntax","text":"
This is the simplest and less verbose syntax.
You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
While this syntax is extremly simple, not all operations are supported. For example, delete, command, script and sleep operations are not.
Another strong limitation is that it is not possible to specify additional configuration per test, step or operation.
Finally, this syntax relies heavily on file naming conventions, it can be error prone and makes it hard to reuse files across multiple tests.
"},{"location":"tests/#test-based-syntax","title":"Test based syntax","text":"
The Test based syntax is a more verbose and explicit syntax.
It does not rely on file naming conventions to determine test steps order and allows to easily reuse files accross multiple tests.
This syntax also comes with no limitations to provide additional configuration at the test, step or operation level.
Making a choice
Choosing one syntax over the other is not a trivial choice, every one will have its own preference and/or constraints.
It's usually easier to start with the manifests based syntax. However, as test suites grow and tests become more complex, it is often necessary to configure options on a per test, step or operation basis and the Test based syntax becomes necessary.
Fortunately Chainsaw has a command to automatically migrate from manifest based to Test based syntax.
Unless configured differently, by default Chainsaw will automatically cleanup the resources it created after a test finishes. Cleanup happens in reverse order of creation (created last, cleaned up first).
Note that Chainsaw performs a blocking deletion, that is, it will wait the resource is actually not present anymore in the cluster before proceeding with the next resource cleanup.
This is important, especially when the controller being tested makes use of finalizers.
Overriding cleanup timeout
A global cleanup timeout can be defined at the configuration level or using command line flags.
It can also be overriden on a per test or per test step basis but not at the operation level.
"},{"location":"tests/#parallel-execution-of-tests","title":"Parallel Execution of Tests","text":"
While Chainsaw ensures that the steps within a test are executed sequentially, it is designed to run multiple tests in parallel to each other. This parallel execution helps in significantly reducing the overall time required to run an entire suite of tests, making the testing process more efficient, especially in scenarios with a large number of tests.
Parallel execution of tests
By default, Chainsaw will run tests in parallel.
This can be configured at the configuration level or using command line flags. However, individual tests can be configured to run concurrently by setting Concurrent: true in their TestSpec.
All non-concurrent tests are executed first, followed by the concurrent tests in parallel.
"},{"location":"tests/manifests-based/","title":"Manifests based syntax","text":"
This is the simplest and less verbose supported syntax.
You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.
While this syntax is simple, it doesn't support deletion operations and doesn't allow specifying additional configuration per test or step.
It also relies a lot on file naming conventions and makes it hard to reuse files across multiple tests.
Manifest files must follow the naming convention <step index>-<name|assert|errors>.yaml.
As an example 00-configmap.yaml, 01-assert.yaml and 02-errors.yaml can all be considered valid file names.
It's also perfectly valid to have multiple files for the same step. Let's imagine we have the following files 00-resources.yaml, 00-more-resources.yaml, 00-assert.yaml and 00-errors.yaml:
00-resources.yaml and 00-more-resources.yaml contain resources that will be applied in step 00
00-assert.yaml contains assert statements in step 00
00-errors.yaml contains error statements in step 00
With the four files above, Chainsaw will assemble a test step made of the combination of all those files.
The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.
The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.
This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the configmap does not contain the lorem: ipsum data.
Assert and error statements are very similar to standard resource definitions, they share the same structure but can be partial. Chainsaw will process only the elements present in the statements against the actual resource definition.
"},{"location":"tests/test-based/","title":"Test based syntax","text":"
The Test based syntax is more verbose than the manifests based syntax but offers more flexibility and features:
Does not rely on file naming conventions for operations ordering
Allows to easily reuse files accross multiple tests
Offers the flexibility to provide additional configuration at the test, step and operation level
Supports all operations and collectors
"},{"location":"tests/test-based/#the-test-resource","title":"The Test resource","text":"
A Test resource, like any Kubernetes resource has an apiVersion, kind and metadata section.
It also comes with a spec section used to declaratively represent the steps of a test and other configuration elements belonging to the test being defined.
Reference documentation
The full structure of the Test resource is documented here.
The manifest below contains a Test in a file called chainsaw-test.yaml.
Chainsaw will load the Test and steps defined in its spec section.
The Test uses a custom timeouts for the whole test. Note that timeouts could have been overridden in specific steps if needed.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n # these timeouts are applied per operation\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step\n # apply a configmap to the cluster\n # the path to the configmap is relative to the folder\n # containing the test, hence allow reusing manifests\n # across multiple tests\n - try:\n - apply:\n file: ../resources/configmap.yaml\n # second step\n # execute assert statements against existing resources\n # in the cluster\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # third step\n # execute error statements against existing resources\n # in the cluster\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#specifying-a-custom-test-file","title":"Specifying a Custom Test File","text":"
If you have your test defined in a different file, you can specify it when running Chainsaw:
$ chainsaw test --test-dir . --test-file=<custom-test-file-name>.yaml\n
Chainsaw now includes the raw resource feature, allowing direct specification of Kubernetes resources within the test definitions.
This feature offers a more streamlined approach for defining resources, especially useful for simpler test scenarios or for cases where resource definitions need to be reused or slightly modified across different tests.
"},{"location":"tests/test-based/#example-raw-resource","title":"Example Raw Resource","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step applies a configmap directly to the cluster\n - try:\n - apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: chainsaw-quick-start\n data:\n foo: bar\n # second step executes assert statements against existing resources\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # third step executes error statements against existing resources\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#url-support-for-file-references","title":"URL Support for File References","text":"
Chainsaw has support for URLs in file references for assert, apply, error and similar operations.
This feature enhances the reach of Chainsaw by allowing users to reference files hosted on remote locations, such as GitHub raw URLs or other web URLs, directly within their test definitions.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test-name\nspec:\n skip: false\n concurrent: false\n skipDelete: false\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # first step\n # apply a Kubernetes manifest hosted at a GitHub raw URL\n - try:\n - apply:\n file: https://raw.githubusercontent.com/username/repo/branch/path/to/configmap.yaml\n # second step\n # execute assert statements against existing resources\n # using a file hosted on another web URL\n - try:\n - assert:\n file: https://example.com/path/to/configmap-assert.yaml\n # third step\n # execute error statements against existing resources\n - try:\n - error:\n file: https://mywebsite.com/path/to/configmap-error.yaml\n
"},{"location":"blog/archive/2023/","title":"2023","text":""},{"location":"blog/category/writing-tests/","title":"Writing Tests","text":""},{"location":"blog/category/announcements/","title":"announcements","text":""}]}
\ No newline at end of file
diff --git a/main/sitemap.xml.gz b/main/sitemap.xml.gz
index 92569dfaf..81b196243 100644
Binary files a/main/sitemap.xml.gz and b/main/sitemap.xml.gz differ
