From b5b7326fe2b6656776fe4c5fdb36d6e7979af564 Mon Sep 17 00:00:00 2001 From: ci-bot Date: Mon, 19 Feb 2024 09:25:04 +0000 Subject: [PATCH] Deployed a1b515a to main with MkDocs 1.5.3 and mike 2.0.0 --- main/apis/chainsaw.v1alpha1/index.html | 2 +- main/commands/chainsaw_test/index.html | 1 + main/search/search_index.json | 2 +- main/sitemap.xml.gz | Bin 849 -> 849 bytes 4 files changed, 3 insertions(+), 2 deletions(-) diff --git a/main/apis/chainsaw.v1alpha1/index.html b/main/apis/chainsaw.v1alpha1/index.html index 039642251..b67705144 100644 --- a/main/apis/chainsaw.v1alpha1/index.html +++ b/main/apis/chainsaw.v1alpha1/index.html @@ -1 +1 @@ - chainsaw (v1alpha1) - Chainsaw
Skip to content

v1alpha1

Package v1alpha1 contains API Schema definitions for the v1alpha1 API group.

Resource Types

Configuration

Configuration is the resource that contains the configuration used to run tests.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ Configuration
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec ConfigurationSpec ✅

Configuration spec.

Test

Test is the resource that contains aa test definition.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ Test
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec TestSpec ✅

Test spec.

TestStep

TestStep is the resource that contains the testStep used to run tests.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ TestStep
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec TestStepSpec ✅

TestStep spec.

Apply

Appears in:

Apply represents a set of configurations or resources that should be applied during testing.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the resources to be applied.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Assert

Appears in:

Assert represents a test condition that is expected to hold true during the testing process.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheck FileRefOrCheck ✅ ✅

FileRefOrAssert provides a reference to the assertion.
template bool

Template determines whether resources should be considered for templating.

Catch

Appears in:

Catch defines actions to be executed on failure.

Field Type Required Inline Description
description string

Description contains a description of the operation.
podLogs PodLogs

PodLogs determines the pod logs collector to execute.
events Events

Events determines the events collector to execute.
describe Describe

Describe determines the resource describe collector to execute.
get Get

Get determines the resource get collector to execute.
delete Delete

Delete represents a deletion operation.
command Command

Command defines a command to run.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Command

Appears in:

Command describes a command to run as a part of a test step.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
entrypoint string ✅

Entrypoint is the command entry point to run.
args []string

Args is the command arguments.
skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.
check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

ConfigurationSpec

Appears in:

ConfigurationSpec contains the configuration used to run tests.

Field Type Required Inline Description
timeouts Timeouts

Global timeouts configuration. Applies to all tests/test steps if not overridden.
skipDelete bool

If set, do not delete the resources after running the tests (implies SkipClusterDelete).
template bool

Template determines whether resources should be considered for templating.
failFast bool

FailFast determines whether the test should stop upon encountering the first failure.
parallel int

The maximum number of tests to run at once.
reportFormat ReportFormatType

ReportFormat determines test report format (JSON
reportName string

ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace string

Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.
fullName bool

FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex string

ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex string

IncludeTestRegex is used to include tests based on a regular expression.
repeatCount int

RepeatCount indicates how many times the tests should be executed.
testFile string

TestFile is the name of the file containing the test to run.
forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

Create

Appears in:

Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the file containing the resources to be created.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Delete

Appears in:

Delete is a reference to an object that should be deleted

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
template bool

Template determines whether resources should be considered for templating.
ref ObjectReference ✅

ObjectReference determines objects to be deleted.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Describe

Appears in:

Describe defines how to describe resources.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
resource string ✅

Resource type.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
showEvents bool

Show Events indicates whether to include related events.

Error

Appears in:

Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheck FileRefOrCheck ✅ ✅

FileRefOrAssert provides a reference to the expected error.
template bool

Template determines whether resources should be considered for templating.

Events

Appears in:

Events defines how to collect events.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
format Format

Format determines the output format (json or yaml).

Expectation

Appears in:

Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.

Field Type Required Inline Description
match policy/v1alpha1.Any

Match defines the matching statement.
check policy/v1alpha1.Any ✅

Check defines the verification statement.

FileRef

Appears in:

FileRef represents a file reference.

Field Type Required Inline Description
file string ✅

File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.

FileRefOrCheck

Appears in:

FileRefOrCheck represents a file reference or resource.

Field Type Required Inline Description
FileRef FileRef ✅

FileRef provides a reference to the file containing the resources to be applied.
resource policy/v1alpha1.Any

Check provides a check used in assertions.

FileRefOrResource

Appears in:

FileRefOrResource represents a file reference or resource.

Field Type Required Inline Description
FileRef FileRef ✅

FileRef provides a reference to the file containing the resources to be applied.
resource meta/v1/unstructured.Unstructured

Resource provides a resource to be applied.

Finally

Appears in:

Finally defines actions to be executed at the end of a test.

Field Type Required Inline Description
description string

Description contains a description of the operation.
podLogs PodLogs

PodLogs determines the pod logs collector to execute.
events Events

Events determines the events collector to execute.
describe Describe

Describe determines the resource describe collector to execute.
get Get

Get determines the resource get collector to execute.
delete Delete

Delete represents a deletion operation.
command Command

Command defines a command to run.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Format

(Alias of string)

Appears in:

Format determines the output format (json or yaml).

Get

Appears in:

Get defines how to get resources.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
resource string ✅

Resource type.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
format Format

Format determines the output format (json or yaml).

ObjectLabelsSelector

Appears in:

ObjectLabelsSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use selector.

Field Type Required Inline Description
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector string

Selector defines labels selector.

ObjectReference

Appears in:

ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description
ObjectSelector ObjectSelector ✅ ✅

ObjectSelector determines the selection process of referenced objects.
apiVersion string ✅

API version of the referent.
kind string ✅

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

ObjectSelector

Appears in:

ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
labels map[string]string

Label selector to match objects to delete

Operation

Appears in:

Operation defines a single operation, only one action is permitted for a given operation.

Field Type Required Inline Description
description string

Description contains a description of the operation.
continueOnError bool

ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
apply Apply

Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assert Assert

Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
command Command

Command defines a command to run.
create Create

Create represents a creation operation.
delete Delete

Delete represents a deletion operation.
error Error

Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
patch Patch

Patch represents a patch operation.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Patch

Appears in:

Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the file containing the resources to be patched.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

PodLogs

Appears in:

PodLogs defines how to collect pod logs.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector string

Selector defines labels selector.
container string

Container in pod to get logs from else --all-containers is used.
tail int

Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.

ReportFormatType

(Alias of string)

Appears in:

Script

Appears in:

Script describes a script to run as a part of a test step.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
content string

Content defines a shell script (run with "sh -c ...").
skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.
check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

Sleep

Appears in:

Sleep represents a duration while nothing happens.

Field Type Required Inline Description
duration meta/v1.Duration ✅

Duration is the delay used for sleeping.

TestSpec

Appears in:

TestSpec contains the test spec.

Field Type Required Inline Description
description string

Description contains a description of the test.
timeouts Timeouts

Timeouts for the test. Overrides the global timeouts set in the Configuration on a per operation basis.
skip bool

Skip determines whether the test should skipped.
concurrent bool

Concurrent determines whether the test should run concurrently with other tests.
skipDelete bool

SkipDelete determines whether the resources created by the test should be deleted after the test is executed.
template bool

Template determines whether resources should be considered for templating.
namespace string

Namespace determines whether the test should run in a random ephemeral namespace or not.
namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.
steps []TestSpecStep ✅

Steps defining the test.
forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

TestSpecStep

Appears in:

TestSpecStep contains the test step definition used in a test spec.

Field Type Required Inline Description
name string

Name of the step.
TestStepSpec TestStepSpec ✅ ✅

TestStepSpec of the step.

TestStepSpec

Appears in:

TestStepSpec defines the desired state and behavior for each test step.

Field Type Required Inline Description
description string

Description contains a description of the test step.
timeouts Timeouts

Timeouts for the test step. Overrides the global timeouts set in the Configuration and the timeouts eventually set in the Test.
skipDelete bool

SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
template bool

Template determines whether resources should be considered for templating.
try []Operation ✅

Try defines what the step will try to execute.
catch []Catch

Catch defines what the step will execute when an error happens.
finally []Finally

Finally defines what the step will execute after the step is terminated.

Timeouts

Appears in:

Timeouts contains timeouts per operation.

Field Type Required Inline Description
apply meta/v1.Duration ✅

Apply defines the timeout for the apply operation
assert meta/v1.Duration ✅

Assert defines the timeout for the assert operation
cleanup meta/v1.Duration ✅

Cleanup defines the timeout for the cleanup operation
delete meta/v1.Duration ✅

Delete defines the timeout for the delete operation
error meta/v1.Duration ✅

Error defines the timeout for the error operation
exec meta/v1.Duration ✅

Exec defines the timeout for exec operations
\ No newline at end of file + chainsaw (v1alpha1) - Chainsaw
Skip to content

v1alpha1

Package v1alpha1 contains API Schema definitions for the v1alpha1 API group.

Resource Types

Configuration

Configuration is the resource that contains the configuration used to run tests.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ Configuration
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec ConfigurationSpec ✅

Configuration spec.

Test

Test is the resource that contains aa test definition.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ Test
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec TestSpec ✅

Test spec.

TestStep

TestStep is the resource that contains the testStep used to run tests.

Field Type Required Inline Description
apiVersion string ✅ chainsaw.kyverno.io/v1alpha1
kind string ✅ TestStep
metadata meta/v1.ObjectMeta

Standard object's metadata.
spec TestStepSpec ✅

TestStep spec.

Apply

Appears in:

Apply represents a set of configurations or resources that should be applied during testing.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the resources to be applied.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Assert

Appears in:

Assert represents a test condition that is expected to hold true during the testing process.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheck FileRefOrCheck ✅ ✅

FileRefOrAssert provides a reference to the assertion.
template bool

Template determines whether resources should be considered for templating.

Catch

Appears in:

Catch defines actions to be executed on failure.

Field Type Required Inline Description
description string

Description contains a description of the operation.
podLogs PodLogs

PodLogs determines the pod logs collector to execute.
events Events

Events determines the events collector to execute.
describe Describe

Describe determines the resource describe collector to execute.
get Get

Get determines the resource get collector to execute.
delete Delete

Delete represents a deletion operation.
command Command

Command defines a command to run.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Cluster

Appears in:

Field Type Required Inline Description
kubeconfig string ✅

Kubeconfig is the path to the referenced file.
context string

Context is the name of the context to use.

Command

Appears in:

Command describes a command to run as a part of a test step.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
entrypoint string ✅

Entrypoint is the command entry point to run.
args []string

Args is the command arguments.
skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.
check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

ConfigurationSpec

Appears in:

ConfigurationSpec contains the configuration used to run tests.

Field Type Required Inline Description
timeouts Timeouts

Global timeouts configuration. Applies to all tests/test steps if not overridden.
skipDelete bool

If set, do not delete the resources after running the tests (implies SkipClusterDelete).
template bool

Template determines whether resources should be considered for templating.
failFast bool

FailFast determines whether the test should stop upon encountering the first failure.
parallel int

The maximum number of tests to run at once.
reportFormat ReportFormatType

ReportFormat determines test report format (JSON
reportName string

ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace string

Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.
fullName bool

FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex string

ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex string

IncludeTestRegex is used to include tests based on a regular expression.
repeatCount int

RepeatCount indicates how many times the tests should be executed.
testFile string

TestFile is the name of the file containing the test to run.
forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
clusters map[string]Cluster

Clusters holds a registry to clusters to support multi-cluster tests.

Create

Appears in:

Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the file containing the resources to be created.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Delete

Appears in:

Delete is a reference to an object that should be deleted

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
template bool

Template determines whether resources should be considered for templating.
ref ObjectReference ✅

ObjectReference determines objects to be deleted.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

Describe

Appears in:

Describe defines how to describe resources.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
resource string ✅

Resource type.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
showEvents bool

Show Events indicates whether to include related events.

Error

Appears in:

Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrCheck FileRefOrCheck ✅ ✅

FileRefOrAssert provides a reference to the expected error.
template bool

Template determines whether resources should be considered for templating.

Events

Appears in:

Events defines how to collect events.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
format Format

Format determines the output format (json or yaml).

Expectation

Appears in:

Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.

Field Type Required Inline Description
match policy/v1alpha1.Any

Match defines the matching statement.
check policy/v1alpha1.Any ✅

Check defines the verification statement.

FileRef

Appears in:

FileRef represents a file reference.

Field Type Required Inline Description
file string ✅

File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.

FileRefOrCheck

Appears in:

FileRefOrCheck represents a file reference or resource.

Field Type Required Inline Description
FileRef FileRef ✅

FileRef provides a reference to the file containing the resources to be applied.
resource policy/v1alpha1.Any

Check provides a check used in assertions.

FileRefOrResource

Appears in:

FileRefOrResource represents a file reference or resource.

Field Type Required Inline Description
FileRef FileRef ✅

FileRef provides a reference to the file containing the resources to be applied.
resource meta/v1/unstructured.Unstructured

Resource provides a resource to be applied.

Finally

Appears in:

Finally defines actions to be executed at the end of a test.

Field Type Required Inline Description
description string

Description contains a description of the operation.
podLogs PodLogs

PodLogs determines the pod logs collector to execute.
events Events

Events determines the events collector to execute.
describe Describe

Describe determines the resource describe collector to execute.
get Get

Get determines the resource get collector to execute.
delete Delete

Delete represents a deletion operation.
command Command

Command defines a command to run.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Format

(Alias of string)

Appears in:

Format determines the output format (json or yaml).

Get

Appears in:

Get defines how to get resources.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
resource string ✅

Resource type.
ObjectLabelsSelector ObjectLabelsSelector ✅ ✅

ObjectLabelsSelector determines the selection process of referenced objects.
format Format

Format determines the output format (json or yaml).

ObjectLabelsSelector

Appears in:

ObjectLabelsSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use selector.

Field Type Required Inline Description
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector string

Selector defines labels selector.

ObjectReference

Appears in:

ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description
ObjectSelector ObjectSelector ✅ ✅

ObjectSelector determines the selection process of referenced objects.
apiVersion string ✅

API version of the referent.
kind string ✅

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

ObjectSelector

Appears in:

ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
labels map[string]string

Label selector to match objects to delete

Operation

Appears in:

Operation defines a single operation, only one action is permitted for a given operation.

Field Type Required Inline Description
description string

Description contains a description of the operation.
continueOnError bool

ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
apply Apply

Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assert Assert

Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
command Command

Command defines a command to run.
create Create

Create represents a creation operation.
delete Delete

Delete represents a deletion operation.
error Error

Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
patch Patch

Patch represents a patch operation.
script Script

Script defines a script to run.
sleep Sleep

Sleep defines zzzz.

Patch

Appears in:

Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
FileRefOrResource FileRefOrResource ✅ ✅

FileRefOrResource provides a reference to the file containing the resources to be patched.
template bool

Template determines whether resources should be considered for templating.
dryRun bool

DryRun determines whether the file should be applied in dry run mode.
expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

PodLogs

Appears in:

PodLogs defines how to collect pod logs.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
selector string

Selector defines labels selector.
container string

Container in pod to get logs from else --all-containers is used.
tail int

Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.

ReportFormatType

(Alias of string)

Appears in:

Script

Appears in:

Script describes a script to run as a part of a test step.

Field Type Required Inline Description
timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.
content string

Content defines a shell script (run with "sh -c ...").
skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.
check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

Sleep

Appears in:

Sleep represents a duration while nothing happens.

Field Type Required Inline Description
duration meta/v1.Duration ✅

Duration is the delay used for sleeping.

TestSpec

Appears in:

TestSpec contains the test spec.

Field Type Required Inline Description
description string

Description contains a description of the test.
timeouts Timeouts

Timeouts for the test. Overrides the global timeouts set in the Configuration on a per operation basis.
skip bool

Skip determines whether the test should skipped.
concurrent bool

Concurrent determines whether the test should run concurrently with other tests.
skipDelete bool

SkipDelete determines whether the resources created by the test should be deleted after the test is executed.
template bool

Template determines whether resources should be considered for templating.
namespace string

Namespace determines whether the test should run in a random ephemeral namespace or not.
namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.
steps []TestSpecStep ✅

Steps defining the test.
forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

TestSpecStep

Appears in:

TestSpecStep contains the test step definition used in a test spec.

Field Type Required Inline Description
name string

Name of the step.
TestStepSpec TestStepSpec ✅ ✅

TestStepSpec of the step.

TestStepSpec

Appears in:

TestStepSpec defines the desired state and behavior for each test step.

Field Type Required Inline Description
description string

Description contains a description of the test step.
timeouts Timeouts

Timeouts for the test step. Overrides the global timeouts set in the Configuration and the timeouts eventually set in the Test.
skipDelete bool

SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
template bool

Template determines whether resources should be considered for templating.
try []Operation ✅

Try defines what the step will try to execute.
catch []Catch

Catch defines what the step will execute when an error happens.
finally []Finally

Finally defines what the step will execute after the step is terminated.

Timeouts

Appears in:

Timeouts contains timeouts per operation.

Field Type Required Inline Description
apply meta/v1.Duration ✅

Apply defines the timeout for the apply operation
assert meta/v1.Duration ✅

Assert defines the timeout for the assert operation
cleanup meta/v1.Duration ✅

Cleanup defines the timeout for the cleanup operation
delete meta/v1.Duration ✅

Delete defines the timeout for the delete operation
error meta/v1.Duration ✅

Error defines the timeout for the error operation
exec meta/v1.Duration ✅

Exec defines the timeout for exec operations
\ No newline at end of file diff --git a/main/commands/chainsaw_test/index.html b/main/commands/chainsaw_test/index.html index 13ba4fed5..c249ac868 100644 --- a/main/commands/chainsaw_test/index.html +++ b/main/commands/chainsaw_test/index.html @@ -3,6 +3,7 @@ --assert-timeout duration The assert timeout to use as default for configuration (default 30s) --cleanup-delay duration Adds a delay between the time a test ends and the time cleanup starts --cleanup-timeout duration The cleanup timeout to use as default for configuration (default 30s) + --cluster strings Register cluster (format <cluster name>=<kubeconfig path>:[context name]) --config string Chainsaw configuration file --delete-timeout duration The delete timeout to use as default for configuration (default 15s) --error-timeout duration The error timeout to use as default for configuration (default 30s) diff --git a/main/search/search_index.json b/main/search/search_index.json index bfff38aec..80a5060f3 100644 --- a/main/search/search_index.json +++ b/main/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"community/","title":"Community","text":"

Chainsaw has a growing community and we would definitely love to see you join and contribute.

Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.

Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters

"},{"location":"community/#slack-channel","title":"Slack channel","text":"

Join our slack channel #kyverno-chainsaw to meet with users, contributors and maintainers.

"},{"location":"community/#community-meetings","title":"Community Meetings","text":"

To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.

This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.

Weekly every Thursday at 2:00 PM UTC

"},{"location":"community/#roadmap","title":"RoadMap","text":"

For detailed information on our planned features and upcoming updates, please view our Roadmap.

"},{"location":"community/#contributing","title":"Contributing","text":"

Please read the contributing guide for details around:

  1. Code of Conduct
  2. Code Culture
  3. Details on how to contribute
"},{"location":"community/#adopters","title":"Adopters","text":"

If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces

"},{"location":"docs/","title":"Documentation","text":"

Starting with Chainsaw v0.0.6 we maintain separate docs per version:

"},{"location":"gh-action/","title":"GitHub action","text":"

A GitHub action is available to easily install Chainsaw in your workflows.

The GitHub action is available at kyverno/action-install-chainsaw or in the marketplace.

"},{"location":"gh-action/#usage","title":"Usage","text":"

This action currently supports GitHub-provided Linux, macOS and Windows runners (self-hosted runners may not work).

Add the following entry to your Github workflow YAML file:

uses: kyverno/action-install-chainsaw@v0.1.0\nwith:\n  release: v0.1.0 # optional\n

Example using a pinned version:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          release: v0.0.9\n      - name: Check install\n        run: chainsaw version\n

Example using the default version:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n      - name: Check install\n        run: chainsaw version\n

Example using cosign verification:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Cosign\n        uses: sigstore/cosign-installer@v3.1.1\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          verify: true\n      - name: Check install\n        run: chainsaw version\n

If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.

Example of installing Chainsaw via go install:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw via go install\n    steps:\n      - name: Install go\n        uses: actions/setup-go@v4\n        with:\n          go-version: '1.21'\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          release: main\n      - name: Check install\n        run: chainsaw version\n
"},{"location":"gh-action/#optional-inputs","title":"Optional Inputs","text":"

The following optional inputs:

Input Description release chainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"install/","title":"Install","text":"

You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.

We also provide a GitHub action to easily install Chainsaw in your workflows.

"},{"location":"install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"install/#homebrew-tap","title":"Homebrew tap","text":"

add tap:

brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n

install chainsaw:

brew install kyverno/chainsaw/chainsaw\n

Don't forget to specify the tap name

Homebrew core already has a tool named chainsaw.

Be sure that you specify the tap name when installing to install the right tool.

"},{"location":"install/#manually","title":"Manually","text":"

Download the pre-compiled binaries for your system from the releases page and copy them to the desired location.

"},{"location":"install/#install-using-go-install","title":"Install using go install","text":"

You can install with go install with:

go install github.com/kyverno/chainsaw@latest\n
"},{"location":"install/#running-with-docker","title":"Running with Docker","text":"

Chainsaw is also available as a Docker image which you can pull and run:

docker pull ghcr.io/kyverno/chainsaw:<version>\n

Info

Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.

docker run --rm                             \\\n    -v ./testdata/e2e/:/chainsaw/           \\\n    -v ${HOME}/.kube/:/etc/kubeconfig/      \\\n    -e KUBECONFIG=/etc/kubeconfig/config    \\\n    --network=host                          \\\n    ghcr.io/kyverno/chainsaw:<version>      \\\n    test /chainsaw --config /chainsaw/config.yaml\n
"},{"location":"install/#compiling-from-sources","title":"Compiling from sources","text":"

clone:

git clone https://github.com/kyverno/chainsaw.git\n
build the binaries:

cd chainsaw\ngo mod tidy\nmake build\n

verify it works:

./chainsaw version\n
"},{"location":"install/#github-action","title":"GitHub action","text":"

A GitHub action is available to install Chainsaw in your workflows. See the GitHub action dedicated documentation.

"},{"location":"intro/","title":"Introduction","text":"

Chainsaw is a tool developed to run end to end tests in Kubernetes clusters.

It is meant to test Kubernetes operators work as expected by running a sequence of test steps for:

"},{"location":"intro/#why-we-made-it","title":"Why we made it ?","text":"

While developing Kyverno we need to run end to end tests to make sure our admission controller works as expected.

A typical Kyverno end to end test

Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:

  1. Create a policy
  2. Create a resource
  3. Check that Kyverno acted as expected
  4. Cleanup and move to the next test
"},{"location":"intro/#from-kuttl-to-chainsaw","title":"From KUTTL to Chainsaw","text":"

We started with another tool called KUTTL.

While KUTTL was a great tool to start with, we quickly identified some limitations and forked it to add the features we needed.

In the end we needed more flexibility than what KUTTL could offer and we started designing a new assertion model and at this point it was simpler to start a new tool from scratch than continuing making changes in our KUTTL fork.

The changes we were making was simply too large to have a chance to be incorporated upstream.

Chainsaw was born

"},{"location":"intro/#kuttl-compatibility","title":"KUTTL compatibility","text":"

Fortunately, Chainsaw assertion model is compatible with KUTTL and therefore migration to Chainsaw is relatively easy.

We provide chainsaw migrate kuttl config and chainsaw migrate kuttl tests commands to assist with the migration, see Migration from KUTTL for details.

"},{"location":"intro/#use-cases","title":"Use cases","text":"

Chainsaw is built with CI tools in mind - you only really need to download and execute it in your build script.

Installing it in your machine is entirely up to you, but still possible.

"},{"location":"json-schemas/","title":"JSON schemas","text":"

JSON schemas for Chainsaw resources are available to enable validation and autocompletion in your IDE:

"},{"location":"json-schemas/#vs-code","title":"VS code","text":"

In VS code, simply add a comment on top of your YAML resources.

"},{"location":"json-schemas/#test","title":"Test","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  steps:\n  - try:\n    - apply:\n        file: configmap.yaml\n    - assert:\n        file: configmap-assert.yaml\n
"},{"location":"json-schemas/#configuration","title":"Configuration","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/configuration-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: congiguration\nspec:\n  fullName: true\n  failFast: true\n  forceTerminationGracePeriod: 5s\n
"},{"location":"json-schemas/#applying-crds","title":"Applying CRDs","text":"

An alternative is to apply the Chainsaw CRDs in a kubernetes cluster and let the Kubernetes extension do the rest.

CRD definitions are provided in our GitHub repository.

The command below will apply the Chainsaw CRDs to the configured cluster:

kubectl apply -f https://raw.githubusercontent.com/kyverno/chainsaw/main/config/crds/chainsaw.kyverno.io_configurations.yaml\nkubectl apply -f https://raw.githubusercontent.com/kyverno/chainsaw/main/config/crds/chainsaw.kyverno.io_tests.yaml\n
"},{"location":"json-schemas/#exporting-schemas","title":"Exporting schemas","text":"

Chainsaw can export JSON schemas locally.

chainsaw export schemas <local path>\n

See Chainsaw export schemas reference for more details.

"},{"location":"quick-start/","title":"Quick start","text":"

To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.

In this Quick start we will use kind but feel free to use the tool of your choice.

Not a cluster management tool

We consider this is not the responsibility of Chainsaw to manage clusters.

There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.

"},{"location":"quick-start/#create-a-kind-cluster","title":"Create a kind cluster","text":"

Please refer to the kind install docs to install it locally.

Once kind creating a local cluster is as simple as running:

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n
"},{"location":"quick-start/#writing-tests","title":"Writing tests","text":"

A Chainsaw test is made of YAML files in a folder.

YAML files can contain raw manifests with a special file naming convention to identify the step operations. This is useful to create tests quickly but doesn't allow great flexibility.

Another option is to have a chainsaw-test.yaml file containing a Test resource. While more verbose, this offers full flexibility over the test and test steps configuration.

"},{"location":"quick-start/#quick-start_1","title":"Quick start","text":"

For this quick start, we will create a Test with one step and two operations:

  1. Create a ConfigMap from a manifest
  2. Verify the ConfigMap was created and contains the expected data
"},{"location":"quick-start/#create-the-test-folder","title":"Create the test folder","text":"
# create test folder\nmkdir chainsaw-quick-start\n\n# enter test folder\ncd chainsaw-quick-start\n
"},{"location":"quick-start/#create-a-configmap","title":"Create a ConfigMap","text":"
# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\nEOF\n
"},{"location":"quick-start/#create-the-test","title":"Create the test","text":"

By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.

# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: quick-start\nspec:\n  steps:\n  - try:\n    # first operation: create the config map\n    - apply:\n        # file is relative to the test folder\n        file: configmap.yaml\n    # second operation: verify the config map exists and contains the expected data\n    - assert:\n        # file is relative to the test folder\n        file: configmap.yaml\nEOF\n
"},{"location":"quick-start/#run-chainsaw","title":"Run Chainsaw","text":"

We finished writing our first test, now we can run Chainsaw to verify the test runs as expected:

chainsaw test\n\nVersion: (devel)\nLoading default configuration...\n- Using test file: chainsaw-test.yaml\n- TestDirs [.]\n- SkipDelete false\n- FailFast false\n- ReportFormat ''\n- ReportName 'chainsaw-report'\n- Namespace ''\n- FullName false\n- IncludeTestRegex ''\n- ExcludeTestRegex ''\n- ApplyTimeout 5s\n- AssertTimeout 30s\n- CleanupTimeout 30s\n- DeleteTimeout 15s\n- ErrorTimeout 30s\n- ExecTimeout 5s\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN   chainsaw\n=== PAUSE chainsaw\n=== CONT  chainsaw\n=== RUN   chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT  chainsaw/quick-start\n    | 10:44:26 | quick-start | @setup   | CREATE    | OK    | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:26 | quick-start | step-1   | TRY       | RUN   |\n    | 10:44:26 | quick-start | step-1   | APPLY     | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | CREATE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | APPLY     | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | ASSERT    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | ASSERT    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | TRY       | DONE  |\n    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:31 | quick-start | @cleanup | DELETE    | DONE  | v1/Namespace @ chainsaw-immense-jay\n--- PASS: chainsaw (0.00s)\n    --- PASS: chainsaw/quick-start (5.25s)\nPASS\nTests Summary...\n- Passed  tests 1\n- Failed  tests 0\n- Skipped tests 0\nDone.\n
"},{"location":"resources/","title":"Resources","text":"

Resources, blog posts and videos talking about Chainsaw:

"},{"location":"writing-tests/","title":"What is Chainsaw","text":""},{"location":"writing-tests/#overview","title":"Overview","text":"

Chainsaw provides a declarative approach to testing production-grade Kubernetes operators and controllers.

It provides a way to inject an operator (subject under test) during the setup and allows tests to be standard YAML files.

In Chainsaw everything is YAML. We use YAML to manipulate resources, define cluster state assertions, describe commands to run, etc... Everything is declarative.

"},{"location":"writing-tests/#motivation","title":"Motivation","text":"

Testing Kubernetes operators is not easy.

At Kyverno we are building complex controllers and admission controllers that need to be deeply end to end tested.

Advantages of a declarative approach

It made sense to be able to declaratively create end to end tests for a couple of reasons:

"},{"location":"writing-tests/#when-would-you-use-chainsaw","title":"When would you use Chainsaw","text":"

The testing eco-system is vast and includes at a minimum low level unit tests, integration tests and end-to-end testing.

Chainsaw is built to support some kubernetes integration test scenarios and is most valuable as an end to end testing tool.

Chainsaw is great for

"},{"location":"apis/chainsaw.v1alpha1/","title":"chainsaw (v1alpha1)","text":"

Package v1alpha1 contains API Schema definitions for the v1alpha1 API group.

"},{"location":"apis/chainsaw.v1alpha1/#resource-types","title":"Resource Types","text":""},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Configuration","title":"Configuration","text":"

Configuration is the resource that contains the configuration used to run tests.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string Configuration metadata meta/v1.ObjectMeta

Standard object's metadata.

spec ConfigurationSpec

Configuration spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Test","title":"Test","text":"

Test is the resource that contains aa test definition.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string Test metadata meta/v1.ObjectMeta

Standard object's metadata.

spec TestSpec

Test spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestStep","title":"TestStep","text":"

TestStep is the resource that contains the testStep used to run tests.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string TestStep metadata meta/v1.ObjectMeta

Standard object's metadata.

spec TestStepSpec

TestStep spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Apply","title":"Apply","text":"

Appears in:

Apply represents a set of configurations or resources that should be applied during testing.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the resources to be applied.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Assert","title":"Assert","text":"

Appears in:

Assert represents a test condition that is expected to hold true during the testing process.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrCheck FileRefOrCheck

FileRefOrAssert provides a reference to the assertion.

template bool

Template determines whether resources should be considered for templating.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Catch","title":"Catch","text":"

Appears in:

Catch defines actions to be executed on failure.

Field Type Required Inline Description description string

Description contains a description of the operation.

podLogs PodLogs

PodLogs determines the pod logs collector to execute.

events Events

Events determines the events collector to execute.

describe Describe

Describe determines the resource describe collector to execute.

get Get

Get determines the resource get collector to execute.

delete Delete

Delete represents a deletion operation.

command Command

Command defines a command to run.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Command","title":"Command","text":"

Appears in:

Command describes a command to run as a part of a test step.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

entrypoint string

Entrypoint is the command entry point to run.

args []string

Args is the command arguments.

skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.

check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ConfigurationSpec","title":"ConfigurationSpec","text":"

Appears in:

ConfigurationSpec contains the configuration used to run tests.

Field Type Required Inline Description timeouts Timeouts

Global timeouts configuration. Applies to all tests/test steps if not overridden.

skipDelete bool

If set, do not delete the resources after running the tests (implies SkipClusterDelete).

template bool

Template determines whether resources should be considered for templating.

failFast bool

FailFast determines whether the test should stop upon encountering the first failure.

parallel int

The maximum number of tests to run at once.

reportFormat ReportFormatType

ReportFormat determines test report format (JSON reportName string

ReportName defines the name of report to create. It defaults to \"chainsaw-report\".

namespace string

Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.

namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.

fullName bool

FullName makes use of the full test case folder path instead of the folder name.

excludeTestRegex string

ExcludeTestRegex is used to exclude tests based on a regular expression.

includeTestRegex string

IncludeTestRegex is used to include tests based on a regular expression.

repeatCount int

RepeatCount indicates how many times the tests should be executed.

testFile string

TestFile is the name of the file containing the test to run.

forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.

delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Create","title":"Create","text":"

Appears in:

Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the file containing the resources to be created.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Delete","title":"Delete","text":"

Appears in:

Delete is a reference to an object that should be deleted

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

template bool

Template determines whether resources should be considered for templating.

ref ObjectReference

ObjectReference determines objects to be deleted.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Describe","title":"Describe","text":"

Appears in:

Describe defines how to describe resources.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

resource string

Resource type.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

showEvents bool

Show Events indicates whether to include related events.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Error","title":"Error","text":"

Appears in:

Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrCheck FileRefOrCheck

FileRefOrAssert provides a reference to the expected error.

template bool

Template determines whether resources should be considered for templating.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Events","title":"Events","text":"

Appears in:

Events defines how to collect events.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

format Format

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Expectation","title":"Expectation","text":"

Appears in:

Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.

Field Type Required Inline Description match policy/v1alpha1.Any

Match defines the matching statement.

check policy/v1alpha1.Any

Check defines the verification statement.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRef","title":"FileRef","text":"

Appears in:

FileRef represents a file reference.

Field Type Required Inline Description file string

File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRefOrCheck","title":"FileRefOrCheck","text":"

Appears in:

FileRefOrCheck represents a file reference or resource.

Field Type Required Inline Description FileRef FileRef

FileRef provides a reference to the file containing the resources to be applied.

resource policy/v1alpha1.Any

Check provides a check used in assertions.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRefOrResource","title":"FileRefOrResource","text":"

Appears in:

FileRefOrResource represents a file reference or resource.

Field Type Required Inline Description FileRef FileRef

FileRef provides a reference to the file containing the resources to be applied.

resource meta/v1/unstructured.Unstructured

Resource provides a resource to be applied.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Finally","title":"Finally","text":"

Appears in:

Finally defines actions to be executed at the end of a test.

Field Type Required Inline Description description string

Description contains a description of the operation.

podLogs PodLogs

PodLogs determines the pod logs collector to execute.

events Events

Events determines the events collector to execute.

describe Describe

Describe determines the resource describe collector to execute.

get Get

Get determines the resource get collector to execute.

delete Delete

Delete represents a deletion operation.

command Command

Command defines a command to run.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Format","title":"Format","text":"

(Alias of string)

Appears in:

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Get","title":"Get","text":"

Appears in:

Get defines how to get resources.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

resource string

Resource type.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

format Format

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectLabelsSelector","title":"ObjectLabelsSelector","text":"

Appears in:

ObjectLabelsSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use selector.

Field Type Required Inline Description namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

selector string

Selector defines labels selector.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectReference","title":"ObjectReference","text":"

Appears in:

ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description ObjectSelector ObjectSelector

ObjectSelector determines the selection process of referenced objects.

apiVersion string

API version of the referent.

kind string

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectSelector","title":"ObjectSelector","text":"

Appears in:

ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

labels map[string]string

Label selector to match objects to delete

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Operation","title":"Operation","text":"

Appears in:

Operation defines a single operation, only one action is permitted for a given operation.

Field Type Required Inline Description description string

Description contains a description of the operation.

continueOnError bool

ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.

apply Apply

Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.

assert Assert

Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.

command Command

Command defines a command to run.

create Create

Create represents a creation operation.

delete Delete

Delete represents a deletion operation.

error Error

Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.

patch Patch

Patch represents a patch operation.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Patch","title":"Patch","text":"

Appears in:

Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the file containing the resources to be patched.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-PodLogs","title":"PodLogs","text":"

Appears in:

PodLogs defines how to collect pod logs.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

selector string

Selector defines labels selector.

container string

Container in pod to get logs from else --all-containers is used.

tail int

Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ReportFormatType","title":"ReportFormatType","text":"

(Alias of string)

Appears in:

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Script","title":"Script","text":"

Appears in:

Script describes a script to run as a part of a test step.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

content string

Content defines a shell script (run with \"sh -c ...\").

skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.

check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Sleep","title":"Sleep","text":"

Appears in:

Sleep represents a duration while nothing happens.

Field Type Required Inline Description duration meta/v1.Duration

Duration is the delay used for sleeping.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestSpec","title":"TestSpec","text":"

Appears in:

TestSpec contains the test spec.

Field Type Required Inline Description description string

Description contains a description of the test.

timeouts Timeouts

Timeouts for the test. Overrides the global timeouts set in the Configuration on a per operation basis.

skip bool

Skip determines whether the test should skipped.

concurrent bool

Concurrent determines whether the test should run concurrently with other tests.

skipDelete bool

SkipDelete determines whether the resources created by the test should be deleted after the test is executed.

template bool

Template determines whether resources should be considered for templating.

namespace string

Namespace determines whether the test should run in a random ephemeral namespace or not.

namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.

steps []TestSpecStep

Steps defining the test.

forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.

delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestSpecStep","title":"TestSpecStep","text":"

Appears in:

TestSpecStep contains the test step definition used in a test spec.

Field Type Required Inline Description name string

Name of the step.

TestStepSpec TestStepSpec

TestStepSpec of the step.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestStepSpec","title":"TestStepSpec","text":"

Appears in:

TestStepSpec defines the desired state and behavior for each test step.

Field Type Required Inline Description description string

Description contains a description of the test step.

timeouts Timeouts

Timeouts for the test step. Overrides the global timeouts set in the Configuration and the timeouts eventually set in the Test.

skipDelete bool

SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.

template bool

Template determines whether resources should be considered for templating.

try []Operation

Try defines what the step will try to execute.

catch []Catch

Catch defines what the step will execute when an error happens.

finally []Finally

Finally defines what the step will execute after the step is terminated.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Timeouts","title":"Timeouts","text":"

Appears in:

Timeouts contains timeouts per operation.

Field Type Required Inline Description apply meta/v1.Duration

Apply defines the timeout for the apply operation

assert meta/v1.Duration

Assert defines the timeout for the assert operation

cleanup meta/v1.Duration

Cleanup defines the timeout for the cleanup operation

delete meta/v1.Duration

Delete defines the timeout for the delete operation

error meta/v1.Duration

Error defines the timeout for the error operation

exec meta/v1.Duration

Exec defines the timeout for exec operations

"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/","title":"How to Perform Efficient E2E Testing with Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#introduction","title":"Introduction","text":"

Chainsaw is a powerful tool that enables efficient end-to-end (E2E) testing, making it an essential asset for software developers. Built to be stronger and more cohesive than KUTTL, Chainsaw offers a superior suite of features for Kubernetes (k8s) operators and objects. In this blog post, we will explore how to leverage Chainsaw for efficient E2E testing and enhance your overall testing workflow. Let's dive in!

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#table-of-contents","title":"Table of Contents","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-1-installing-chainsaw","title":"Section 1: Installing Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#11-downloading-chainsaw","title":"1.1 Downloading Chainsaw","text":"

To get started with Chainsaw, you'll first need to download the appropriate release for your operating system and architecture. The releases are available on the official GitHub page at the following link: Release

Here's a breakdown of the available releases:

Each release also comes with associated .pem, .sbom, .sbom.pem, and .sig files. These are used for verification and security purposes. If you're interested in verifying the integrity of your download, you can use these files. However, for the purpose of this guide, we'll focus on the main .tar.gz files.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#12-installing-chainsaw","title":"1.2 Installing Chainsaw","text":"

Once you've downloaded the appropriate .tar.gz file for your system, you can proceed with the installation. Here's a step-by-step guide:

tar -xzf chainsaw_<your_version_and_architecture>.tar.gz\n
sudo mv chainsaw /usr/local/bin/\n
chainsaw --version\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-2-writing-configurations","title":"Section 2: Writing Configurations","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#21-understanding-chainsaw-configurations","title":"2.1 Understanding Chainsaw Configurations","text":"

Chainsaw is a robust tool for end-to-end (e2e) testing in Kubernetes environments. It offers a flexible configuration system that allows users to define their testing parameters and conditions. The configuration can be set up in two primary ways:

Chainsaw follows a hierarchy in loading its configurations:

  1. User-specified Configuration: If you provide a configuration file explicitly using a command-line flag.
  2. Default Configuration File: If no configuration is specified, Chainsaw will search for a default file named .chainsaw.yaml in the current directory.
  3. Internal Default Configuration: If neither of the above is available, Chainsaw will resort to a default configuration embedded within its binary.

To specify a custom configuration, use the following command:

chainsaw test --config path/to/your/config.yaml\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#22-writing-chainsaw-configurations","title":"2.2 Writing Chainsaw Configurations","text":"

When writing a configuration for Chainsaw, you'll be working with a YAML file. Here's a basic example of what this might look like:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  timeout: 45s\n  skipDelete: false\n  failFast: true\n  parallel: 4\n  # ... other configurations\n

The full structure of the configuration file, including all possible fields and their descriptions, can be found in the Configuration API reference.

If you don't provide a custom configuration, Chainsaw will use its default:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: default\nspec: {}\n

You can also override specific configurations using command-line flags, even after loading a configuration file. For instance:

chainsaw test --config path/to/your/config.yaml --timeout 45s --parallel 4\n

In this example, even if the configuration file specifies different values for timeout and parallel, the command-line flags will take precedence.

For a comprehensive list of all supported flags and their descriptions, refer to the Chainsaw test command reference.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-3-creating-tests","title":"Section 3: Creating Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#31-understanding-chainsaw-tests","title":"3.1 Understanding Chainsaw Tests","text":"

In Chainsaw, a test is essentially an ordered sequence of test steps. These steps are executed sequentially, and if any step fails, the entire test is considered failed. Each test step can consist of multiple operations, such as creating, updating, or deleting resources in a Kubernetes cluster, or asserting that certain conditions are met.

Chainsaw supports three primary test definition mechanisms:

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#32-creating-chainsaw-tests","title":"3.2 Creating Chainsaw Tests","text":"

Creating tests in Chainsaw requires a clear understanding of the test definition mechanisms. Here's a detailed guide for each syntax:

  1. Using Manifests based syntax

File Naming: Files should follow the convention <step index>-<name|assert|error>.yaml. For instance, 00-configmap.yaml for resource creation, 01-assert.yaml for assertions, and 02-error.yaml for expected errors.

Example:

# 00-configmap.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-example\ndata:\n  key: value\n

  1. Using TestSteps based syntax

    Define a TestStep: Create a YAML file that defines a TestStep resource. This resource should specify operations like Apply, Assert, Delete, and Error.

Example:

# 01-test-step.yaml\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: TestStep\nmetadata:\n  name: apply-configmap-step\nspec:\n  apply:\n  - file: /resources/configmap.yaml\n

You can combine TestStep resources with raw Kubernetes manifests. For instance, a TestStep might apply a resource, while a separate manifest file makes assertions about that resource.

Example:

# 02-assert.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-example\ndata:\n  key: value\n
  1. Using Test based syntax

Define a Test: Start by creating a YAML file that defines a Test resource. This resource will have a spec section that outlines the entire test.

Example:

# chainsaw-test.yaml\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: full-test-example\nspec:\n  timeout: 10s\n  steps:\n  - spec:\n      apply:\n      - file: /resources/configmap.yaml\n  - spec:\n      assert:\n      - file: /resources/configmap-assert.yaml\n

Chainsaw processes the test by executing each step in sequence, ensuring that your Kubernetes environment meets the defined conditions.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-4-running-tests","title":"Section 4: Running Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#41-preparing-for-test-execution","title":"4.1 Preparing for Test Execution","text":"

Before executing your tests, ensure that:

  1. Kubernetes Cluster: You have access to a Kubernetes cluster where the tests will run. This could be a local cluster (like Minikube or kind) or a remote one.
  2. Kubeconfig: Your kubeconfig is correctly set up to point to the desired cluster. Chainsaw will use the current context from your kubeconfig to interact with the cluster.
  3. Test Files: All your test files, whether they are written using Manifests based, TestSteps based, or Test based syntax, are organized and accessible.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#42-executing-tests-in-chainsaw","title":"4.2 Executing Tests in Chainsaw","text":"

Once you're set up, running tests in Chainsaw is straightforward:

  1. Navigate to the Test Directory:
cd path/to/your/test/directory\n
  1. Run the Tests:
chainsaw test\n

This command will execute all tests in the current directory.

  1. View the Results: Chainsaw will display the results in the terminal. It provides a summary of passed, failed, and skipped tests. For any failed tests, Chainsaw will display detailed error messages to help diagnose the issue.

The output will resemble:

Loading default configuration...\n...\nRunning tests...\n=== RUN   quick-start\n...\n--- PASS: quick-start (5.22s)\nPASS\nTests Summary...\n- Passed  tests: 3\n- Failed  tests: 1\n- Skipped tests: 2\nDone.\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#conclusion","title":"Conclusion","text":"

Chainsaw provides a robust and flexible framework for end-to-end testing in Kubernetes environments. By understanding its configuration and test definition mechanisms, developers can create comprehensive test suites that ensure their Kubernetes applications and configurations are functioning as expected.

"},{"location":"blog/first-test/","title":"First working test !","text":"

First test run !

The quick start test is passing 

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n\n# create test folder\nmkdir quick-start\n\n# enter test folder\ncd quick-start\n\n# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\nEOF\n\n# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: quick-start\nspec:\n  steps:\n  # first step applies the config map\n  - apply:\n    # file is relative to the test folder\n    - file: configmap.yaml\n  # second step verifies the config map exists and contains the expected data\n  - assert:\n    # file is relative to the test folder\n    - file: configmap.yaml\nEOF\n\n../chainsaw test --test-dir .\n\nRunning without configuration\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN   chainsaw\n=== RUN   chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT  chainsaw/quick-start\n    runner.go:35: step-1\n    runner.go:35: apply chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n    runner.go:35: step-2\n    runner.go:35: assert chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n    runner.go:68: cleanup namespace: chainsaw-polite-chamois\n--- PASS: chainsaw (0.00s)\n    --- PASS: chainsaw/quick-start (5.09s)\nPASS\nDone.\n

Impressive !

"},{"location":"blog/wip/","title":"It's happening - Finally !","text":"

Hello everyone!

We finally started writing our own testing tool !

Let's join forces and make better, stronger Open Source and Community Driven tool.

"},{"location":"blog/wip/#why-we-made-it","title":"Why we made it ?","text":"

While developing Kyverno we needed to run end to end tests to make sure our admission controller worked as expected.

Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:

  1. Create a policy
  2. Create a resource
  3. Check that Kyverno acted as expected
  4. Cleanup and move to the next test

We started with another tool called KUTTL. While KUTTL is great we identified some limitations and forked the tool to add the features we needed.

At some point we needed more flexibility than what KUTTL offered and we designed a new assertion model.

This was simpler to start from scratch than continuing making changes in our KUTTL fork.

"},{"location":"blog/wip/#whats-next","title":"What's next ?","text":"

This is still WIP and needs a lot of work before we can consider it ready but things are moving fast.

We would love to build a community driven tool and welcome all contributors.

Feel free to fork this repository and start submitting pull requests

"},{"location":"collectors/","title":"Collectors","text":""},{"location":"collectors/#purpose","title":"Purpose","text":"

The purpose of collectors is to collect certain information about the outcome of a step should it fail (in the case of catch) or at the end of the step (in the case of finally).

The ultimate goal of collectors is to gather information about the failure of a step and therefore help understand what caused it to fail.

A test step can have an arbitrary number of collectors.

"},{"location":"collectors/#available-collectors","title":"Available collectors","text":""},{"location":"collectors/describe/","title":"Describe","text":"

Describing resources present in the cluster can help understand what happened and troubleshoot test failures.

"},{"location":"collectors/describe/#configuration","title":"Configuration","text":"

The full structure of the Describe resource is documented here.

"},{"location":"collectors/describe/#describe-pods","title":"Describe pods","text":"

If a name is specified, Chainsaw will describe resources that have a name starting with the provided name in the test namespace (or in the cluster if it is a clustered-level resource).

Describe pods in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        name: my-pod\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        name: my-pod\n    # ...\n

If a namespace is specified, Chainsaw will describe resources in the specified namespace.

Describe pods in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        namespace: foo\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        namespace: foo\n    # ...\n
"},{"location":"collectors/describe/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the resources to be described.

Describe pods using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        selector: app=my-app\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will describe resources in the specified namespace.

Describe pods using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/describe/#show-events","title":"Show events","text":"

The showEvents field can be used to enable or disable showing events when describing resources.

Default

By default, showEventsis true.

Do not show events

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        namespace: foo\n        showEvents: false\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        namespace: foo\n        showEvents: false\n    # ...\n
"},{"location":"collectors/events/","title":"Events","text":"

Collecting namespace events can help understand what happened inside the cluster.

"},{"location":"collectors/events/#configuration","title":"Configuration","text":"

The full structure of the Events resource is documented here.

"},{"location":"collectors/events/#single-event","title":"Single event","text":"

If a name is specified, Chainsaw will retrieve the specified event in the test namespace.

Collect event in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        name: my-event\n    # ...\n    finally:\n    - events:\n        name: my-event\n    # ...\n

If a namespace is specified, Chainsaw will retrieve the specified event in the specified namespace.

Collect event in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        name: my-event\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        name: my-event\n        namespace: foo\n    # ...\n
"},{"location":"collectors/events/#all-events","title":"All events","text":"

If no name and namespace is specified, Chainsaw will retrieve all events in the test namespace.

Collect all events in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events: {}\n    # ...\n    finally:\n    - events: {}\n    # ...\n

On the other hand, if a namespace is specified, Chainsaw will retrieve all events in the specified namespace.

Collect all events in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        namespace: foo\n    # ...\n
"},{"location":"collectors/events/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the events to be retrieved.

Collect events using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        selector: app=my-app\n    # ...\n    finally:\n    - events:\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will retrieve events using the specified namespace.

Collect events using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/","title":"Pod logs","text":"

Collecting pod logs can help understand what happened inside one or more pods.

"},{"location":"collectors/pod-logs/#configuration","title":"Configuration","text":"

The full structure of the PodLogs resource is documented here.

"},{"location":"collectors/pod-logs/#single-pod","title":"Single pod","text":"

If a pod name is specified, Chainsaw will retrieve logs from this specific pod in the test namespace.

Collect pod logs in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        name: my-pod\n    # ...\n    finally:\n    - podLogs:\n        name: my-pod\n    # ...\n

If a namespace is specified, Chainsaw will retrieve logs from this specific pod in the specified namespace.

Collect pod logs in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        name: my-pod\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        name: my-pod\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#all-pods","title":"All pods","text":"

If no pod name and namespace is specified, Chainsaw will retrieve logs from all pods in the test namespace.

Collect all pod logs in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs: {}\n    # ...\n    finally:\n    - podLogs: {}\n    # ...\n

On the other hand, if a namespace is specified, Chainsaw will retrieve logs from all pods in the specified namespace.

Collect all pod logs in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the pods to retrieve logs from.

Collect pod logs using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will retrieve pod logs using the specified namespace.

Collect pod logs using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#tail","title":"Tail","text":"

The tail field can be used to limit the amount of log lines retrieved when querying pod logs.

Default

By default, tail will be 10 when a label selector is used, and all if a pod name is specified.

Tail example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        tail: 30\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        tail: 30\n    # ...\n
"},{"location":"collectors/pod-logs/#container","title":"Container","text":"

The container field can be used to retrieve logs from a specific container in the pod.

Default

By default logs from all containers will be fetched.

Container example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        container: nginx\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        container: nginx\n    # ...\n
"},{"location":"commands/chainsaw/","title":"Chainsaw","text":""},{"location":"commands/chainsaw/#chainsaw","title":"chainsaw","text":"

Stronger tool for e2e testing

chainsaw [flags]\n
"},{"location":"commands/chainsaw/#options","title":"Options","text":"
  -h, --help   help for chainsaw\n
"},{"location":"commands/chainsaw/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_assert/","title":"Chainsaw assert","text":""},{"location":"commands/chainsaw_assert/#chainsaw-assert","title":"chainsaw assert","text":"

Evaluate assertion

chainsaw assert [flags] [FILE]\n
"},{"location":"commands/chainsaw_assert/#options","title":"Options","text":"
  -f, --file string                         Path to the file to assert or '-' to read from stdin\n  -h, --help                                help for assert\n      --kube-as string                      Username to impersonate for the operation\n      --kube-as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n      --kube-as-uid string                  UID to impersonate for the operation\n      --kube-certificate-authority string   Path to a cert file for the certificate authority\n      --kube-client-certificate string      Path to a client certificate file for TLS\n      --kube-client-key string              Path to a client key file for TLS\n      --kube-cluster string                 The name of the kubeconfig cluster to use\n      --kube-context string                 The name of the kubeconfig context to use\n      --kube-disable-compression            If true, opt-out of response compression for all requests to the server\n      --kube-insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n  -n, --kube-namespace string               If present, the namespace scope for this CLI request\n      --kube-password string                Password for basic authentication to the API server\n      --kube-proxy-url string               If provided, this URL will be used to connect via proxy\n      --kube-request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n      --kube-server string                  The address and port of the Kubernetes API server\n      --kube-tls-server-name string         If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n      --kube-token string                   Bearer token for authentication to the API server\n      --kube-user string                    The name of the kubeconfig user to use\n      --kube-username string                Username for basic authentication to the API server\n      --namespace string                    Namespace to use (default \"default\")\n      --no-color                            Removes output colors\n      --timeout duration                    The assert timeout to use (default 30s)\n
"},{"location":"commands/chainsaw_assert/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_build/","title":"Chainsaw build","text":""},{"location":"commands/chainsaw_build/#chainsaw-build","title":"chainsaw build","text":"

Build commands

chainsaw build [flags]\n
"},{"location":"commands/chainsaw_build/#options","title":"Options","text":"
  -h, --help   help for build\n
"},{"location":"commands/chainsaw_build/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_build_docs/","title":"Chainsaw build docs","text":""},{"location":"commands/chainsaw_build_docs/#chainsaw-build-docs","title":"chainsaw build docs","text":"

Build tests documentation

chainsaw build docs [flags]\n
"},{"location":"commands/chainsaw_build_docs/#options","title":"Options","text":"
      --catalog string         Path to the built test catalog file\n  -h, --help                   help for docs\n      --readme-file string     Name of the built docs file (default \"README.md\")\n      --test-dir stringArray   Directories containing test cases to run\n      --test-file string       Name of the test file (default \"chainsaw-test.yaml\")\n
"},{"location":"commands/chainsaw_build_docs/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion/","title":"Chainsaw completion","text":""},{"location":"commands/chainsaw_completion/#chainsaw-completion","title":"chainsaw completion","text":"

Generate the autocompletion script for the specified shell

"},{"location":"commands/chainsaw_completion/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for chainsaw for the specified shell. See each sub-command's help for details on how to use the generated script.

"},{"location":"commands/chainsaw_completion/#options","title":"Options","text":"
  -h, --help   help for completion\n
"},{"location":"commands/chainsaw_completion/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_bash/","title":"Chainsaw completion bash","text":""},{"location":"commands/chainsaw_completion_bash/#chainsaw-completion-bash","title":"chainsaw completion bash","text":"

Generate the autocompletion script for bash

"},{"location":"commands/chainsaw_completion_bash/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package. If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

source <(chainsaw completion bash)\n

To load completions for every new session, execute once:

"},{"location":"commands/chainsaw_completion_bash/#linux","title":"Linux:","text":"
chainsaw completion bash > /etc/bash_completion.d/chainsaw\n
"},{"location":"commands/chainsaw_completion_bash/#macos","title":"macOS:","text":"
chainsaw completion bash > $(brew --prefix)/etc/bash_completion.d/chainsaw\n

You will need to start a new shell for this setup to take effect.

chainsaw completion bash\n
"},{"location":"commands/chainsaw_completion_bash/#options","title":"Options","text":"
  -h, --help              help for bash\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_bash/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_fish/","title":"Chainsaw completion fish","text":""},{"location":"commands/chainsaw_completion_fish/#chainsaw-completion-fish","title":"chainsaw completion fish","text":"

Generate the autocompletion script for fish

"},{"location":"commands/chainsaw_completion_fish/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

chainsaw completion fish | source\n

To load completions for every new session, execute once:

chainsaw completion fish > ~/.config/fish/completions/chainsaw.fish\n

You will need to start a new shell for this setup to take effect.

chainsaw completion fish [flags]\n
"},{"location":"commands/chainsaw_completion_fish/#options","title":"Options","text":"
  -h, --help              help for fish\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_fish/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_powershell/","title":"Chainsaw completion powershell","text":""},{"location":"commands/chainsaw_completion_powershell/#chainsaw-completion-powershell","title":"chainsaw completion powershell","text":"

Generate the autocompletion script for powershell

"},{"location":"commands/chainsaw_completion_powershell/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for powershell.

To load completions in your current shell session:

chainsaw completion powershell | Out-String | Invoke-Expression\n

To load completions for every new session, add the output of the above command to your powershell profile.

chainsaw completion powershell [flags]\n
"},{"location":"commands/chainsaw_completion_powershell/#options","title":"Options","text":"
  -h, --help              help for powershell\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_powershell/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_zsh/","title":"Chainsaw completion zsh","text":""},{"location":"commands/chainsaw_completion_zsh/#chainsaw-completion-zsh","title":"chainsaw completion zsh","text":"

Generate the autocompletion script for zsh

"},{"location":"commands/chainsaw_completion_zsh/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:

echo \"autoload -U compinit; compinit\" >> ~/.zshrc\n

To load completions in your current shell session:

source <(chainsaw completion zsh)\n

To load completions for every new session, execute once:

"},{"location":"commands/chainsaw_completion_zsh/#linux","title":"Linux:","text":"
chainsaw completion zsh > \"${fpath[1]}/_chainsaw\"\n
"},{"location":"commands/chainsaw_completion_zsh/#macos","title":"macOS:","text":"
chainsaw completion zsh > $(brew --prefix)/share/zsh/site-functions/_chainsaw\n

You will need to start a new shell for this setup to take effect.

chainsaw completion zsh [flags]\n
"},{"location":"commands/chainsaw_completion_zsh/#options","title":"Options","text":"
  -h, --help              help for zsh\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_zsh/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_create/","title":"Chainsaw create","text":""},{"location":"commands/chainsaw_create/#chainsaw-create","title":"chainsaw create","text":"

Create Chainsaw resources

chainsaw create [flags]\n
"},{"location":"commands/chainsaw_create/#options","title":"Options","text":"
  -h, --help   help for create\n
"},{"location":"commands/chainsaw_create/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_create_test/","title":"Chainsaw create test","text":""},{"location":"commands/chainsaw_create_test/#chainsaw-create-test","title":"chainsaw create test","text":"

Create a Chainsaw test

chainsaw create test [flags]\n
"},{"location":"commands/chainsaw_create_test/#options","title":"Options","text":"
      --description   If set, adds description when applicable (default true)\n      --force         If set, existing test will be deleted if needed\n  -h, --help          help for test\n      --save          If set, created test will be saved\n
"},{"location":"commands/chainsaw_create_test/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_docs/","title":"Chainsaw docs","text":""},{"location":"commands/chainsaw_docs/#chainsaw-docs","title":"chainsaw docs","text":"

Generate reference documentation

chainsaw docs [flags]\n
"},{"location":"commands/chainsaw_docs/#options","title":"Options","text":"
      --autogenTag      Determines if the generated docs should contain a timestamp (default true)\n  -h, --help            help for docs\n  -o, --output string   Output path (default \".\")\n      --website         Website version\n
"},{"location":"commands/chainsaw_docs/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_export/","title":"Chainsaw export","text":""},{"location":"commands/chainsaw_export/#chainsaw-export","title":"chainsaw export","text":"

Export commands

chainsaw export [flags]\n
"},{"location":"commands/chainsaw_export/#options","title":"Options","text":"
  -h, --help   help for export\n
"},{"location":"commands/chainsaw_export/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_export_schemas/","title":"Chainsaw export schemas","text":""},{"location":"commands/chainsaw_export_schemas/#chainsaw-export-schemas","title":"chainsaw export schemas","text":"

Export JSON schemas

chainsaw export schemas [flags]\n
"},{"location":"commands/chainsaw_export_schemas/#options","title":"Options","text":"
  -h, --help   help for schemas\n
"},{"location":"commands/chainsaw_export_schemas/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_lint/","title":"Chainsaw lint","text":""},{"location":"commands/chainsaw_lint/#chainsaw-lint","title":"chainsaw lint","text":"

Lint a file or read from standard input

"},{"location":"commands/chainsaw_lint/#synopsis","title":"Synopsis","text":"

Use chainsaw lint to lint a specific file or read from standard input for either test or configuration.

chainsaw lint [test|configuration] [flags]\n
"},{"location":"commands/chainsaw_lint/#options","title":"Options","text":"
  -f, --file string   Specify the file to lint or '-' for standard input\n  -h, --help          help for lint\n
"},{"location":"commands/chainsaw_lint/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate/","title":"Chainsaw migrate","text":""},{"location":"commands/chainsaw_migrate/#chainsaw-migrate","title":"chainsaw migrate","text":"

Migrate resources to Chainsaw

chainsaw migrate [flags]\n
"},{"location":"commands/chainsaw_migrate/#options","title":"Options","text":"
  -h, --help   help for migrate\n
"},{"location":"commands/chainsaw_migrate/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl/","title":"Chainsaw migrate kuttl","text":""},{"location":"commands/chainsaw_migrate_kuttl/#chainsaw-migrate-kuttl","title":"chainsaw migrate kuttl","text":"

Migrate KUTTL resources to Chainsaw

chainsaw migrate kuttl [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl/#options","title":"Options","text":"
  -h, --help   help for kuttl\n
"},{"location":"commands/chainsaw_migrate_kuttl/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl_config/","title":"Chainsaw migrate kuttl config","text":""},{"location":"commands/chainsaw_migrate_kuttl_config/#chainsaw-migrate-kuttl-config","title":"chainsaw migrate kuttl config","text":"

Migrate KUTTL config to Chainsaw

chainsaw migrate kuttl config [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl_config/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for config\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_kuttl_config/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl_tests/","title":"Chainsaw migrate kuttl tests","text":""},{"location":"commands/chainsaw_migrate_kuttl_tests/#chainsaw-migrate-kuttl-tests","title":"chainsaw migrate kuttl tests","text":"

Migrate KUTTL tests to Chainsaw

chainsaw migrate kuttl tests [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl_tests/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for tests\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_kuttl_tests/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_tests/","title":"Chainsaw migrate tests","text":""},{"location":"commands/chainsaw_migrate_tests/#chainsaw-migrate-tests","title":"chainsaw migrate tests","text":"

Migrate test steps to test

chainsaw migrate tests [flags]\n
"},{"location":"commands/chainsaw_migrate_tests/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for tests\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_tests/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_test/","title":"Chainsaw test","text":""},{"location":"commands/chainsaw_test/#chainsaw-test","title":"chainsaw test","text":"

Run tests

chainsaw test [flags]... [test directories]...\n
"},{"location":"commands/chainsaw_test/#options","title":"Options","text":"
      --apply-timeout duration                    The apply timeout to use as default for configuration (default 5s)\n      --assert-timeout duration                   The assert timeout to use as default for configuration (default 30s)\n      --cleanup-delay duration                    Adds a delay between the time a test ends and the time cleanup starts\n      --cleanup-timeout duration                  The cleanup timeout to use as default for configuration (default 30s)\n      --config string                             Chainsaw configuration file\n      --delete-timeout duration                   The delete timeout to use as default for configuration (default 15s)\n      --error-timeout duration                    The error timeout to use as default for configuration (default 30s)\n      --exclude-test-regex string                 Regular expression to exclude tests\n      --exec-timeout duration                     The exec timeout to use as default for configuration (default 5s)\n      --fail-fast                                 Stop the test upon encountering the first failure\n      --force-termination-grace-period duration   If specified, overrides termination grace periods in applicable resources\n      --full-name                                 Use full test case folder path instead of folder name\n  -h, --help                                      help for test\n      --include-test-regex string                 Regular expression to include tests\n      --kube-as string                            Username to impersonate for the operation\n      --kube-as-group stringArray                 Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n      --kube-as-uid string                        UID to impersonate for the operation\n      --kube-certificate-authority string         Path to a cert file for the certificate authority\n      --kube-client-certificate string            Path to a client certificate file for TLS\n      --kube-client-key string                    Path to a client key file for TLS\n      --kube-cluster string                       The name of the kubeconfig cluster to use\n      --kube-context string                       The name of the kubeconfig context to use\n      --kube-disable-compression                  If true, opt-out of response compression for all requests to the server\n      --kube-insecure-skip-tls-verify             If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n  -n, --kube-namespace string                     If present, the namespace scope for this CLI request\n      --kube-password string                      Password for basic authentication to the API server\n      --kube-proxy-url string                     If provided, this URL will be used to connect via proxy\n      --kube-request-timeout string               The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n      --kube-server string                        The address and port of the Kubernetes API server\n      --kube-tls-server-name string               If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n      --kube-token string                         Bearer token for authentication to the API server\n      --kube-user string                          The name of the kubeconfig user to use\n      --kube-username string                      Username for basic authentication to the API server\n      --namespace string                          Namespace to use for tests\n      --no-cluster                                Runs without cluster\n      --no-color                                  Removes output colors\n      --parallel int                              The maximum number of tests to run at once\n      --repeat-count int                          Number of times to repeat each test (default 1)\n      --report-format string                      Test report format (JSON|XML|nil)\n      --report-name string                        The name of the report to create (default \"chainsaw-report\")\n      --selector strings                          Selector (label query) to filter on\n      --skip-delete                               If set, do not delete the resources after running the tests\n      --template                                  If set, resources will be considered for templating\n      --test-dir strings                          Directories containing test cases to run\n      --test-file string                          Name of the test file (default \"chainsaw-test.yaml\")\n      --values strings                            Values passed to the tests\n
"},{"location":"commands/chainsaw_test/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_version/","title":"Chainsaw version","text":""},{"location":"commands/chainsaw_version/#chainsaw-version","title":"chainsaw version","text":"

Print the version informations

chainsaw version [flags]\n
"},{"location":"commands/chainsaw_version/#options","title":"Options","text":"
  -h, --help   help for version\n
"},{"location":"commands/chainsaw_version/#see-also","title":"SEE ALSO","text":""},{"location":"configuration/","title":"Configuring Chainsaw","text":"

Chainsaw is a comprehensive tool designed to facilitate end-to-end testing in Kubernetes.

This documentation will focus on providing a breakdown of its configuration structure and how to use it.

Chainsaw can be configured in two different and complementary ways:

Precedence

If both are specified, command-line flags will take precedence over configuration coming from a configuration file.

"},{"location":"configuration/#specific-configuration-options","title":"Specific configuration options","text":"

Please pay attention to the configuration options below, they may or may not be relevant in your case but can be useful in certain cases:

"},{"location":"configuration/cleanup-delay/","title":"Delay before cleanup","text":"

At the end of each test, Chainsaw will delete the resources it created during the test.

When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update the internal state.

For this reason, Chainsaw provides the delayBeforeCleanup configuration option and the corresponding --delay-before-cleanup flag.

"},{"location":"configuration/cleanup-delay/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  delayBeforeCleanup: 5s\n  # ...\n
"},{"location":"configuration/cleanup-delay/#flag","title":"Flag","text":"
chainsaw test --delay-before-cleanup 5s ...\n
"},{"location":"configuration/file/","title":"Configuration file","text":"

Chainsaw is described as a Stronger tool for e2e testing.

With its versatile configuration options, you can customize the testing process to fit your needs.

Chainsaw prioritizes its configuration in the following order:

  1. User-specified configuration

    If you explicitly provide a configuration file using a command-line flag

  2. Default configuration file

    If no configuration is specified, Chainsaw will look for a default file named .chainsaw.yaml in the current working directory

  3. Internal default configuration

    In the absence of both the above, Chainsaw will use a default configuration file embedded in the Chainsaw binary

"},{"location":"configuration/file/#example","title":"Example","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  timeouts:\n    apply: 45s\n    assert: 20s\n    cleanup: 45s\n    delete: 25s\n    error: 10s\n    exec: 45s\n  skipDelete: false\n  failFast: true\n  parallel: 4\n  # ...\n
"},{"location":"configuration/file/#how-to-specify-a-configuration","title":"How to specify a configuration","text":"

To use a custom configuration file:

chainsaw test --config path/to/your/config.yaml\n

Defaults

If you don't specify any configuration, Chainsaw will look for the default configuration file .chainsaw.yaml in the current working directory.

If that file is not found, it will fall back to its internal default configuration.

"},{"location":"configuration/file/#default-configuration","title":"Default configuration","text":"

The default configuration below is used by Chainsaw when no configuration file was provided and the default file .chainsaw.yaml does not exist.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: default\nspec: {}\n
"},{"location":"configuration/file/#reference-documentation","title":"Reference documentation","text":"

See Configuration API reference for more details.

"},{"location":"configuration/flags/","title":"Command line flags","text":"

Even after a configuration file is loaded, you can override specific settings using command-line flags.

"},{"location":"configuration/flags/#example","title":"Example","text":"
chainsaw test                         \\\n    path/to/test/dir                    \\\n    --config path/to/your/config.yaml   \\\n    --assert-timeout 45s                \\\n    --skip-delete false                 \\\n    --fail-fast true                    \\\n    --parallel 4                        \\\n    ...\n

In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.

"},{"location":"configuration/flags/#usage","title":"Usage","text":"

The command below will run tests using the configuration from my-config.yaml, taking tests from /path/to/tests, and running a maximum of 10 tests simultaneously.

chainsaw test /path/to/tests --config my-config.yaml --parallel 10\n
"},{"location":"configuration/flags/#reference-documentation","title":"Reference documentation","text":"

See Chainsaw test command reference for more details.

"},{"location":"configuration/grace/","title":"Termination graceful period","text":"

Some Kubernetes resources can take time before being stopped. For example, deleting a Pod can take time if the underlying container doesn't quit quickly enough.

For this reason, Chainsaw provides the forceTerminationGracePeriod configuration option and the corresponding --force-termination-grace-period flag. If set, Chainsaw will override the terminationGracePeriodSeconds when working with the following resource kinds:

"},{"location":"configuration/grace/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  forceTerminationGracePeriod: 5s\n  # ...\n
"},{"location":"configuration/grace/#flag","title":"Flag","text":"
chainsaw test --force-termination-grace-period 5s ...\n
"},{"location":"configuration/no-cluster/","title":"Running without a cluster","text":"

Chainsaw can be run without connection to a Kubernetes cluster. In this case, chainsaw will not try to create an ephemeral namespace and all operations requiring a Kubernetes cluster will fail.

To run chainsaw in this mode pass the --no-cluster flag.

"},{"location":"configuration/no-cluster/#example","title":"Example","text":"
# run chainsaw without connection to a Kubernetes cluster\nchainsaw test --no-cluster\n
"},{"location":"configuration/reports/","title":"Reports","text":"

Chainsaw can generate JUnit reports in XML or JSON format.

To produce a test report, configure the report format and report name in the configuration or using CLI flags.

"},{"location":"configuration/reports/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  reportFormat: JSON\n  reportName: chainsaw-report.json\n  # ...\n
"},{"location":"configuration/reports/#flag","title":"Flag","text":"
chainsaw test --report-format JSON --report-name chainsaw-report.json ...\n
"},{"location":"configuration/selector/","title":"Label selectors","text":"

Chainsaw can filter the tests to run using label selectors.

You can pass label selectors using the --selector flag when invoking the chainsaw test command.

"},{"location":"configuration/selector/#example","title":"Example","text":"

Given the test below:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\n  labels:\n    foo: bar\nspec:\n  # ...\n

Invoking Chainsaw with the command below will take the test above into account:

chainsaw test --selector foo=bar\n
"},{"location":"configuration/templating/","title":"Resource templating","text":"

Experimental status

This is an experimental feature, and implementation could change slightly in the next versions.

Chainsaw can apply transformations to the resources before they are processed by the operation.

This is useful when a resource needs some runtime configuration.

Templating must be enabled at the configuration, test, step, or operation level for the templating process to kick in. Alternatively, templating can be enabled using the --template flag when invoking chainsaw from the command line.

Note

Unlike assertion trees, templating can only be specified in leave nodes of the YAML tree.

"},{"location":"configuration/templating/#example","title":"Example","text":"

The test below will create, assert, and delete a ConfigMap with a dynamic name configured at runtime using the $namespace binding.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: values\nspec:\n  steps:\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: ($namespace)\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: ($namespace)\n    - delete:\n        ref:\n          apiVersion: v1\n          kind: ConfigMap\n          name: ($namespace)\n
"},{"location":"configuration/timeouts/","title":"Timeouts","text":"

Timeouts in Chainsaw are specified per type of operation. This is required because the timeout varies greatly depending on the nature of an operation.

For example, applying a manifest in a cluster is expected to be reasonably fast, while validating a resource can be a long operation.

Chainsaw supports separately configuring the timeouts below:

Overriding timeouts

Each timeout can be overridden at the test level, test step level, or individual operation level.

Timeouts defined in the Configuration are used in operations when not overridden.

"},{"location":"configuration/timeouts/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  timeouts:\n    apply: 45s\n    assert: 20s\n    cleanup: 45s\n    delete: 25s\n    error: 10s\n    exec: 45s\n  # ...\n
"},{"location":"configuration/timeouts/#flag","title":"Flag","text":"
chainsaw test                     \\\n    --apply-timeout 45s             \\\n    --assert-timeout 45s            \\\n    --cleanup-timeout 45s           \\\n    --delete-timeout 45s            \\\n    --error-timeout 45s             \\\n    --exec-timeout 45s              \\\n    ...\n
"},{"location":"configuration/values/","title":"Passing data to tests","text":"

Chainsaw can pass arbitrary values when running tests using the --values flag. Values will be available to tests under the $values binding.

This is useful when a test needs to be configured externally.

"},{"location":"configuration/values/#example","title":"Example","text":"

The test below expects the $value.foo to be provided when chainsaw is invoked.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: values\nspec:\n  steps:\n  - try:\n    - assert:\n        resource:\n          ($values.foo): bar\n

Now you can invoke chainsaw like this:

# pass object { \"foo\": \"bar\"\u00a0} as values to the executed tests\n# `--values -` means values are read from standard input\necho \"foo: bar\" | chainsaw test --values -\n\n# read values from a file\nchainsaw test --values ./values.yaml\n\n# pass values using heredoc\nchainsaw test --values - <<EOF\nfoo: bar\nEOF\n
"},{"location":"examples/","title":"Setup","text":"

To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.

In these examples, we will use kind but feel free to use the tool of your choice.

Not a cluster management tool

We consider this is not the responsibility of Chainsaw to manage clusters.

There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.

"},{"location":"examples/#create-a-kind-cluster","title":"Create a kind cluster","text":"

Please refer to the kind install docs to install it locally.

Once kind creating a local cluster is as simple as running:

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n
"},{"location":"examples/#install-chainsaw","title":"Install Chainsaw","text":"

Refer to install docs to install Chainsaw.

"},{"location":"examples/array-assertions/","title":"Array assertions","text":"

This example demonstrates how to perform complex assertions on arrays.

"},{"location":"examples/array-assertions/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/array-assertions/#manifests","title":"Manifests","text":""},{"location":"examples/array-assertions/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: Pod\nmetadata:\n  name: example\nspec:\n  containers:\n  - name: container-1\n    image: nginx-1\n    env:\n    - name: ENV_1\n      value: value-1\n  - name: container-2\n    image: nginx-2\n    env:\n    - name: ENV_2\n      value: value-2\n  - name: container-3\n    image: nginx-3\n    env:\n    - name: ENV_3\n      value: value-3\n
"},{"location":"examples/array-assertions/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: Pod\nmetadata:\n  name: example\nspec:\n  # iterate over all containers having `name: container-1`\n  ~.(containers[?name == 'container-1']):\n    image: nginx-1\n  # iterate over all containers, bind `$index` to the element index\n  ~index.(containers):\n    image: (join('-', ['nginx', to_string($index + `1`)]))\n  # nested iteration\n  ~index2.(containers):\n    ~.(env):\n      name: (join('_', ['ENV', to_string($index2 + `1`)]))\n      value: (join('-', ['value', to_string($index2 + `1`)]))\n
"},{"location":"examples/array-assertions/#test","title":"Test","text":""},{"location":"examples/array-assertions/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: array-assertions\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/array-assertions/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/assertion-tree/","title":"Assertion trees","text":"

Test steps:

  1. Creates a Deployment with 2 replicas
  2. Asserts that the number of replicas is > 1 and that status.replicas == spec.replicas
"},{"location":"examples/assertion-tree/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/assertion-tree/#manifests","title":"Manifests","text":""},{"location":"examples/assertion-tree/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx-deployment\nspec:\n  selector:\n    matchLabels:\n      app: nginx\n  replicas: 2\n  template:\n    metadata:\n      labels:\n        app: nginx\n    spec:\n      containers:\n      - name: nginx\n        image: nginx\n
"},{"location":"examples/assertion-tree/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx-deployment\n(status.replicas == spec.replicas): true\nspec:\n  (replicas > `1`): true\n
"},{"location":"examples/assertion-tree/#test","title":"Test","text":""},{"location":"examples/assertion-tree/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: assertion-tree\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/assertion-tree/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/basic/","title":"Basic","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data
"},{"location":"examples/basic/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/basic/#manifests","title":"Manifests","text":""},{"location":"examples/basic/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/basic/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/basic/#test","title":"Test","text":""},{"location":"examples/basic/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/basic/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/inline/","title":"Inline resources","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data
"},{"location":"examples/inline/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/inline/#test","title":"Test","text":""},{"location":"examples/inline/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: inline-resources\nspec:\n  steps:\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: quick-start\n          data:\n            foo: bar\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: quick-start\n          data:\n            foo: bar\n
"},{"location":"examples/inline/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/labels-selector/","title":"Labels selector","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data

This test is similar to the basic example but Chainsaw is invoked with the --selector flag.

"},{"location":"examples/labels-selector/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/labels-selector/#manifests","title":"Manifests","text":""},{"location":"examples/labels-selector/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/labels-selector/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/labels-selector/#test","title":"Test","text":""},{"location":"examples/labels-selector/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: labels-selector\n  labels:\n    # test labels\n    test-suite: examples\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/labels-selector/#execute","title":"Execute","text":"
# invoke chainsaw with `--selector` to filter tests to run\nchainsaw test --selector test-suite=examples\n
"},{"location":"examples/non-resource-assertion/","title":"Non resource assertion","text":"

This test example demonstrates how to perform assertions not based on resources.

  1. Asserts that the number of nodes in the cluster is equal to 1
"},{"location":"examples/non-resource-assertion/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/non-resource-assertion/#manifests","title":"Manifests","text":""},{"location":"examples/non-resource-assertion/#assertionsyaml","title":"assertions.yaml","text":"
(length(x_k8s_list($client, 'v1', 'Node'))): 1\n
"},{"location":"examples/non-resource-assertion/#test","title":"Test","text":""},{"location":"examples/non-resource-assertion/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: non-resource-assertion\nspec:\n  steps:\n  - try:\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/non-resource-assertion/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/test-output/","title":"Testing command output","text":"

Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.

"},{"location":"examples/test-output/#checking-output-contains","title":"Checking Output Contains","text":"

One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.

kubectl version\n\nClient Version: v1.28.2\nKustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\nServer Version: v1.27.4+k3s1\n

Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl version\n        check:\n          # This check ensures that the string '1.28' is found\n          # in stdout or else fails\n          (contains($stdout, '1.28')): true\n

Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.

- script:\n    content: kubectl version\n    check:\n      # This check ensures that the string '1.25' is NOT found\n      # in stdout or else fails\n      (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"

In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl version\n        check:\n          # This check ensures the contents of stdout are exactly as shown.\n          # Any deviations will cause a failure.\n          ($stdout): |-\n            Client Version: v1.28.2\n            Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n            Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"

In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.

kubectl foo\n\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n        top\n

Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check bad kubectl command\n    try:\n    - script:\n        content: kubectl foo\n        check:\n          # This checks that the result of the content was an error.\n          ($error != null): true\n          # This check below ensures that the string 'top' is found in stderr or else fails\n          (contains($stderr, 'top')): true\n

Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl foo\n        check:\n          # This checks that the result of the content was an error.\n          ($error != null): true\n          # This checks that the output is exactly as intended.\n          ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"jp/functions/","title":"Functions","text":"

Experimental functions

Experimental functions are denoted by the x_ prefix.

These are functions that are subject to signature change in a future version.

"},{"location":"jp/functions/#built-in-functions","title":"built-in functions","text":"Name Signature abs abs(number) avg avg(array[number]) ceil ceil(number) contains contains(array|string, any) ends_with ends_with(string, string) find_first find_first(string, string, number, number) find_last find_last(string, string, number, number) floor floor(number) from_items from_items(array[array]) group_by group_by(array, expref) items items(object) join join(string, array[string]) keys keys(object) length length(string|array|object) lower lower(string) map map(expref, array) max max(array[number]|array[string]) max_by max_by(array, expref) merge merge(object) min min(array[number]|array[string]) min_by min_by(array, expref) not_null not_null(any) pad_left pad_left(string, number, string) pad_right pad_right(string, number, string) replace replace(string, string, string, number) reverse reverse(array|string) sort sort(array[string]|array[number]) sort_by sort_by(array, expref) split split(string, string, number) starts_with starts_with(string, string) sum sum(array[number]) to_array to_array(any) to_number to_number(any) to_string to_string(any) trim trim(string, string) trim_left trim_left(string, string) trim_right trim_right(string, string) type type(any) upper upper(string) values values(object) zip zip(array, array)"},{"location":"jp/functions/#kyverno-json-functions","title":"kyverno-json functions","text":"Name Signature at at(array, any) concat concat(string, string) json_parse json_parse(string) wildcard wildcard(string, string)"},{"location":"jp/functions/#kyverno-functions","title":"kyverno functions","text":"Name Signature compare compare(string, string) equal_fold equal_fold(string, string) replace replace(string, string, string, number) replace_all replace_all(string, string, string) to_upper to_upper(string) to_lower to_lower(string) trim trim(string, string) trim_prefix trim_prefix(string, string) split split(string, string) regex_replace_all regex_replace_all(string, string|number, string|number) regex_replace_all_literal regex_replace_all_literal(string, string|number, string|number) regex_match regex_match(string, string|number) pattern_match pattern_match(string, string|number) label_match label_match(object, object) to_boolean to_boolean(string) add add(any, any) sum sum(array) subtract subtract(any, any) multiply multiply(any, any) divide divide(any, any) modulo modulo(any, any) round round(number, number) base64_decode base64_decode(string) base64_encode base64_encode(string) time_since time_since(string, string, string) time_now time_now() time_now_utc time_now_utc() path_canonicalize path_canonicalize(string) truncate truncate(string, number) semver_compare semver_compare(string, string) parse_json parse_json(string) parse_yaml parse_yaml(string) lookup lookup(object|array, string|number) items items(object|array, string, string) object_from_lists object_from_lists(array, array) random random(string) x509_decode x509_decode(string) time_to_cron time_to_cron(string) time_add time_add(string, string) time_parse time_parse(string, string) time_utc time_utc(string) time_diff time_diff(string, string) time_before time_before(string, string) time_after time_after(string, string) time_between time_between(string, string, string) time_truncate time_truncate(string, string)"},{"location":"jp/functions/#chainsaw-functions","title":"chainsaw functions","text":"Name Signature env env(string) x_k8s_get x_k8s_get(any, string, string, string, string) x_k8s_list x_k8s_list(any, string, string, string) x_k8s_exists x_k8s_exists(any, string, string, string, string) x_k8s_resource_exists x_k8s_resource_exists(any, string, string)"},{"location":"more/crds/","title":"Working with CRDs","text":"

New Custom Resource Definitions are not immediately available for use in the Kubernetes API until the Kubernetes API has acknowledged them.

If a Custom Resource Definition is being defined inside of a test step, be sure to wait for the CustomResourceDefinition object to appear.

For example, given this Custom Resource Definition:

apiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\nmetadata:\n  name: mycrds.mycrd.k8s.io\nspec:\n  group: mycrd.k8s.io\n  version: v1alpha1\n  names:\n    kind: MyCRD\n    listKind: MyCRDList\n    plural: mycrds\n    singular: mycrd\n  scope: Namespaced\n

Create the following assert:

apiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\nmetadata:\n  name: mycrds.mycrd.k8s.io\nstatus:\n  acceptedNames:\n    kind: MyCRD\n    listKind: MyCRDList\n    plural: mycrds\n    singular: mycrd\n  storedVersions:\n  - v1alpha1\n

And then the CRD can be used in subsequent steps:

apiVersion: mycrd.k8s.io/v1alpha1\nkind: MyCRD\nmetadata:\n  name: my-crds\nspec:\n  test: test\n
"},{"location":"more/events/","title":"Working with events","text":"

Kubernetes events are regular Kubernetes objects and can be asserted on just like any other object:

apiVersion: v1\nkind: Event\nreason: Started\nsource:\n  component: kubelet\ninvolvedObject:\n  apiVersion: v1\n  kind: Pod\n  name: my-pod\n
"},{"location":"more/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"more/kuttl-migration/#overview","title":"Overview","text":"

The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/kuttl-migration/#examples","title":"Examples","text":""},{"location":"more/kuttl-migration/#migrate-tests","title":"Migrate tests","text":"

The command below will migrate KUTTL tests to Chainsaw and overwrite original files with converted ones.

chainsaw migrate kuttl tests path/to/kuttl/tests --save --cleanup\n

This will generate a chainsaw-test.yaml for every KUTTL test discovered.

"},{"location":"more/kuttl-migration/#migrate-configuration","title":"Migrate configuration","text":"

The command below will migrate a KUTTL test suite file to the corresponding Chainsaw Configuration.

chainsaw migrate kuttl config path/to/kuttl/testsuite --save --cleanup\n

This will generate a .chainsaw.yaml configuration file.

"},{"location":"more/lint/","title":"Lint tests","text":""},{"location":"more/lint/#overview","title":"Overview","text":"

Chainsaw comes with a lint command to detect ill-formated tests.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/lint/#usage","title":"Usage","text":"

To build the docs of a test, Chainsaw provides the chainsaw lint test -f path/to/chainsaw-test.yaml command.

chainsaw lint test -f - <<EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: assertion-tree\nspec:\n  steps:\n  - try:\n    - assert:\n        file: assert.yaml\nEOF\n
Processing input...\nThe document is valid\n
"},{"location":"more/test-docs/","title":"Building test docs","text":""},{"location":"more/test-docs/#overview","title":"Overview","text":"

Chainsaw makes it simple to build the documentation of your tests.

As test suites grow, it becomes important to document what a test does and how it is supposed to work.

Going through the implementation of a test to understand its purpose is not an efficient strategy.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/test-docs/#usage","title":"Usage","text":"

To build the docs of a test, Chainsaw provides the chainsaw build docs command.

chainsaw build docs --test-dir path/to/chainsaw/tests\n

This will automatically discover tests and document steps and operations in try, catch and finally statements.

"},{"location":"more/test-docs/#the-description-field","title":"The description field","text":"

Additionally, you can set the description field in:

Chainsaw will output them nicely in the built docs.

"},{"location":"more/test-docs/#example","title":"Example","text":"

See below for an example test and the corresponding built docs.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  description: This is a very simple test that creates a configmap and checks the content is as expected.\n  steps:\n  - description: This steps applies the configmap in the cluster and checks the configmap content.\n    try:\n    - description: Create the configmap.\n      apply:\n        file: configmap.yaml\n    - description: Check the configmap content.\n      assert:\n        file: configmap-assert.yaml\n
"},{"location":"more/test-docs/#test-basic","title":"Test: basic","text":"

This is a very simple test that creates a configmap and checks the content is as expected.

"},{"location":"more/test-docs/#steps","title":"Steps","text":"# Name Try Catch Finally 1 step-1 2 0 0"},{"location":"more/test-docs/#step-step-1","title":"Step: step-1","text":"

This step applies the configmap in the cluster and checks the configmap content.

"},{"location":"more/test-docs/#try","title":"Try","text":"# Operation Description 1 apply Create the configmap. 2 assert Check the configmap content."},{"location":"operations/","title":"Operations","text":"

While tests are made of test steps, test steps can be considered made of operations.

Every operation in a test step runs sequentially.

Only one action per operation

Every operation consists of a single action. While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.

The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed in case an operation consists of multiple actions. For this reason, operations consisting of multiple actions are disallowed.

"},{"location":"operations/#common-fields","title":"Common fields","text":"

All operations share some configuration fields.

Reference documentation

The full structure of the Operation is documented here.

"},{"location":"operations/#continueonerror","title":"ContinueOnError","text":"

Determines whether a test step should continue or not in case the operation is not successful.

Even if the test continues executing, it will still be reported as failed.

"},{"location":"operations/#available-operations","title":"Available operations","text":""},{"location":"operations/#non-resource-assertions","title":"Non-resource assertions","text":"

It is possible to evaluate assertions that do not depend on resources.

See Non-resource assertions for details.

"},{"location":"operations/#operation-checks","title":"Operation checks","text":"

Some operations support checking the operation execution result against specific expectations.

See Operation checks for use case details and supported operations.

"},{"location":"operations/#resource-templating","title":"Resource templating","text":"

Chainsaw can apply transformations to the resources before they are processed by the operation.

See Resource templating for use case details and supported operations.

"},{"location":"operations/apply/","title":"Apply","text":"

The apply operation lets you define resources that should be applied to the Kubernetes cluster during the test step.

These can be configurations, deployments, services, or any other Kubernetes resource.

Reference documentation

The full structure of the Apply is documented here.

"},{"location":"operations/apply/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using apply in a Test resource.

Using a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: my-configmap.yaml\n    # ...\n

Using file path expressions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: \"configs/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/step/configmap.yaml\n    # ...\n

Using an inline resource

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n    # ...\n
"},{"location":"operations/apply/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- apply:\n    file: my-configmap.yaml\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n

With check

# ...\n- apply:\n    resource:\n      apiVersion: v1\n      kind: ConfigMap\n      metadata:\n        name: chainsaw-quick-start\n      data:\n        foo: bar\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/assert/","title":"Assert","text":"

The assert operation allows you to specify conditions that should hold true for a successful test.

For example, after applying resources, you might want to ensure that a particular pod is running or a service is accessible.

Assertion trees

Assertions in Chainsaw are based on assertion trees.

Assertion trees are a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Reference documentation

The full structure of the Assert is documented here.

"},{"location":"operations/assert/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using assert in a Test resource.

Using a specific file for assertions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: ../resources/deployment-assert.yaml\n    # ...\n

Using file path expressions for assertions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: \"../assertions/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/resource/valid.yaml\n    # ...\n

Using an inline assertion tree

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: Deployment\n          metadata:\n            name: foo\n          spec:\n            (replicas > 3): true\n    # ...\n
"},{"location":"operations/check/","title":"Operation checks","text":"

Considering an operation success or failure is not always as simple as checking an error code.

To support those kind of use cases, some operations support an additional check field to evaluate the operation result against an assertion tree.

Assertion trees

Assertions in Chainsaw are based on assertion trees.

Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Checked model

Different operation have a different model passed through the assertion tree.

The object passed to the assertion tree is the output object of the operation. Additional data like error or standard logs are passed using bindings ($error, $stdout, $stderr)

"},{"location":"operations/check/#expect-vs-check","title":"Expect vs Check","text":"

While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply and create operations. Those operations can operate on files containing multiple manifests and every manifest can have a different result.

To support more granular checks we use the expect field that contains an array of Expectation. Every expectation is made of an optional match and a check statement.

This way it is possible to control the scope of a check.

Null match

If the match statement is null, the check statement applies to all manifests in the operation.

If no expectation matches a given manifest, the default expectation will be used, checking that no error occured.

"},{"location":"operations/check/#apply","title":"Apply","text":"

apply supports expect and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#command","title":"Command","text":"

command supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string $stdout The content of the standard console output (if any) at the end of the operation string $stderr The content of the standard console error output (if any) at the end of the operation string @ Always null"},{"location":"operations/check/#create","title":"Create","text":"

create supports expect and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#delete","title":"Delete","text":"

delete supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#script","title":"Script","text":"

script supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string $stdout The content of the standard console output (if any) at the end of the operation string $stderr The content of the standard console error output (if any) at the end of the operation string @ Always null"},{"location":"operations/command/","title":"Command","text":"

The command operation provides a means to execute a specific command during the test step.

Reference documentation

The full structure of the Command is documented here.

"},{"location":"operations/command/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using command in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - command:\n        entrypoint: echo\n        args:\n        - hello chainsaw\n    # ...\n
"},{"location":"operations/command/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- command:\n    entrypoint: echo\n    args:\n    - hello chainsaw\n    check:\n      # an error is expected, this will:\n      # - succeed if the operation failed\n      # - fail if the operation succeeded\n      ($error != null): true\n# ...\n
"},{"location":"operations/create/","title":"Create","text":"

The create operation lets you define resources that should be created in the Kubernetes cluster during the test step. These can be configurations, deployments, services, or any other Kubernetes resource.

Warning

If the resource to be created already exists in the cluster, the step will fail.

Reference documentation

The full structure of the Create is documented here.

"},{"location":"operations/create/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using create in a Test resource.

Using a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        file: my-configmap.yaml\n    # ...\n

Using file path expressions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: \"configs/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/resource/valid.yaml\n    # ...\n

Using an inline resource

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n    # ...\n
"},{"location":"operations/create/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- apply:\n    file: my-configmap.yaml\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n

With check

# ...\n- apply:\n    resource:\n      apiVersion: v1\n      kind: ConfigMap\n      metadata:\n        name: chainsaw-quick-start\n      data:\n        foo: bar\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/delete/","title":"Delete","text":"

The delete operation allows you to specify resources that should be deleted from the Kubernetes cluster before a particular test step is executed.

Reference documentation

The full structure of the Delete is documented here.

"},{"location":"operations/delete/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using delete in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - delete:\n        ref:\n          apiVersion: v1\n          kind: Pod\n          namespace: default\n          name: my-test-pod\n    # ...\n
"},{"location":"operations/delete/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- delete:\n    ref:\n      apiVersion: v1\n      kind: Pod\n      namespace: default\n      name: my-test-pod\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: Pod\n        metadata:\n          namespace: default\n          name: my-test-pod\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/error/","title":"Error","text":"

The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.

Assertion trees

Errors in Chainsaw are based on assertion trees.

Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Reference documentation

The full structure of the Error is documented here.

"},{"location":"operations/error/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using error in a Test resource.

Expecting an error from a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: ../resources/deployment-error.yaml\n    # ...\n

Using file path expressions to expect errors from multiple files

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: \"../errors/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: https://raw.githubusercontent.com/user/repo/branch/path/to/deployment-error.yaml\n    # ...\n

Using an inline assertion tree

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        resource:\n          apiVersion: v1\n          kind: Deployment\n          metadata:\n            name: foo\n          spec:\n            (replicas > 3): true\n    # ...\n
"},{"location":"operations/non-resource-assert/","title":"Non-resource assertions","text":"

Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.

"},{"location":"operations/non-resource-assert/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using assert in a Test resource.

Using a file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: non-resource-assertion\nspec:\n  steps:\n  - try:\n    - assert:\n        resource:\n          (x_k8s_list($client, 'v1', 'Node')):\n            (length(@)): 1\n    - error:\n        resource:\n          (x_k8s_list($client, 'v1', 'Node')):\n            (length(@)): 2\n
"},{"location":"operations/script/","title":"Script","text":"

The script operation provides a means to run a script during the test step.

Reference documentation

The full structure of the Script is documented here.

"},{"location":"operations/script/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using script in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - script:\n        content: |\n          echo \"hello chainsaw\"\n    # ...\n
"},{"location":"operations/script/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- script:\n    content: |\n      echo \"hello chainsaw\"\n    check:\n      # an error is expected, this will:\n      # - succeed if the operation failed\n      # - fail if the operation succeeded\n      ($error != null): true\n# ...\n
"},{"location":"operations/sleep/","title":"Sleep","text":"

The sleep operation provides a means to sleep for a configured duration.

Reference documentation

The full structure of the Sleep is documented here.

"},{"location":"operations/sleep/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using sleep in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - sleep:\n        duration: 30s\n    # ...\n
"},{"location":"operations/templating/","title":"Resource templating","text":"

Experimental status

This is an experimental feature, and implementation could change slightly in the next versions.

Chainsaw can apply transformations to the resources before they are processed by the operation.

This is useful when a resource needs some runtime configuration.

Templating must be enabled at the configuration, test, step, or operation level for the templating process to kick in. Alternatively, templating can be enabled using the --template flag when invoking chainsaw from the command line.

Note

Unlike assertion trees, templating can only be specified in leave nodes of the YAML tree.

"},{"location":"operations/templating/#supported-operations","title":"Supported operations","text":"

Resource templating is supported in the following operations:

"},{"location":"operations/templating/#assert-and-error","title":"Assert and Error","text":"

When templating assert or error operations, the content is already an assertion tree.

For this reason, only the elements used for looking up the resources to be processed by the operation will be considered for templating. That is, only apiVersion, kind, name, namespace and labels are considered for templating. Other fields are not, they are part of the assertion tree only.

assert and error example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - assert:\n      resource:\n        # apiVersion, kind, name, namespace and labels are considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        # other fields are not (they are part of the assertion tree)\n        data:\n          foo: ($namespace)\n  - error:\n      resource:\n        # apiVersion, kind, name, namespace and labels are considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        # other fields are not (they are part of the assertion tree)\n        data:\n          bar: ($namespace)\n
"},{"location":"operations/templating/#apply-and-create","title":"Apply and Create","text":"

When templating apply or create operations, the whole content is considered for templating.

apply and create example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - apply:\n      resource:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        data:\n          foo: ($namespace)\n  - create:\n      resource:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        data:\n          foo: ($namespace)\n
"},{"location":"operations/templating/#delete","title":"Delete","text":"

When templating delete operations, the whole content is considered for templating.

apply and create example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - delete:\n      ref:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        name: ($namespace)\n
"},{"location":"steps/","title":"What is a test step","text":"

A test step is made of three main components used to dermine the actions Chainsaw will perform when executing the step.

  1. The try statement (required)
  2. The catch statement (optional)
  3. The finally statement (optional)

Reference documentation

The full structure of the TestStep is documented here.

"},{"location":"steps/#test-step-lifecycle","title":"Test step lifecycle","text":"

Test step lifecycle

  1. The step starts executing operations in the try statement
  2. If an operation fails in the try statement
    1. If a catch statement is present, all operations and collectors are executed
  3. If a finally statement is present, all operations and collectors are executed
"},{"location":"steps/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    - apply:\n        file: path/to/apply.yaml\n    - assert:\n        file: path/to/assert.yaml\n    catch: []\n    finally: []\n
"},{"location":"steps/catch/","title":"Catch","text":"

A catch statement is also a sequence of operations or collectors.

Operations and collectors contained in a catch statement will be executed only if the step failed when executing the operations in the step's try statement.

Tip

All operations and collectors of a catch statement will be executed regardless of the success or failure of each of them.

"},{"location":"steps/catch/#operations","title":"Operations","text":"

A catch statement supports only the following operations:

"},{"location":"steps/catch/#collectors","title":"Collectors","text":"

A catch statement supports all collectors:

"},{"location":"steps/catch/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try: []\n    catch:\n    - command:\n        # ...\n    - script:\n        # ...\n    - events:\n        # ...\n    - podLogs:\n        # ...\n    - describe:\n        # ...\n    - sleep:\n        # ...\n    finally: []\n
"},{"location":"steps/finally/","title":"Finally","text":"

A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.

Tip

All operations and collectors of a finally statement will be executed regardless of the success or failure of each of them.

"},{"location":"steps/finally/#operations","title":"Operations","text":"

A finally statement supports only the following operations:

"},{"location":"steps/finally/#collectors","title":"Collectors","text":"

A finally statement supports all collectors:

"},{"location":"steps/finally/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try: []\n    catch: []\n    finally:\n    - command:\n        # ...\n    - script:\n        # ...\n    - events:\n        # ...\n    - podLogs:\n        # ...\n    - describe:\n        # ...\n    - sleep:\n        # ...\n
"},{"location":"steps/try/","title":"Try","text":"

A try statement is a sequence of operations executed in the same order they are declared. If an operation fails the entire step is considered failed.

The try statement is at the heart of a test step, it represents what the step is supposed to be about.

catch and finally statements should be viewed as complementary to the try statement.

Continue on error

By default a test step stops executing when an operation fails and the following operations are not executed.

This behaviour can be changed using the continueOnError field, if continueOnError is set to true the step will still be considered failed but execution will continue with the next operations.

"},{"location":"steps/try/#operations","title":"Operations","text":"

A try statement supports all operations:

"},{"location":"steps/try/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    - command:\n        # ...\n    - sleep:\n        # ...\n    - script:\n        # ...\n    - apply:\n        # ...\n    - assert:\n        # ...\n    - error:\n        # ...\n    - create:\n        # ...\n    catch: []\n    finally: []\n
"},{"location":"tests/","title":"What is a test","text":"

To put it simply, a test can be represented as an ordered sequence of test steps.

Test steps within a test are run sequentially: if any of the test steps fail, the entire test is considered failed.

A test step can consist in one or more operations:

"},{"location":"tests/#different-syntaxes","title":"Different syntaxes","text":"

Chainsaw supports two different test definition syntaxes. Each syntax has pros and cons, see the descriptions below for more details on advantages and pitfalls.

"},{"location":"tests/#manifests-based-syntax","title":"Manifests based syntax","text":"

This is the simplest and less verbose syntax.

You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.

While this syntax is extremly simple, not all operations are supported. For example, delete, command, script and sleep operations are not.

Another strong limitation is that it is not possible to specify additional configuration per test, step or operation.

Finally, this syntax relies heavily on file naming conventions, it can be error prone and makes it hard to reuse files across multiple tests.

"},{"location":"tests/#test-based-syntax","title":"Test based syntax","text":"

The Test based syntax is a more verbose and explicit syntax.

It does not rely on file naming conventions to determine test steps order and allows to easily reuse files accross multiple tests.

This syntax also comes with no limitations to provide additional configuration at the test, step or operation level.

Making a choice

Choosing one syntax over the other is not a trivial choice, every one will have its own preference and/or constraints.

It's usually easier to start with the manifests based syntax. However, as test suites grow and tests become more complex, it is often necessary to configure options on a per test, step or operation basis and the Test based syntax becomes necessary.

Fortunately Chainsaw has a command to automatically migrate from manifest based to Test based syntax.

"},{"location":"tests/#namespaced-resources","title":"Namespaced resources","text":"

Kubernetes organizes resources into two primary scopes: namespaced and cluster-scoped.

While namespaced resources belong to a specific namespace, cluster-scoped resources span across the entire Kubernetes cluster.

"},{"location":"tests/#purpose-of-the-namespacer-interface","title":"Purpose of the Namespacer Interface","text":"

The Namespacer interface ensures automated and consistent namespace assignment to Kubernetes resources.

"},{"location":"tests/#cleanup","title":"Cleanup","text":"

Unless configured differently, by default Chainsaw will automatically cleanup the resources it created after a test finishes. Cleanup happens in reverse order of creation (created last, cleaned up first).

Note that Chainsaw performs a blocking deletion, that is, it will wait the resource is actually not present anymore in the cluster before proceeding with the next resource cleanup.

This is important, especially when the controller being tested makes use of finalizers.

Overriding cleanup timeout

A global cleanup timeout can be defined at the configuration level or using command line flags.

It can also be overriden on a per test or per test step basis but not at the operation level.

"},{"location":"tests/#parallel-execution-of-tests","title":"Parallel Execution of Tests","text":"

While Chainsaw ensures that the steps within a test are executed sequentially, it is designed to run multiple tests in parallel to each other. This parallel execution helps in significantly reducing the overall time required to run an entire suite of tests, making the testing process more efficient, especially in scenarios with a large number of tests.

Parallel execution of tests

By default, Chainsaw will run tests in parallel.

This can be configured at the configuration level or using command line flags. However, individual tests can be configured to run concurrently by setting Concurrent: true in their TestSpec.

All non-concurrent tests are executed first, followed by the concurrent tests in parallel.

"},{"location":"tests/manifests-based/","title":"Manifests based syntax","text":"

This is the simplest and less verbose supported syntax.

You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.

While this syntax is simple, it doesn't support deletion operations and doesn't allow specifying additional configuration per test or step.

It also relies a lot on file naming conventions and makes it hard to reuse files across multiple tests.

"},{"location":"tests/manifests-based/#file-naming-convention","title":"File naming convention","text":"

Manifest files must follow the naming convention <step index>-<name|assert|errors>.yaml.

As an example 00-configmap.yaml, 01-assert.yaml and 02-errors.yaml can all be considered valid file names.

It's also perfectly valid to have multiple files for the same step. Let's imagine we have the following files 00-resources.yaml, 00-more-resources.yaml, 00-assert.yaml and 00-errors.yaml:

With the four files above, Chainsaw will assemble a test step made of the combination of all those files.

"},{"location":"tests/manifests-based/#manifests-loading-process","title":"Manifests loading process","text":"

The logic to determine the content of a step is always:

"},{"location":"tests/manifests-based/#example","title":"Example","text":""},{"location":"tests/manifests-based/#01-configmapyaml","title":"01-configmap.yaml","text":"

The manifest below contains a config map in a file called 01-configmap.yaml. Chainsaw will associate this manifest with an apply operation in step 01.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\n
"},{"location":"tests/manifests-based/#02-assertyaml","title":"02-assert.yaml","text":"

The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\n
"},{"location":"tests/manifests-based/#03-errorsyaml","title":"03-errors.yaml","text":"

The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  lorem: ipsum\n
"},{"location":"tests/manifests-based/#conclusion","title":"Conclusion","text":"

This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the configmap does not contain the lorem: ipsum data.

Assert and error statements are very similar to standard resource definitions, they share the same structure but can be partial. Chainsaw will process only the elements present in the statements against the actual resource definition.

"},{"location":"tests/test-based/","title":"Test based syntax","text":"

The Test based syntax is more verbose than the manifests based syntax but offers more flexibility and features:

"},{"location":"tests/test-based/#the-test-resource","title":"The Test resource","text":"

A Test resource, like any Kubernetes resource has an apiVersion, kind and metadata section.

It also comes with a spec section used to declaratively represent the steps of a test and other configuration elements belonging to the test being defined.

Reference documentation

The full structure of the Test resource is documented here.

"},{"location":"tests/test-based/#test-steps","title":"Test steps","text":"

A Test is mostly made of test steps. Test steps are detailed in a dedicated documentation.

"},{"location":"tests/test-based/#example","title":"Example","text":""},{"location":"tests/test-based/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"

The manifest below contains a Test in a file called chainsaw-test.yaml.

Chainsaw will load the Test and steps defined in its spec section.

The Test uses a custom timeouts for the whole test. Note that timeouts could have been overridden in specific steps if needed.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  # these timeouts are applied per operation\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step\n  # apply a configmap to the cluster\n  # the path to the configmap is relative to the folder\n  # containing the test, hence allow reusing manifests\n  # across multiple tests\n  - try:\n    - apply:\n        file: ../resources/configmap.yaml\n  # second step\n  # execute assert statements against existing resources\n  # in the cluster\n  - try:\n    - assert:\n        file: ../resources/configmap-assert.yaml\n  # third step\n  # execute error statements against existing resources\n  # in the cluster\n  - try:\n    - error:\n        file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#specifying-a-custom-test-file","title":"Specifying a Custom Test File","text":"

If you have your test defined in a different file, you can specify it when running Chainsaw:

chainsaw test . --test-file=<custom-test-file-name>.yaml\n
"},{"location":"tests/test-based/#raw-resource-support","title":"Raw Resource Support","text":"

Chainsaw now includes the raw resource feature, allowing direct specification of Kubernetes resources within the test definitions.

This feature offers a more streamlined approach for defining resources, especially useful for simpler test scenarios or for cases where resource definitions need to be reused or slightly modified across different tests.

"},{"location":"tests/test-based/#example-raw-resource","title":"Example Raw Resource","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step applies a configmap directly to the cluster\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n  # second step executes assert statements against existing resources\n  - try:\n    - assert:\n        file: ../resources/configmap-assert.yaml\n  # third step executes error statements against existing resources\n  - try:\n    - error:\n        file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#url-support-for-file-references","title":"URL Support for File References","text":"

Chainsaw has support for URLs in file references for assert, apply, error and similar operations.

This feature enhances the reach of Chainsaw by allowing users to reference files hosted on remote locations, such as GitHub raw URLs or other web URLs, directly within their test definitions.

"},{"location":"tests/test-based/#example-url-file-reference","title":"Example URL File Reference","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step\n  # apply a Kubernetes manifest hosted at a GitHub raw URL\n  - try:\n    - apply:\n        file: https://raw.githubusercontent.com/username/repo/branch/path/to/configmap.yaml\n  # second step\n  # execute assert statements against existing resources\n  # using a file hosted on another web URL\n  - try:\n    - assert:\n        file: https://example.com/path/to/configmap-assert.yaml\n  # third step\n  # execute error statements against existing resources\n  - try:\n    - error:\n        file: https://mywebsite.com/path/to/configmap-error.yaml\n
"},{"location":"blog/archive/2023/","title":"2023","text":""},{"location":"blog/category/writing-tests/","title":"Writing Tests","text":""},{"location":"blog/category/announcements/","title":"announcements","text":""}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"community/","title":"Community","text":"

Chainsaw has a growing community and we would definitely love to see you join and contribute.

Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.

Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters

"},{"location":"community/#slack-channel","title":"Slack channel","text":"

Join our slack channel #kyverno-chainsaw to meet with users, contributors and maintainers.

"},{"location":"community/#community-meetings","title":"Community Meetings","text":"

To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.

This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.

Weekly every Thursday at 2:00 PM UTC

"},{"location":"community/#roadmap","title":"RoadMap","text":"

For detailed information on our planned features and upcoming updates, please view our Roadmap.

"},{"location":"community/#contributing","title":"Contributing","text":"

Please read the contributing guide for details around:

  1. Code of Conduct
  2. Code Culture
  3. Details on how to contribute
"},{"location":"community/#adopters","title":"Adopters","text":"

If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces

"},{"location":"docs/","title":"Documentation","text":"

Starting with Chainsaw v0.0.6 we maintain separate docs per version:

"},{"location":"gh-action/","title":"GitHub action","text":"

A GitHub action is available to easily install Chainsaw in your workflows.

The GitHub action is available at kyverno/action-install-chainsaw or in the marketplace.

"},{"location":"gh-action/#usage","title":"Usage","text":"

This action currently supports GitHub-provided Linux, macOS and Windows runners (self-hosted runners may not work).

Add the following entry to your Github workflow YAML file:

uses: kyverno/action-install-chainsaw@v0.1.0\nwith:\n  release: v0.1.0 # optional\n

Example using a pinned version:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          release: v0.0.9\n      - name: Check install\n        run: chainsaw version\n

Example using the default version:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n      - name: Check install\n        run: chainsaw version\n

Example using cosign verification:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw\n    steps:\n      - name: Install Cosign\n        uses: sigstore/cosign-installer@v3.1.1\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          verify: true\n      - name: Check install\n        run: chainsaw version\n

If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.

Example of installing Chainsaw via go install:

jobs:\n  example:\n    runs-on: ubuntu-latest\n\n    permissions: {}\n\n    name: Install Chainsaw via go install\n    steps:\n      - name: Install go\n        uses: actions/setup-go@v4\n        with:\n          go-version: '1.21'\n      - name: Install Chainsaw\n        uses: kyverno/action-install-chainsaw@v0.1.0\n        with:\n          release: main\n      - name: Check install\n        run: chainsaw version\n
"},{"location":"gh-action/#optional-inputs","title":"Optional Inputs","text":"

The following optional inputs:

Input Description release chainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"install/","title":"Install","text":"

You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.

We also provide a GitHub action to easily install Chainsaw in your workflows.

"},{"location":"install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"install/#homebrew-tap","title":"Homebrew tap","text":"

add tap:

brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n

install chainsaw:

brew install kyverno/chainsaw/chainsaw\n

Don't forget to specify the tap name

Homebrew core already has a tool named chainsaw.

Be sure that you specify the tap name when installing to install the right tool.

"},{"location":"install/#manually","title":"Manually","text":"

Download the pre-compiled binaries for your system from the releases page and copy them to the desired location.

"},{"location":"install/#install-using-go-install","title":"Install using go install","text":"

You can install with go install with:

go install github.com/kyverno/chainsaw@latest\n
"},{"location":"install/#running-with-docker","title":"Running with Docker","text":"

Chainsaw is also available as a Docker image which you can pull and run:

docker pull ghcr.io/kyverno/chainsaw:<version>\n

Info

Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.

docker run --rm                             \\\n    -v ./testdata/e2e/:/chainsaw/           \\\n    -v ${HOME}/.kube/:/etc/kubeconfig/      \\\n    -e KUBECONFIG=/etc/kubeconfig/config    \\\n    --network=host                          \\\n    ghcr.io/kyverno/chainsaw:<version>      \\\n    test /chainsaw --config /chainsaw/config.yaml\n
"},{"location":"install/#compiling-from-sources","title":"Compiling from sources","text":"

clone:

git clone https://github.com/kyverno/chainsaw.git\n
build the binaries:

cd chainsaw\ngo mod tidy\nmake build\n

verify it works:

./chainsaw version\n
"},{"location":"install/#github-action","title":"GitHub action","text":"

A GitHub action is available to install Chainsaw in your workflows. See the GitHub action dedicated documentation.

"},{"location":"intro/","title":"Introduction","text":"

Chainsaw is a tool developed to run end to end tests in Kubernetes clusters.

It is meant to test Kubernetes operators work as expected by running a sequence of test steps for:

"},{"location":"intro/#why-we-made-it","title":"Why we made it ?","text":"

While developing Kyverno we need to run end to end tests to make sure our admission controller works as expected.

A typical Kyverno end to end test

Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:

  1. Create a policy
  2. Create a resource
  3. Check that Kyverno acted as expected
  4. Cleanup and move to the next test
"},{"location":"intro/#from-kuttl-to-chainsaw","title":"From KUTTL to Chainsaw","text":"

We started with another tool called KUTTL.

While KUTTL was a great tool to start with, we quickly identified some limitations and forked it to add the features we needed.

In the end we needed more flexibility than what KUTTL could offer and we started designing a new assertion model and at this point it was simpler to start a new tool from scratch than continuing making changes in our KUTTL fork.

The changes we were making was simply too large to have a chance to be incorporated upstream.

Chainsaw was born

"},{"location":"intro/#kuttl-compatibility","title":"KUTTL compatibility","text":"

Fortunately, Chainsaw assertion model is compatible with KUTTL and therefore migration to Chainsaw is relatively easy.

We provide chainsaw migrate kuttl config and chainsaw migrate kuttl tests commands to assist with the migration, see Migration from KUTTL for details.

"},{"location":"intro/#use-cases","title":"Use cases","text":"

Chainsaw is built with CI tools in mind - you only really need to download and execute it in your build script.

Installing it in your machine is entirely up to you, but still possible.

"},{"location":"json-schemas/","title":"JSON schemas","text":"

JSON schemas for Chainsaw resources are available to enable validation and autocompletion in your IDE:

"},{"location":"json-schemas/#vs-code","title":"VS code","text":"

In VS code, simply add a comment on top of your YAML resources.

"},{"location":"json-schemas/#test","title":"Test","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  steps:\n  - try:\n    - apply:\n        file: configmap.yaml\n    - assert:\n        file: configmap-assert.yaml\n
"},{"location":"json-schemas/#configuration","title":"Configuration","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/configuration-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: congiguration\nspec:\n  fullName: true\n  failFast: true\n  forceTerminationGracePeriod: 5s\n
"},{"location":"json-schemas/#applying-crds","title":"Applying CRDs","text":"

An alternative is to apply the Chainsaw CRDs in a kubernetes cluster and let the Kubernetes extension do the rest.

CRD definitions are provided in our GitHub repository.

The command below will apply the Chainsaw CRDs to the configured cluster:

kubectl apply -f https://raw.githubusercontent.com/kyverno/chainsaw/main/config/crds/chainsaw.kyverno.io_configurations.yaml\nkubectl apply -f https://raw.githubusercontent.com/kyverno/chainsaw/main/config/crds/chainsaw.kyverno.io_tests.yaml\n
"},{"location":"json-schemas/#exporting-schemas","title":"Exporting schemas","text":"

Chainsaw can export JSON schemas locally.

chainsaw export schemas <local path>\n

See Chainsaw export schemas reference for more details.

"},{"location":"quick-start/","title":"Quick start","text":"

To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.

In this Quick start we will use kind but feel free to use the tool of your choice.

Not a cluster management tool

We consider this is not the responsibility of Chainsaw to manage clusters.

There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.

"},{"location":"quick-start/#create-a-kind-cluster","title":"Create a kind cluster","text":"

Please refer to the kind install docs to install it locally.

Once kind creating a local cluster is as simple as running:

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n
"},{"location":"quick-start/#writing-tests","title":"Writing tests","text":"

A Chainsaw test is made of YAML files in a folder.

YAML files can contain raw manifests with a special file naming convention to identify the step operations. This is useful to create tests quickly but doesn't allow great flexibility.

Another option is to have a chainsaw-test.yaml file containing a Test resource. While more verbose, this offers full flexibility over the test and test steps configuration.

"},{"location":"quick-start/#quick-start_1","title":"Quick start","text":"

For this quick start, we will create a Test with one step and two operations:

  1. Create a ConfigMap from a manifest
  2. Verify the ConfigMap was created and contains the expected data
"},{"location":"quick-start/#create-the-test-folder","title":"Create the test folder","text":"
# create test folder\nmkdir chainsaw-quick-start\n\n# enter test folder\ncd chainsaw-quick-start\n
"},{"location":"quick-start/#create-a-configmap","title":"Create a ConfigMap","text":"
# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\nEOF\n
"},{"location":"quick-start/#create-the-test","title":"Create the test","text":"

By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.

# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: quick-start\nspec:\n  steps:\n  - try:\n    # first operation: create the config map\n    - apply:\n        # file is relative to the test folder\n        file: configmap.yaml\n    # second operation: verify the config map exists and contains the expected data\n    - assert:\n        # file is relative to the test folder\n        file: configmap.yaml\nEOF\n
"},{"location":"quick-start/#run-chainsaw","title":"Run Chainsaw","text":"

We finished writing our first test, now we can run Chainsaw to verify the test runs as expected:

chainsaw test\n\nVersion: (devel)\nLoading default configuration...\n- Using test file: chainsaw-test.yaml\n- TestDirs [.]\n- SkipDelete false\n- FailFast false\n- ReportFormat ''\n- ReportName 'chainsaw-report'\n- Namespace ''\n- FullName false\n- IncludeTestRegex ''\n- ExcludeTestRegex ''\n- ApplyTimeout 5s\n- AssertTimeout 30s\n- CleanupTimeout 30s\n- DeleteTimeout 15s\n- ErrorTimeout 30s\n- ExecTimeout 5s\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN   chainsaw\n=== PAUSE chainsaw\n=== CONT  chainsaw\n=== RUN   chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT  chainsaw/quick-start\n    | 10:44:26 | quick-start | @setup   | CREATE    | OK    | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:26 | quick-start | step-1   | TRY       | RUN   |\n    | 10:44:26 | quick-start | step-1   | APPLY     | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | CREATE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | APPLY     | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | ASSERT    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | ASSERT    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | step-1   | TRY       | DONE  |\n    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | DONE  | v1/ConfigMap @ chainsaw-immense-jay/chainsaw-quick-start\n    | 10:44:26 | quick-start | @cleanup | DELETE    | RUN   | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:26 | quick-start | @cleanup | DELETE    | OK    | v1/Namespace @ chainsaw-immense-jay\n    | 10:44:31 | quick-start | @cleanup | DELETE    | DONE  | v1/Namespace @ chainsaw-immense-jay\n--- PASS: chainsaw (0.00s)\n    --- PASS: chainsaw/quick-start (5.25s)\nPASS\nTests Summary...\n- Passed  tests 1\n- Failed  tests 0\n- Skipped tests 0\nDone.\n
"},{"location":"resources/","title":"Resources","text":"

Resources, blog posts and videos talking about Chainsaw:

"},{"location":"writing-tests/","title":"What is Chainsaw","text":""},{"location":"writing-tests/#overview","title":"Overview","text":"

Chainsaw provides a declarative approach to testing production-grade Kubernetes operators and controllers.

It provides a way to inject an operator (subject under test) during the setup and allows tests to be standard YAML files.

In Chainsaw everything is YAML. We use YAML to manipulate resources, define cluster state assertions, describe commands to run, etc... Everything is declarative.

"},{"location":"writing-tests/#motivation","title":"Motivation","text":"

Testing Kubernetes operators is not easy.

At Kyverno we are building complex controllers and admission controllers that need to be deeply end to end tested.

Advantages of a declarative approach

It made sense to be able to declaratively create end to end tests for a couple of reasons:

"},{"location":"writing-tests/#when-would-you-use-chainsaw","title":"When would you use Chainsaw","text":"

The testing eco-system is vast and includes at a minimum low level unit tests, integration tests and end-to-end testing.

Chainsaw is built to support some kubernetes integration test scenarios and is most valuable as an end to end testing tool.

Chainsaw is great for

"},{"location":"apis/chainsaw.v1alpha1/","title":"chainsaw (v1alpha1)","text":"

Package v1alpha1 contains API Schema definitions for the v1alpha1 API group.

"},{"location":"apis/chainsaw.v1alpha1/#resource-types","title":"Resource Types","text":""},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Configuration","title":"Configuration","text":"

Configuration is the resource that contains the configuration used to run tests.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string Configuration metadata meta/v1.ObjectMeta

Standard object's metadata.

spec ConfigurationSpec

Configuration spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Test","title":"Test","text":"

Test is the resource that contains aa test definition.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string Test metadata meta/v1.ObjectMeta

Standard object's metadata.

spec TestSpec

Test spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestStep","title":"TestStep","text":"

TestStep is the resource that contains the testStep used to run tests.

Field Type Required Inline Description apiVersion string chainsaw.kyverno.io/v1alpha1 kind string TestStep metadata meta/v1.ObjectMeta

Standard object's metadata.

spec TestStepSpec

TestStep spec.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Apply","title":"Apply","text":"

Appears in:

Apply represents a set of configurations or resources that should be applied during testing.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the resources to be applied.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Assert","title":"Assert","text":"

Appears in:

Assert represents a test condition that is expected to hold true during the testing process.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrCheck FileRefOrCheck

FileRefOrAssert provides a reference to the assertion.

template bool

Template determines whether resources should be considered for templating.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Catch","title":"Catch","text":"

Appears in:

Catch defines actions to be executed on failure.

Field Type Required Inline Description description string

Description contains a description of the operation.

podLogs PodLogs

PodLogs determines the pod logs collector to execute.

events Events

Events determines the events collector to execute.

describe Describe

Describe determines the resource describe collector to execute.

get Get

Get determines the resource get collector to execute.

delete Delete

Delete represents a deletion operation.

command Command

Command defines a command to run.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Cluster","title":"Cluster","text":"

Appears in:

Field Type Required Inline Description kubeconfig string

Kubeconfig is the path to the referenced file.

context string

Context is the name of the context to use.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Command","title":"Command","text":"

Appears in:

Command describes a command to run as a part of a test step.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

entrypoint string

Entrypoint is the command entry point to run.

args []string

Args is the command arguments.

skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.

check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ConfigurationSpec","title":"ConfigurationSpec","text":"

Appears in:

ConfigurationSpec contains the configuration used to run tests.

Field Type Required Inline Description timeouts Timeouts

Global timeouts configuration. Applies to all tests/test steps if not overridden.

skipDelete bool

If set, do not delete the resources after running the tests (implies SkipClusterDelete).

template bool

Template determines whether resources should be considered for templating.

failFast bool

FailFast determines whether the test should stop upon encountering the first failure.

parallel int

The maximum number of tests to run at once.

reportFormat ReportFormatType

ReportFormat determines test report format (JSON reportName string

ReportName defines the name of report to create. It defaults to \"chainsaw-report\".

namespace string

Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.

namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.

fullName bool

FullName makes use of the full test case folder path instead of the folder name.

excludeTestRegex string

ExcludeTestRegex is used to exclude tests based on a regular expression.

includeTestRegex string

IncludeTestRegex is used to include tests based on a regular expression.

repeatCount int

RepeatCount indicates how many times the tests should be executed.

testFile string

TestFile is the name of the file containing the test to run.

forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.

delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

clusters map[string]Cluster

Clusters holds a registry to clusters to support multi-cluster tests.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Create","title":"Create","text":"

Appears in:

Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the file containing the resources to be created.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Delete","title":"Delete","text":"

Appears in:

Delete is a reference to an object that should be deleted

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

template bool

Template determines whether resources should be considered for templating.

ref ObjectReference

ObjectReference determines objects to be deleted.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Describe","title":"Describe","text":"

Appears in:

Describe defines how to describe resources.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

resource string

Resource type.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

showEvents bool

Show Events indicates whether to include related events.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Error","title":"Error","text":"

Appears in:

Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrCheck FileRefOrCheck

FileRefOrAssert provides a reference to the expected error.

template bool

Template determines whether resources should be considered for templating.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Events","title":"Events","text":"

Appears in:

Events defines how to collect events.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

format Format

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Expectation","title":"Expectation","text":"

Appears in:

Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.

Field Type Required Inline Description match policy/v1alpha1.Any

Match defines the matching statement.

check policy/v1alpha1.Any

Check defines the verification statement.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRef","title":"FileRef","text":"

Appears in:

FileRef represents a file reference.

Field Type Required Inline Description file string

File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRefOrCheck","title":"FileRefOrCheck","text":"

Appears in:

FileRefOrCheck represents a file reference or resource.

Field Type Required Inline Description FileRef FileRef

FileRef provides a reference to the file containing the resources to be applied.

resource policy/v1alpha1.Any

Check provides a check used in assertions.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-FileRefOrResource","title":"FileRefOrResource","text":"

Appears in:

FileRefOrResource represents a file reference or resource.

Field Type Required Inline Description FileRef FileRef

FileRef provides a reference to the file containing the resources to be applied.

resource meta/v1/unstructured.Unstructured

Resource provides a resource to be applied.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Finally","title":"Finally","text":"

Appears in:

Finally defines actions to be executed at the end of a test.

Field Type Required Inline Description description string

Description contains a description of the operation.

podLogs PodLogs

PodLogs determines the pod logs collector to execute.

events Events

Events determines the events collector to execute.

describe Describe

Describe determines the resource describe collector to execute.

get Get

Get determines the resource get collector to execute.

delete Delete

Delete represents a deletion operation.

command Command

Command defines a command to run.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Format","title":"Format","text":"

(Alias of string)

Appears in:

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Get","title":"Get","text":"

Appears in:

Get defines how to get resources.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

resource string

Resource type.

ObjectLabelsSelector ObjectLabelsSelector

ObjectLabelsSelector determines the selection process of referenced objects.

format Format

Format determines the output format (json or yaml).

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectLabelsSelector","title":"ObjectLabelsSelector","text":"

Appears in:

ObjectLabelsSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use selector.

Field Type Required Inline Description namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

selector string

Selector defines labels selector.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectReference","title":"ObjectReference","text":"

Appears in:

ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description ObjectSelector ObjectSelector

ObjectSelector determines the selection process of referenced objects.

apiVersion string

API version of the referent.

kind string

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectSelector","title":"ObjectSelector","text":"

Appears in:

ObjectSelector represents a strategy to select objects. For a single object name and namespace are used to identify the object. For multiple objects use labels.

Field Type Required Inline Description namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

labels map[string]string

Label selector to match objects to delete

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Operation","title":"Operation","text":"

Appears in:

Operation defines a single operation, only one action is permitted for a given operation.

Field Type Required Inline Description description string

Description contains a description of the operation.

continueOnError bool

ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.

apply Apply

Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.

assert Assert

Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.

command Command

Command defines a command to run.

create Create

Create represents a creation operation.

delete Delete

Delete represents a deletion operation.

error Error

Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.

patch Patch

Patch represents a patch operation.

script Script

Script defines a script to run.

sleep Sleep

Sleep defines zzzz.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Patch","title":"Patch","text":"

Appears in:

Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

FileRefOrResource FileRefOrResource

FileRefOrResource provides a reference to the file containing the resources to be patched.

template bool

Template determines whether resources should be considered for templating.

dryRun bool

DryRun determines whether the file should be applied in dry run mode.

expect []Expectation

Expect defines a list of matched checks to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-PodLogs","title":"PodLogs","text":"

Appears in:

PodLogs defines how to collect pod logs.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

namespace string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

name string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

selector string

Selector defines labels selector.

container string

Container in pod to get logs from else --all-containers is used.

tail int

Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ReportFormatType","title":"ReportFormatType","text":"

(Alias of string)

Appears in:

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Script","title":"Script","text":"

Appears in:

Script describes a script to run as a part of a test step.

Field Type Required Inline Description timeout meta/v1.Duration

Timeout for the operation. Overrides the global timeout set in the Configuration.

content string

Content defines a shell script (run with \"sh -c ...\").

skipLogOutput bool

SkipLogOutput removes the output from the command. Useful for sensitive logs or to reduce noise.

check policy/v1alpha1.Any

Check is an assertion tree to validate the operation outcome.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Sleep","title":"Sleep","text":"

Appears in:

Sleep represents a duration while nothing happens.

Field Type Required Inline Description duration meta/v1.Duration

Duration is the delay used for sleeping.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestSpec","title":"TestSpec","text":"

Appears in:

TestSpec contains the test spec.

Field Type Required Inline Description description string

Description contains a description of the test.

timeouts Timeouts

Timeouts for the test. Overrides the global timeouts set in the Configuration on a per operation basis.

skip bool

Skip determines whether the test should skipped.

concurrent bool

Concurrent determines whether the test should run concurrently with other tests.

skipDelete bool

SkipDelete determines whether the resources created by the test should be deleted after the test is executed.

template bool

Template determines whether resources should be considered for templating.

namespace string

Namespace determines whether the test should run in a random ephemeral namespace or not.

namespaceTemplate policy/v1alpha1.Any

NamespaceTemplate defines a template to create the test namespace.

steps []TestSpecStep

Steps defining the test.

forceTerminationGracePeriod meta/v1.Duration

ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.

delayBeforeCleanup meta/v1.Duration

DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestSpecStep","title":"TestSpecStep","text":"

Appears in:

TestSpecStep contains the test step definition used in a test spec.

Field Type Required Inline Description name string

Name of the step.

TestStepSpec TestStepSpec

TestStepSpec of the step.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-TestStepSpec","title":"TestStepSpec","text":"

Appears in:

TestStepSpec defines the desired state and behavior for each test step.

Field Type Required Inline Description description string

Description contains a description of the test step.

timeouts Timeouts

Timeouts for the test step. Overrides the global timeouts set in the Configuration and the timeouts eventually set in the Test.

skipDelete bool

SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.

template bool

Template determines whether resources should be considered for templating.

try []Operation

Try defines what the step will try to execute.

catch []Catch

Catch defines what the step will execute when an error happens.

finally []Finally

Finally defines what the step will execute after the step is terminated.

"},{"location":"apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Timeouts","title":"Timeouts","text":"

Appears in:

Timeouts contains timeouts per operation.

Field Type Required Inline Description apply meta/v1.Duration

Apply defines the timeout for the apply operation

assert meta/v1.Duration

Assert defines the timeout for the assert operation

cleanup meta/v1.Duration

Cleanup defines the timeout for the cleanup operation

delete meta/v1.Duration

Delete defines the timeout for the delete operation

error meta/v1.Duration

Error defines the timeout for the error operation

exec meta/v1.Duration

Exec defines the timeout for exec operations

"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/","title":"How to Perform Efficient E2E Testing with Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#introduction","title":"Introduction","text":"

Chainsaw is a powerful tool that enables efficient end-to-end (E2E) testing, making it an essential asset for software developers. Built to be stronger and more cohesive than KUTTL, Chainsaw offers a superior suite of features for Kubernetes (k8s) operators and objects. In this blog post, we will explore how to leverage Chainsaw for efficient E2E testing and enhance your overall testing workflow. Let's dive in!

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#table-of-contents","title":"Table of Contents","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-1-installing-chainsaw","title":"Section 1: Installing Chainsaw","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#11-downloading-chainsaw","title":"1.1 Downloading Chainsaw","text":"

To get started with Chainsaw, you'll first need to download the appropriate release for your operating system and architecture. The releases are available on the official GitHub page at the following link: Release

Here's a breakdown of the available releases:

Each release also comes with associated .pem, .sbom, .sbom.pem, and .sig files. These are used for verification and security purposes. If you're interested in verifying the integrity of your download, you can use these files. However, for the purpose of this guide, we'll focus on the main .tar.gz files.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#12-installing-chainsaw","title":"1.2 Installing Chainsaw","text":"

Once you've downloaded the appropriate .tar.gz file for your system, you can proceed with the installation. Here's a step-by-step guide:

tar -xzf chainsaw_<your_version_and_architecture>.tar.gz\n
sudo mv chainsaw /usr/local/bin/\n
chainsaw --version\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-2-writing-configurations","title":"Section 2: Writing Configurations","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#21-understanding-chainsaw-configurations","title":"2.1 Understanding Chainsaw Configurations","text":"

Chainsaw is a robust tool for end-to-end (e2e) testing in Kubernetes environments. It offers a flexible configuration system that allows users to define their testing parameters and conditions. The configuration can be set up in two primary ways:

Chainsaw follows a hierarchy in loading its configurations:

  1. User-specified Configuration: If you provide a configuration file explicitly using a command-line flag.
  2. Default Configuration File: If no configuration is specified, Chainsaw will search for a default file named .chainsaw.yaml in the current directory.
  3. Internal Default Configuration: If neither of the above is available, Chainsaw will resort to a default configuration embedded within its binary.

To specify a custom configuration, use the following command:

chainsaw test --config path/to/your/config.yaml\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#22-writing-chainsaw-configurations","title":"2.2 Writing Chainsaw Configurations","text":"

When writing a configuration for Chainsaw, you'll be working with a YAML file. Here's a basic example of what this might look like:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  timeout: 45s\n  skipDelete: false\n  failFast: true\n  parallel: 4\n  # ... other configurations\n

The full structure of the configuration file, including all possible fields and their descriptions, can be found in the Configuration API reference.

If you don't provide a custom configuration, Chainsaw will use its default:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: default\nspec: {}\n

You can also override specific configurations using command-line flags, even after loading a configuration file. For instance:

chainsaw test --config path/to/your/config.yaml --timeout 45s --parallel 4\n

In this example, even if the configuration file specifies different values for timeout and parallel, the command-line flags will take precedence.

For a comprehensive list of all supported flags and their descriptions, refer to the Chainsaw test command reference.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-3-creating-tests","title":"Section 3: Creating Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#31-understanding-chainsaw-tests","title":"3.1 Understanding Chainsaw Tests","text":"

In Chainsaw, a test is essentially an ordered sequence of test steps. These steps are executed sequentially, and if any step fails, the entire test is considered failed. Each test step can consist of multiple operations, such as creating, updating, or deleting resources in a Kubernetes cluster, or asserting that certain conditions are met.

Chainsaw supports three primary test definition mechanisms:

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#32-creating-chainsaw-tests","title":"3.2 Creating Chainsaw Tests","text":"

Creating tests in Chainsaw requires a clear understanding of the test definition mechanisms. Here's a detailed guide for each syntax:

  1. Using Manifests based syntax

File Naming: Files should follow the convention <step index>-<name|assert|error>.yaml. For instance, 00-configmap.yaml for resource creation, 01-assert.yaml for assertions, and 02-error.yaml for expected errors.

Example:

# 00-configmap.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-example\ndata:\n  key: value\n

  1. Using TestSteps based syntax

    Define a TestStep: Create a YAML file that defines a TestStep resource. This resource should specify operations like Apply, Assert, Delete, and Error.

Example:

# 01-test-step.yaml\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: TestStep\nmetadata:\n  name: apply-configmap-step\nspec:\n  apply:\n  - file: /resources/configmap.yaml\n

You can combine TestStep resources with raw Kubernetes manifests. For instance, a TestStep might apply a resource, while a separate manifest file makes assertions about that resource.

Example:

# 02-assert.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-example\ndata:\n  key: value\n
  1. Using Test based syntax

Define a Test: Start by creating a YAML file that defines a Test resource. This resource will have a spec section that outlines the entire test.

Example:

# chainsaw-test.yaml\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: full-test-example\nspec:\n  timeout: 10s\n  steps:\n  - spec:\n      apply:\n      - file: /resources/configmap.yaml\n  - spec:\n      assert:\n      - file: /resources/configmap-assert.yaml\n

Chainsaw processes the test by executing each step in sequence, ensuring that your Kubernetes environment meets the defined conditions.

"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#section-4-running-tests","title":"Section 4: Running Tests","text":""},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#41-preparing-for-test-execution","title":"4.1 Preparing for Test Execution","text":"

Before executing your tests, ensure that:

  1. Kubernetes Cluster: You have access to a Kubernetes cluster where the tests will run. This could be a local cluster (like Minikube or kind) or a remote one.
  2. Kubeconfig: Your kubeconfig is correctly set up to point to the desired cluster. Chainsaw will use the current context from your kubeconfig to interact with the cluster.
  3. Test Files: All your test files, whether they are written using Manifests based, TestSteps based, or Test based syntax, are organized and accessible.
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#42-executing-tests-in-chainsaw","title":"4.2 Executing Tests in Chainsaw","text":"

Once you're set up, running tests in Chainsaw is straightforward:

  1. Navigate to the Test Directory:
cd path/to/your/test/directory\n
  1. Run the Tests:
chainsaw test\n

This command will execute all tests in the current directory.

  1. View the Results: Chainsaw will display the results in the terminal. It provides a summary of passed, failed, and skipped tests. For any failed tests, Chainsaw will display detailed error messages to help diagnose the issue.

The output will resemble:

Loading default configuration...\n...\nRunning tests...\n=== RUN   quick-start\n...\n--- PASS: quick-start (5.22s)\nPASS\nTests Summary...\n- Passed  tests: 3\n- Failed  tests: 1\n- Skipped tests: 2\nDone.\n
"},{"location":"blog/How-to-Perform-Efficient-E2E-Testing-with-Chainsaw/#conclusion","title":"Conclusion","text":"

Chainsaw provides a robust and flexible framework for end-to-end testing in Kubernetes environments. By understanding its configuration and test definition mechanisms, developers can create comprehensive test suites that ensure their Kubernetes applications and configurations are functioning as expected.

"},{"location":"blog/first-test/","title":"First working test !","text":"

First test run !

The quick start test is passing 

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n\n# create test folder\nmkdir quick-start\n\n# enter test folder\ncd quick-start\n\n# create a ConfigMap\ncat > configmap.yaml << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\nEOF\n\n# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: quick-start\nspec:\n  steps:\n  # first step applies the config map\n  - apply:\n    # file is relative to the test folder\n    - file: configmap.yaml\n  # second step verifies the config map exists and contains the expected data\n  - assert:\n    # file is relative to the test folder\n    - file: configmap.yaml\nEOF\n\n../chainsaw test --test-dir .\n\nRunning without configuration\nLoading tests...\n- quick-start (.)\nRunning tests...\n=== RUN   chainsaw\n=== RUN   chainsaw/quick-start\n=== PAUSE chainsaw/quick-start\n=== CONT  chainsaw/quick-start\n    runner.go:35: step-1\n    runner.go:35: apply chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n    runner.go:35: step-2\n    runner.go:35: assert chainsaw-polite-chamois/chainsaw-quick-start (v1/ConfigMap)\n    runner.go:68: cleanup namespace: chainsaw-polite-chamois\n--- PASS: chainsaw (0.00s)\n    --- PASS: chainsaw/quick-start (5.09s)\nPASS\nDone.\n

Impressive !

"},{"location":"blog/wip/","title":"It's happening - Finally !","text":"

Hello everyone!

We finally started writing our own testing tool !

Let's join forces and make better, stronger Open Source and Community Driven tool.

"},{"location":"blog/wip/#why-we-made-it","title":"Why we made it ?","text":"

While developing Kyverno we needed to run end to end tests to make sure our admission controller worked as expected.

Kyverno can validate, mutate and generate resources based on policies installed in a cluster and a typical test is:

  1. Create a policy
  2. Create a resource
  3. Check that Kyverno acted as expected
  4. Cleanup and move to the next test

We started with another tool called KUTTL. While KUTTL is great we identified some limitations and forked the tool to add the features we needed.

At some point we needed more flexibility than what KUTTL offered and we designed a new assertion model.

This was simpler to start from scratch than continuing making changes in our KUTTL fork.

"},{"location":"blog/wip/#whats-next","title":"What's next ?","text":"

This is still WIP and needs a lot of work before we can consider it ready but things are moving fast.

We would love to build a community driven tool and welcome all contributors.

Feel free to fork this repository and start submitting pull requests

"},{"location":"collectors/","title":"Collectors","text":""},{"location":"collectors/#purpose","title":"Purpose","text":"

The purpose of collectors is to collect certain information about the outcome of a step should it fail (in the case of catch) or at the end of the step (in the case of finally).

The ultimate goal of collectors is to gather information about the failure of a step and therefore help understand what caused it to fail.

A test step can have an arbitrary number of collectors.

"},{"location":"collectors/#available-collectors","title":"Available collectors","text":""},{"location":"collectors/describe/","title":"Describe","text":"

Describing resources present in the cluster can help understand what happened and troubleshoot test failures.

"},{"location":"collectors/describe/#configuration","title":"Configuration","text":"

The full structure of the Describe resource is documented here.

"},{"location":"collectors/describe/#describe-pods","title":"Describe pods","text":"

If a name is specified, Chainsaw will describe resources that have a name starting with the provided name in the test namespace (or in the cluster if it is a clustered-level resource).

Describe pods in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        name: my-pod\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        name: my-pod\n    # ...\n

If a namespace is specified, Chainsaw will describe resources in the specified namespace.

Describe pods in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        namespace: foo\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        namespace: foo\n    # ...\n
"},{"location":"collectors/describe/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the resources to be described.

Describe pods using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        selector: app=my-app\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will describe resources in the specified namespace.

Describe pods using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/describe/#show-events","title":"Show events","text":"

The showEvents field can be used to enable or disable showing events when describing resources.

Default

By default, showEventsis true.

Do not show events

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - describe:\n        resource: pods\n        namespace: foo\n        showEvents: false\n    # ...\n    finally:\n    - describe:\n        resource: pods\n        namespace: foo\n        showEvents: false\n    # ...\n
"},{"location":"collectors/events/","title":"Events","text":"

Collecting namespace events can help understand what happened inside the cluster.

"},{"location":"collectors/events/#configuration","title":"Configuration","text":"

The full structure of the Events resource is documented here.

"},{"location":"collectors/events/#single-event","title":"Single event","text":"

If a name is specified, Chainsaw will retrieve the specified event in the test namespace.

Collect event in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        name: my-event\n    # ...\n    finally:\n    - events:\n        name: my-event\n    # ...\n

If a namespace is specified, Chainsaw will retrieve the specified event in the specified namespace.

Collect event in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        name: my-event\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        name: my-event\n        namespace: foo\n    # ...\n
"},{"location":"collectors/events/#all-events","title":"All events","text":"

If no name and namespace is specified, Chainsaw will retrieve all events in the test namespace.

Collect all events in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events: {}\n    # ...\n    finally:\n    - events: {}\n    # ...\n

On the other hand, if a namespace is specified, Chainsaw will retrieve all events in the specified namespace.

Collect all events in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        namespace: foo\n    # ...\n
"},{"location":"collectors/events/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the events to be retrieved.

Collect events using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        selector: app=my-app\n    # ...\n    finally:\n    - events:\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will retrieve events using the specified namespace.

Collect events using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - events:\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - events:\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/","title":"Pod logs","text":"

Collecting pod logs can help understand what happened inside one or more pods.

"},{"location":"collectors/pod-logs/#configuration","title":"Configuration","text":"

The full structure of the PodLogs resource is documented here.

"},{"location":"collectors/pod-logs/#single-pod","title":"Single pod","text":"

If a pod name is specified, Chainsaw will retrieve logs from this specific pod in the test namespace.

Collect pod logs in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        name: my-pod\n    # ...\n    finally:\n    - podLogs:\n        name: my-pod\n    # ...\n

If a namespace is specified, Chainsaw will retrieve logs from this specific pod in the specified namespace.

Collect pod logs in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        name: my-pod\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        name: my-pod\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#all-pods","title":"All pods","text":"

If no pod name and namespace is specified, Chainsaw will retrieve logs from all pods in the test namespace.

Collect all pod logs in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs: {}\n    # ...\n    finally:\n    - podLogs: {}\n    # ...\n

On the other hand, if a namespace is specified, Chainsaw will retrieve logs from all pods in the specified namespace.

Collect all pod logs in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#label-selector","title":"Label selector","text":"

An optional label selector can be configured to refine the pods to retrieve logs from.

Collect pod logs using a label selector in the test namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n    # ...\n

If a namespace is specified, Chainsaw will retrieve pod logs using the specified namespace.

Collect pod logs using a label selector in a specific namespace

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n    # ...\n
"},{"location":"collectors/pod-logs/#tail","title":"Tail","text":"

The tail field can be used to limit the amount of log lines retrieved when querying pod logs.

Default

By default, tail will be 10 when a label selector is used, and all if a pod name is specified.

Tail example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        tail: 30\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        tail: 30\n    # ...\n
"},{"location":"collectors/pod-logs/#container","title":"Container","text":"

The container field can be used to retrieve logs from a specific container in the pod.

Default

By default logs from all containers will be fetched.

Container example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    catch:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        container: nginx\n    # ...\n    finally:\n    - podLogs:\n        selector: app=my-app\n        namespace: foo\n        container: nginx\n    # ...\n
"},{"location":"commands/chainsaw/","title":"Chainsaw","text":""},{"location":"commands/chainsaw/#chainsaw","title":"chainsaw","text":"

Stronger tool for e2e testing

chainsaw [flags]\n
"},{"location":"commands/chainsaw/#options","title":"Options","text":"
  -h, --help   help for chainsaw\n
"},{"location":"commands/chainsaw/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_assert/","title":"Chainsaw assert","text":""},{"location":"commands/chainsaw_assert/#chainsaw-assert","title":"chainsaw assert","text":"

Evaluate assertion

chainsaw assert [flags] [FILE]\n
"},{"location":"commands/chainsaw_assert/#options","title":"Options","text":"
  -f, --file string                         Path to the file to assert or '-' to read from stdin\n  -h, --help                                help for assert\n      --kube-as string                      Username to impersonate for the operation\n      --kube-as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n      --kube-as-uid string                  UID to impersonate for the operation\n      --kube-certificate-authority string   Path to a cert file for the certificate authority\n      --kube-client-certificate string      Path to a client certificate file for TLS\n      --kube-client-key string              Path to a client key file for TLS\n      --kube-cluster string                 The name of the kubeconfig cluster to use\n      --kube-context string                 The name of the kubeconfig context to use\n      --kube-disable-compression            If true, opt-out of response compression for all requests to the server\n      --kube-insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n  -n, --kube-namespace string               If present, the namespace scope for this CLI request\n      --kube-password string                Password for basic authentication to the API server\n      --kube-proxy-url string               If provided, this URL will be used to connect via proxy\n      --kube-request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n      --kube-server string                  The address and port of the Kubernetes API server\n      --kube-tls-server-name string         If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n      --kube-token string                   Bearer token for authentication to the API server\n      --kube-user string                    The name of the kubeconfig user to use\n      --kube-username string                Username for basic authentication to the API server\n      --namespace string                    Namespace to use (default \"default\")\n      --no-color                            Removes output colors\n      --timeout duration                    The assert timeout to use (default 30s)\n
"},{"location":"commands/chainsaw_assert/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_build/","title":"Chainsaw build","text":""},{"location":"commands/chainsaw_build/#chainsaw-build","title":"chainsaw build","text":"

Build commands

chainsaw build [flags]\n
"},{"location":"commands/chainsaw_build/#options","title":"Options","text":"
  -h, --help   help for build\n
"},{"location":"commands/chainsaw_build/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_build_docs/","title":"Chainsaw build docs","text":""},{"location":"commands/chainsaw_build_docs/#chainsaw-build-docs","title":"chainsaw build docs","text":"

Build tests documentation

chainsaw build docs [flags]\n
"},{"location":"commands/chainsaw_build_docs/#options","title":"Options","text":"
      --catalog string         Path to the built test catalog file\n  -h, --help                   help for docs\n      --readme-file string     Name of the built docs file (default \"README.md\")\n      --test-dir stringArray   Directories containing test cases to run\n      --test-file string       Name of the test file (default \"chainsaw-test.yaml\")\n
"},{"location":"commands/chainsaw_build_docs/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion/","title":"Chainsaw completion","text":""},{"location":"commands/chainsaw_completion/#chainsaw-completion","title":"chainsaw completion","text":"

Generate the autocompletion script for the specified shell

"},{"location":"commands/chainsaw_completion/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for chainsaw for the specified shell. See each sub-command's help for details on how to use the generated script.

"},{"location":"commands/chainsaw_completion/#options","title":"Options","text":"
  -h, --help   help for completion\n
"},{"location":"commands/chainsaw_completion/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_bash/","title":"Chainsaw completion bash","text":""},{"location":"commands/chainsaw_completion_bash/#chainsaw-completion-bash","title":"chainsaw completion bash","text":"

Generate the autocompletion script for bash

"},{"location":"commands/chainsaw_completion_bash/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package. If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

source <(chainsaw completion bash)\n

To load completions for every new session, execute once:

"},{"location":"commands/chainsaw_completion_bash/#linux","title":"Linux:","text":"
chainsaw completion bash > /etc/bash_completion.d/chainsaw\n
"},{"location":"commands/chainsaw_completion_bash/#macos","title":"macOS:","text":"
chainsaw completion bash > $(brew --prefix)/etc/bash_completion.d/chainsaw\n

You will need to start a new shell for this setup to take effect.

chainsaw completion bash\n
"},{"location":"commands/chainsaw_completion_bash/#options","title":"Options","text":"
  -h, --help              help for bash\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_bash/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_fish/","title":"Chainsaw completion fish","text":""},{"location":"commands/chainsaw_completion_fish/#chainsaw-completion-fish","title":"chainsaw completion fish","text":"

Generate the autocompletion script for fish

"},{"location":"commands/chainsaw_completion_fish/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

chainsaw completion fish | source\n

To load completions for every new session, execute once:

chainsaw completion fish > ~/.config/fish/completions/chainsaw.fish\n

You will need to start a new shell for this setup to take effect.

chainsaw completion fish [flags]\n
"},{"location":"commands/chainsaw_completion_fish/#options","title":"Options","text":"
  -h, --help              help for fish\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_fish/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_powershell/","title":"Chainsaw completion powershell","text":""},{"location":"commands/chainsaw_completion_powershell/#chainsaw-completion-powershell","title":"chainsaw completion powershell","text":"

Generate the autocompletion script for powershell

"},{"location":"commands/chainsaw_completion_powershell/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for powershell.

To load completions in your current shell session:

chainsaw completion powershell | Out-String | Invoke-Expression\n

To load completions for every new session, add the output of the above command to your powershell profile.

chainsaw completion powershell [flags]\n
"},{"location":"commands/chainsaw_completion_powershell/#options","title":"Options","text":"
  -h, --help              help for powershell\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_powershell/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_completion_zsh/","title":"Chainsaw completion zsh","text":""},{"location":"commands/chainsaw_completion_zsh/#chainsaw-completion-zsh","title":"chainsaw completion zsh","text":"

Generate the autocompletion script for zsh

"},{"location":"commands/chainsaw_completion_zsh/#synopsis","title":"Synopsis","text":"

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:

echo \"autoload -U compinit; compinit\" >> ~/.zshrc\n

To load completions in your current shell session:

source <(chainsaw completion zsh)\n

To load completions for every new session, execute once:

"},{"location":"commands/chainsaw_completion_zsh/#linux","title":"Linux:","text":"
chainsaw completion zsh > \"${fpath[1]}/_chainsaw\"\n
"},{"location":"commands/chainsaw_completion_zsh/#macos","title":"macOS:","text":"
chainsaw completion zsh > $(brew --prefix)/share/zsh/site-functions/_chainsaw\n

You will need to start a new shell for this setup to take effect.

chainsaw completion zsh [flags]\n
"},{"location":"commands/chainsaw_completion_zsh/#options","title":"Options","text":"
  -h, --help              help for zsh\n      --no-descriptions   disable completion descriptions\n
"},{"location":"commands/chainsaw_completion_zsh/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_create/","title":"Chainsaw create","text":""},{"location":"commands/chainsaw_create/#chainsaw-create","title":"chainsaw create","text":"

Create Chainsaw resources

chainsaw create [flags]\n
"},{"location":"commands/chainsaw_create/#options","title":"Options","text":"
  -h, --help   help for create\n
"},{"location":"commands/chainsaw_create/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_create_test/","title":"Chainsaw create test","text":""},{"location":"commands/chainsaw_create_test/#chainsaw-create-test","title":"chainsaw create test","text":"

Create a Chainsaw test

chainsaw create test [flags]\n
"},{"location":"commands/chainsaw_create_test/#options","title":"Options","text":"
      --description   If set, adds description when applicable (default true)\n      --force         If set, existing test will be deleted if needed\n  -h, --help          help for test\n      --save          If set, created test will be saved\n
"},{"location":"commands/chainsaw_create_test/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_docs/","title":"Chainsaw docs","text":""},{"location":"commands/chainsaw_docs/#chainsaw-docs","title":"chainsaw docs","text":"

Generate reference documentation

chainsaw docs [flags]\n
"},{"location":"commands/chainsaw_docs/#options","title":"Options","text":"
      --autogenTag      Determines if the generated docs should contain a timestamp (default true)\n  -h, --help            help for docs\n  -o, --output string   Output path (default \".\")\n      --website         Website version\n
"},{"location":"commands/chainsaw_docs/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_export/","title":"Chainsaw export","text":""},{"location":"commands/chainsaw_export/#chainsaw-export","title":"chainsaw export","text":"

Export commands

chainsaw export [flags]\n
"},{"location":"commands/chainsaw_export/#options","title":"Options","text":"
  -h, --help   help for export\n
"},{"location":"commands/chainsaw_export/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_export_schemas/","title":"Chainsaw export schemas","text":""},{"location":"commands/chainsaw_export_schemas/#chainsaw-export-schemas","title":"chainsaw export schemas","text":"

Export JSON schemas

chainsaw export schemas [flags]\n
"},{"location":"commands/chainsaw_export_schemas/#options","title":"Options","text":"
  -h, --help   help for schemas\n
"},{"location":"commands/chainsaw_export_schemas/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_lint/","title":"Chainsaw lint","text":""},{"location":"commands/chainsaw_lint/#chainsaw-lint","title":"chainsaw lint","text":"

Lint a file or read from standard input

"},{"location":"commands/chainsaw_lint/#synopsis","title":"Synopsis","text":"

Use chainsaw lint to lint a specific file or read from standard input for either test or configuration.

chainsaw lint [test|configuration] [flags]\n
"},{"location":"commands/chainsaw_lint/#options","title":"Options","text":"
  -f, --file string   Specify the file to lint or '-' for standard input\n  -h, --help          help for lint\n
"},{"location":"commands/chainsaw_lint/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate/","title":"Chainsaw migrate","text":""},{"location":"commands/chainsaw_migrate/#chainsaw-migrate","title":"chainsaw migrate","text":"

Migrate resources to Chainsaw

chainsaw migrate [flags]\n
"},{"location":"commands/chainsaw_migrate/#options","title":"Options","text":"
  -h, --help   help for migrate\n
"},{"location":"commands/chainsaw_migrate/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl/","title":"Chainsaw migrate kuttl","text":""},{"location":"commands/chainsaw_migrate_kuttl/#chainsaw-migrate-kuttl","title":"chainsaw migrate kuttl","text":"

Migrate KUTTL resources to Chainsaw

chainsaw migrate kuttl [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl/#options","title":"Options","text":"
  -h, --help   help for kuttl\n
"},{"location":"commands/chainsaw_migrate_kuttl/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl_config/","title":"Chainsaw migrate kuttl config","text":""},{"location":"commands/chainsaw_migrate_kuttl_config/#chainsaw-migrate-kuttl-config","title":"chainsaw migrate kuttl config","text":"

Migrate KUTTL config to Chainsaw

chainsaw migrate kuttl config [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl_config/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for config\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_kuttl_config/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_kuttl_tests/","title":"Chainsaw migrate kuttl tests","text":""},{"location":"commands/chainsaw_migrate_kuttl_tests/#chainsaw-migrate-kuttl-tests","title":"chainsaw migrate kuttl tests","text":"

Migrate KUTTL tests to Chainsaw

chainsaw migrate kuttl tests [flags]\n
"},{"location":"commands/chainsaw_migrate_kuttl_tests/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for tests\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_kuttl_tests/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_migrate_tests/","title":"Chainsaw migrate tests","text":""},{"location":"commands/chainsaw_migrate_tests/#chainsaw-migrate-tests","title":"chainsaw migrate tests","text":"

Migrate test steps to test

chainsaw migrate tests [flags]\n
"},{"location":"commands/chainsaw_migrate_tests/#options","title":"Options","text":"
      --cleanup   If set, delete converted files\n  -h, --help      help for tests\n      --save      If set, converted files will be saved\n
"},{"location":"commands/chainsaw_migrate_tests/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_test/","title":"Chainsaw test","text":""},{"location":"commands/chainsaw_test/#chainsaw-test","title":"chainsaw test","text":"

Run tests

chainsaw test [flags]... [test directories]...\n
"},{"location":"commands/chainsaw_test/#options","title":"Options","text":"
      --apply-timeout duration                    The apply timeout to use as default for configuration (default 5s)\n      --assert-timeout duration                   The assert timeout to use as default for configuration (default 30s)\n      --cleanup-delay duration                    Adds a delay between the time a test ends and the time cleanup starts\n      --cleanup-timeout duration                  The cleanup timeout to use as default for configuration (default 30s)\n      --cluster strings                           Register cluster (format <cluster name>=<kubeconfig path>:[context name])\n      --config string                             Chainsaw configuration file\n      --delete-timeout duration                   The delete timeout to use as default for configuration (default 15s)\n      --error-timeout duration                    The error timeout to use as default for configuration (default 30s)\n      --exclude-test-regex string                 Regular expression to exclude tests\n      --exec-timeout duration                     The exec timeout to use as default for configuration (default 5s)\n      --fail-fast                                 Stop the test upon encountering the first failure\n      --force-termination-grace-period duration   If specified, overrides termination grace periods in applicable resources\n      --full-name                                 Use full test case folder path instead of folder name\n  -h, --help                                      help for test\n      --include-test-regex string                 Regular expression to include tests\n      --kube-as string                            Username to impersonate for the operation\n      --kube-as-group stringArray                 Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n      --kube-as-uid string                        UID to impersonate for the operation\n      --kube-certificate-authority string         Path to a cert file for the certificate authority\n      --kube-client-certificate string            Path to a client certificate file for TLS\n      --kube-client-key string                    Path to a client key file for TLS\n      --kube-cluster string                       The name of the kubeconfig cluster to use\n      --kube-context string                       The name of the kubeconfig context to use\n      --kube-disable-compression                  If true, opt-out of response compression for all requests to the server\n      --kube-insecure-skip-tls-verify             If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n  -n, --kube-namespace string                     If present, the namespace scope for this CLI request\n      --kube-password string                      Password for basic authentication to the API server\n      --kube-proxy-url string                     If provided, this URL will be used to connect via proxy\n      --kube-request-timeout string               The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n      --kube-server string                        The address and port of the Kubernetes API server\n      --kube-tls-server-name string               If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n      --kube-token string                         Bearer token for authentication to the API server\n      --kube-user string                          The name of the kubeconfig user to use\n      --kube-username string                      Username for basic authentication to the API server\n      --namespace string                          Namespace to use for tests\n      --no-cluster                                Runs without cluster\n      --no-color                                  Removes output colors\n      --parallel int                              The maximum number of tests to run at once\n      --repeat-count int                          Number of times to repeat each test (default 1)\n      --report-format string                      Test report format (JSON|XML|nil)\n      --report-name string                        The name of the report to create (default \"chainsaw-report\")\n      --selector strings                          Selector (label query) to filter on\n      --skip-delete                               If set, do not delete the resources after running the tests\n      --template                                  If set, resources will be considered for templating\n      --test-dir strings                          Directories containing test cases to run\n      --test-file string                          Name of the test file (default \"chainsaw-test.yaml\")\n      --values strings                            Values passed to the tests\n
"},{"location":"commands/chainsaw_test/#see-also","title":"SEE ALSO","text":""},{"location":"commands/chainsaw_version/","title":"Chainsaw version","text":""},{"location":"commands/chainsaw_version/#chainsaw-version","title":"chainsaw version","text":"

Print the version informations

chainsaw version [flags]\n
"},{"location":"commands/chainsaw_version/#options","title":"Options","text":"
  -h, --help   help for version\n
"},{"location":"commands/chainsaw_version/#see-also","title":"SEE ALSO","text":""},{"location":"configuration/","title":"Configuring Chainsaw","text":"

Chainsaw is a comprehensive tool designed to facilitate end-to-end testing in Kubernetes.

This documentation will focus on providing a breakdown of its configuration structure and how to use it.

Chainsaw can be configured in two different and complementary ways:

Precedence

If both are specified, command-line flags will take precedence over configuration coming from a configuration file.

"},{"location":"configuration/#specific-configuration-options","title":"Specific configuration options","text":"

Please pay attention to the configuration options below, they may or may not be relevant in your case but can be useful in certain cases:

"},{"location":"configuration/cleanup-delay/","title":"Delay before cleanup","text":"

At the end of each test, Chainsaw will delete the resources it created during the test.

When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update the internal state.

For this reason, Chainsaw provides the delayBeforeCleanup configuration option and the corresponding --delay-before-cleanup flag.

"},{"location":"configuration/cleanup-delay/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  delayBeforeCleanup: 5s\n  # ...\n
"},{"location":"configuration/cleanup-delay/#flag","title":"Flag","text":"
chainsaw test --delay-before-cleanup 5s ...\n
"},{"location":"configuration/file/","title":"Configuration file","text":"

Chainsaw is described as a Stronger tool for e2e testing.

With its versatile configuration options, you can customize the testing process to fit your needs.

Chainsaw prioritizes its configuration in the following order:

  1. User-specified configuration

    If you explicitly provide a configuration file using a command-line flag

  2. Default configuration file

    If no configuration is specified, Chainsaw will look for a default file named .chainsaw.yaml in the current working directory

  3. Internal default configuration

    In the absence of both the above, Chainsaw will use a default configuration file embedded in the Chainsaw binary

"},{"location":"configuration/file/#example","title":"Example","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  timeouts:\n    apply: 45s\n    assert: 20s\n    cleanup: 45s\n    delete: 25s\n    error: 10s\n    exec: 45s\n  skipDelete: false\n  failFast: true\n  parallel: 4\n  # ...\n
"},{"location":"configuration/file/#how-to-specify-a-configuration","title":"How to specify a configuration","text":"

To use a custom configuration file:

chainsaw test --config path/to/your/config.yaml\n

Defaults

If you don't specify any configuration, Chainsaw will look for the default configuration file .chainsaw.yaml in the current working directory.

If that file is not found, it will fall back to its internal default configuration.

"},{"location":"configuration/file/#default-configuration","title":"Default configuration","text":"

The default configuration below is used by Chainsaw when no configuration file was provided and the default file .chainsaw.yaml does not exist.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: default\nspec: {}\n
"},{"location":"configuration/file/#reference-documentation","title":"Reference documentation","text":"

See Configuration API reference for more details.

"},{"location":"configuration/flags/","title":"Command line flags","text":"

Even after a configuration file is loaded, you can override specific settings using command-line flags.

"},{"location":"configuration/flags/#example","title":"Example","text":"
chainsaw test                         \\\n    path/to/test/dir                    \\\n    --config path/to/your/config.yaml   \\\n    --assert-timeout 45s                \\\n    --skip-delete false                 \\\n    --fail-fast true                    \\\n    --parallel 4                        \\\n    ...\n

In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.

"},{"location":"configuration/flags/#usage","title":"Usage","text":"

The command below will run tests using the configuration from my-config.yaml, taking tests from /path/to/tests, and running a maximum of 10 tests simultaneously.

chainsaw test /path/to/tests --config my-config.yaml --parallel 10\n
"},{"location":"configuration/flags/#reference-documentation","title":"Reference documentation","text":"

See Chainsaw test command reference for more details.

"},{"location":"configuration/grace/","title":"Termination graceful period","text":"

Some Kubernetes resources can take time before being stopped. For example, deleting a Pod can take time if the underlying container doesn't quit quickly enough.

For this reason, Chainsaw provides the forceTerminationGracePeriod configuration option and the corresponding --force-termination-grace-period flag. If set, Chainsaw will override the terminationGracePeriodSeconds when working with the following resource kinds:

"},{"location":"configuration/grace/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  forceTerminationGracePeriod: 5s\n  # ...\n
"},{"location":"configuration/grace/#flag","title":"Flag","text":"
chainsaw test --force-termination-grace-period 5s ...\n
"},{"location":"configuration/no-cluster/","title":"Running without a cluster","text":"

Chainsaw can be run without connection to a Kubernetes cluster. In this case, chainsaw will not try to create an ephemeral namespace and all operations requiring a Kubernetes cluster will fail.

To run chainsaw in this mode pass the --no-cluster flag.

"},{"location":"configuration/no-cluster/#example","title":"Example","text":"
# run chainsaw without connection to a Kubernetes cluster\nchainsaw test --no-cluster\n
"},{"location":"configuration/reports/","title":"Reports","text":"

Chainsaw can generate JUnit reports in XML or JSON format.

To produce a test report, configure the report format and report name in the configuration or using CLI flags.

"},{"location":"configuration/reports/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  reportFormat: JSON\n  reportName: chainsaw-report.json\n  # ...\n
"},{"location":"configuration/reports/#flag","title":"Flag","text":"
chainsaw test --report-format JSON --report-name chainsaw-report.json ...\n
"},{"location":"configuration/selector/","title":"Label selectors","text":"

Chainsaw can filter the tests to run using label selectors.

You can pass label selectors using the --selector flag when invoking the chainsaw test command.

"},{"location":"configuration/selector/#example","title":"Example","text":"

Given the test below:

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\n  labels:\n    foo: bar\nspec:\n  # ...\n

Invoking Chainsaw with the command below will take the test above into account:

chainsaw test --selector foo=bar\n
"},{"location":"configuration/templating/","title":"Resource templating","text":"

Experimental status

This is an experimental feature, and implementation could change slightly in the next versions.

Chainsaw can apply transformations to the resources before they are processed by the operation.

This is useful when a resource needs some runtime configuration.

Templating must be enabled at the configuration, test, step, or operation level for the templating process to kick in. Alternatively, templating can be enabled using the --template flag when invoking chainsaw from the command line.

Note

Unlike assertion trees, templating can only be specified in leave nodes of the YAML tree.

"},{"location":"configuration/templating/#example","title":"Example","text":"

The test below will create, assert, and delete a ConfigMap with a dynamic name configured at runtime using the $namespace binding.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: values\nspec:\n  steps:\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: ($namespace)\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: ($namespace)\n    - delete:\n        ref:\n          apiVersion: v1\n          kind: ConfigMap\n          name: ($namespace)\n
"},{"location":"configuration/timeouts/","title":"Timeouts","text":"

Timeouts in Chainsaw are specified per type of operation. This is required because the timeout varies greatly depending on the nature of an operation.

For example, applying a manifest in a cluster is expected to be reasonably fast, while validating a resource can be a long operation.

Chainsaw supports separately configuring the timeouts below:

Overriding timeouts

Each timeout can be overridden at the test level, test step level, or individual operation level.

Timeouts defined in the Configuration are used in operations when not overridden.

"},{"location":"configuration/timeouts/#configuration","title":"Configuration","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Configuration\nmetadata:\n  name: custom-config\nspec:\n  # ...\n  timeouts:\n    apply: 45s\n    assert: 20s\n    cleanup: 45s\n    delete: 25s\n    error: 10s\n    exec: 45s\n  # ...\n
"},{"location":"configuration/timeouts/#flag","title":"Flag","text":"
chainsaw test                     \\\n    --apply-timeout 45s             \\\n    --assert-timeout 45s            \\\n    --cleanup-timeout 45s           \\\n    --delete-timeout 45s            \\\n    --error-timeout 45s             \\\n    --exec-timeout 45s              \\\n    ...\n
"},{"location":"configuration/values/","title":"Passing data to tests","text":"

Chainsaw can pass arbitrary values when running tests using the --values flag. Values will be available to tests under the $values binding.

This is useful when a test needs to be configured externally.

"},{"location":"configuration/values/#example","title":"Example","text":"

The test below expects the $value.foo to be provided when chainsaw is invoked.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: values\nspec:\n  steps:\n  - try:\n    - assert:\n        resource:\n          ($values.foo): bar\n

Now you can invoke chainsaw like this:

# pass object { \"foo\": \"bar\"\u00a0} as values to the executed tests\n# `--values -` means values are read from standard input\necho \"foo: bar\" | chainsaw test --values -\n\n# read values from a file\nchainsaw test --values ./values.yaml\n\n# pass values using heredoc\nchainsaw test --values - <<EOF\nfoo: bar\nEOF\n
"},{"location":"examples/","title":"Setup","text":"

To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.

In these examples, we will use kind but feel free to use the tool of your choice.

Not a cluster management tool

We consider this is not the responsibility of Chainsaw to manage clusters.

There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.

"},{"location":"examples/#create-a-kind-cluster","title":"Create a kind cluster","text":"

Please refer to the kind install docs to install it locally.

Once kind creating a local cluster is as simple as running:

# define kind image\nexport KIND_IMAGE=\"kindest/node:v1.28.0\"\n\n# create cluster\nkind create cluster --image $KIND_IMAGE\n
"},{"location":"examples/#install-chainsaw","title":"Install Chainsaw","text":"

Refer to install docs to install Chainsaw.

"},{"location":"examples/array-assertions/","title":"Array assertions","text":"

This example demonstrates how to perform complex assertions on arrays.

"},{"location":"examples/array-assertions/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/array-assertions/#manifests","title":"Manifests","text":""},{"location":"examples/array-assertions/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: Pod\nmetadata:\n  name: example\nspec:\n  containers:\n  - name: container-1\n    image: nginx-1\n    env:\n    - name: ENV_1\n      value: value-1\n  - name: container-2\n    image: nginx-2\n    env:\n    - name: ENV_2\n      value: value-2\n  - name: container-3\n    image: nginx-3\n    env:\n    - name: ENV_3\n      value: value-3\n
"},{"location":"examples/array-assertions/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: Pod\nmetadata:\n  name: example\nspec:\n  # iterate over all containers having `name: container-1`\n  ~.(containers[?name == 'container-1']):\n    image: nginx-1\n  # iterate over all containers, bind `$index` to the element index\n  ~index.(containers):\n    image: (join('-', ['nginx', to_string($index + `1`)]))\n  # nested iteration\n  ~index2.(containers):\n    ~.(env):\n      name: (join('_', ['ENV', to_string($index2 + `1`)]))\n      value: (join('-', ['value', to_string($index2 + `1`)]))\n
"},{"location":"examples/array-assertions/#test","title":"Test","text":""},{"location":"examples/array-assertions/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: array-assertions\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/array-assertions/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/assertion-tree/","title":"Assertion trees","text":"

Test steps:

  1. Creates a Deployment with 2 replicas
  2. Asserts that the number of replicas is > 1 and that status.replicas == spec.replicas
"},{"location":"examples/assertion-tree/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/assertion-tree/#manifests","title":"Manifests","text":""},{"location":"examples/assertion-tree/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx-deployment\nspec:\n  selector:\n    matchLabels:\n      app: nginx\n  replicas: 2\n  template:\n    metadata:\n      labels:\n        app: nginx\n    spec:\n      containers:\n      - name: nginx\n        image: nginx\n
"},{"location":"examples/assertion-tree/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx-deployment\n(status.replicas == spec.replicas): true\nspec:\n  (replicas > `1`): true\n
"},{"location":"examples/assertion-tree/#test","title":"Test","text":""},{"location":"examples/assertion-tree/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: assertion-tree\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/assertion-tree/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/basic/","title":"Basic","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data
"},{"location":"examples/basic/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/basic/#manifests","title":"Manifests","text":""},{"location":"examples/basic/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/basic/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/basic/#test","title":"Test","text":""},{"location":"examples/basic/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/basic/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/inline/","title":"Inline resources","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data
"},{"location":"examples/inline/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/inline/#test","title":"Test","text":""},{"location":"examples/inline/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: inline-resources\nspec:\n  steps:\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: quick-start\n          data:\n            foo: bar\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: quick-start\n          data:\n            foo: bar\n
"},{"location":"examples/inline/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/labels-selector/","title":"Labels selector","text":"

Test steps:

  1. Creates a ConfigMap
  2. Asserts the ConfigMap contains the expected data

This test is similar to the basic example but Chainsaw is invoked with the --selector flag.

"},{"location":"examples/labels-selector/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/labels-selector/#manifests","title":"Manifests","text":""},{"location":"examples/labels-selector/#resourcesyaml","title":"resources.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/labels-selector/#assertionsyaml","title":"assertions.yaml","text":"
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: quick-start\ndata:\n  foo: bar\n
"},{"location":"examples/labels-selector/#test","title":"Test","text":""},{"location":"examples/labels-selector/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: labels-selector\n  labels:\n    # test labels\n    test-suite: examples\nspec:\n  steps:\n  - try:\n    - apply:\n        file: resources.yaml\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/labels-selector/#execute","title":"Execute","text":"
# invoke chainsaw with `--selector` to filter tests to run\nchainsaw test --selector test-suite=examples\n
"},{"location":"examples/non-resource-assertion/","title":"Non resource assertion","text":"

This test example demonstrates how to perform assertions not based on resources.

  1. Asserts that the number of nodes in the cluster is equal to 1
"},{"location":"examples/non-resource-assertion/#setup","title":"Setup","text":"

See Setup docs

"},{"location":"examples/non-resource-assertion/#manifests","title":"Manifests","text":""},{"location":"examples/non-resource-assertion/#assertionsyaml","title":"assertions.yaml","text":"
(length(x_k8s_list($client, 'v1', 'Node'))): 1\n
"},{"location":"examples/non-resource-assertion/#test","title":"Test","text":""},{"location":"examples/non-resource-assertion/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: non-resource-assertion\nspec:\n  steps:\n  - try:\n    - assert:\n        file: assertions.yaml\n
"},{"location":"examples/non-resource-assertion/#execute","title":"Execute","text":"
chainsaw test\n
"},{"location":"examples/test-output/","title":"Testing command output","text":"

Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.

"},{"location":"examples/test-output/#checking-output-contains","title":"Checking Output Contains","text":"

One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.

kubectl version\n\nClient Version: v1.28.2\nKustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\nServer Version: v1.27.4+k3s1\n

Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl version\n        check:\n          # This check ensures that the string '1.28' is found\n          # in stdout or else fails\n          (contains($stdout, '1.28')): true\n

Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.

- script:\n    content: kubectl version\n    check:\n      # This check ensures that the string '1.25' is NOT found\n      # in stdout or else fails\n      (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"

In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl version\n        check:\n          # This check ensures the contents of stdout are exactly as shown.\n          # Any deviations will cause a failure.\n          ($stdout): |-\n            Client Version: v1.28.2\n            Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n            Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"

In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.

kubectl foo\n\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n        top\n

Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check bad kubectl command\n    try:\n    - script:\n        content: kubectl foo\n        check:\n          # This checks that the result of the content was an error.\n          ($error != null): true\n          # This check below ensures that the string 'top' is found in stderr or else fails\n          (contains($stderr, 'top')): true\n

Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test\nspec:\n  steps:\n  - name: Check kubectl\n    try:\n    - script:\n        content: kubectl foo\n        check:\n          # This checks that the result of the content was an error.\n          ($error != null): true\n          # This checks that the output is exactly as intended.\n          ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"jp/functions/","title":"Functions","text":"

Experimental functions

Experimental functions are denoted by the x_ prefix.

These are functions that are subject to signature change in a future version.

"},{"location":"jp/functions/#built-in-functions","title":"built-in functions","text":"Name Signature abs abs(number) avg avg(array[number]) ceil ceil(number) contains contains(array|string, any) ends_with ends_with(string, string) find_first find_first(string, string, number, number) find_last find_last(string, string, number, number) floor floor(number) from_items from_items(array[array]) group_by group_by(array, expref) items items(object) join join(string, array[string]) keys keys(object) length length(string|array|object) lower lower(string) map map(expref, array) max max(array[number]|array[string]) max_by max_by(array, expref) merge merge(object) min min(array[number]|array[string]) min_by min_by(array, expref) not_null not_null(any) pad_left pad_left(string, number, string) pad_right pad_right(string, number, string) replace replace(string, string, string, number) reverse reverse(array|string) sort sort(array[string]|array[number]) sort_by sort_by(array, expref) split split(string, string, number) starts_with starts_with(string, string) sum sum(array[number]) to_array to_array(any) to_number to_number(any) to_string to_string(any) trim trim(string, string) trim_left trim_left(string, string) trim_right trim_right(string, string) type type(any) upper upper(string) values values(object) zip zip(array, array)"},{"location":"jp/functions/#kyverno-json-functions","title":"kyverno-json functions","text":"Name Signature at at(array, any) concat concat(string, string) json_parse json_parse(string) wildcard wildcard(string, string)"},{"location":"jp/functions/#kyverno-functions","title":"kyverno functions","text":"Name Signature compare compare(string, string) equal_fold equal_fold(string, string) replace replace(string, string, string, number) replace_all replace_all(string, string, string) to_upper to_upper(string) to_lower to_lower(string) trim trim(string, string) trim_prefix trim_prefix(string, string) split split(string, string) regex_replace_all regex_replace_all(string, string|number, string|number) regex_replace_all_literal regex_replace_all_literal(string, string|number, string|number) regex_match regex_match(string, string|number) pattern_match pattern_match(string, string|number) label_match label_match(object, object) to_boolean to_boolean(string) add add(any, any) sum sum(array) subtract subtract(any, any) multiply multiply(any, any) divide divide(any, any) modulo modulo(any, any) round round(number, number) base64_decode base64_decode(string) base64_encode base64_encode(string) time_since time_since(string, string, string) time_now time_now() time_now_utc time_now_utc() path_canonicalize path_canonicalize(string) truncate truncate(string, number) semver_compare semver_compare(string, string) parse_json parse_json(string) parse_yaml parse_yaml(string) lookup lookup(object|array, string|number) items items(object|array, string, string) object_from_lists object_from_lists(array, array) random random(string) x509_decode x509_decode(string) time_to_cron time_to_cron(string) time_add time_add(string, string) time_parse time_parse(string, string) time_utc time_utc(string) time_diff time_diff(string, string) time_before time_before(string, string) time_after time_after(string, string) time_between time_between(string, string, string) time_truncate time_truncate(string, string)"},{"location":"jp/functions/#chainsaw-functions","title":"chainsaw functions","text":"Name Signature env env(string) x_k8s_get x_k8s_get(any, string, string, string, string) x_k8s_list x_k8s_list(any, string, string, string) x_k8s_exists x_k8s_exists(any, string, string, string, string) x_k8s_resource_exists x_k8s_resource_exists(any, string, string)"},{"location":"more/crds/","title":"Working with CRDs","text":"

New Custom Resource Definitions are not immediately available for use in the Kubernetes API until the Kubernetes API has acknowledged them.

If a Custom Resource Definition is being defined inside of a test step, be sure to wait for the CustomResourceDefinition object to appear.

For example, given this Custom Resource Definition:

apiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\nmetadata:\n  name: mycrds.mycrd.k8s.io\nspec:\n  group: mycrd.k8s.io\n  version: v1alpha1\n  names:\n    kind: MyCRD\n    listKind: MyCRDList\n    plural: mycrds\n    singular: mycrd\n  scope: Namespaced\n

Create the following assert:

apiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\nmetadata:\n  name: mycrds.mycrd.k8s.io\nstatus:\n  acceptedNames:\n    kind: MyCRD\n    listKind: MyCRDList\n    plural: mycrds\n    singular: mycrd\n  storedVersions:\n  - v1alpha1\n

And then the CRD can be used in subsequent steps:

apiVersion: mycrd.k8s.io/v1alpha1\nkind: MyCRD\nmetadata:\n  name: my-crds\nspec:\n  test: test\n
"},{"location":"more/events/","title":"Working with events","text":"

Kubernetes events are regular Kubernetes objects and can be asserted on just like any other object:

apiVersion: v1\nkind: Event\nreason: Started\nsource:\n  component: kubelet\ninvolvedObject:\n  apiVersion: v1\n  kind: Pod\n  name: my-pod\n
"},{"location":"more/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"more/kuttl-migration/#overview","title":"Overview","text":"

The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/kuttl-migration/#examples","title":"Examples","text":""},{"location":"more/kuttl-migration/#migrate-tests","title":"Migrate tests","text":"

The command below will migrate KUTTL tests to Chainsaw and overwrite original files with converted ones.

chainsaw migrate kuttl tests path/to/kuttl/tests --save --cleanup\n

This will generate a chainsaw-test.yaml for every KUTTL test discovered.

"},{"location":"more/kuttl-migration/#migrate-configuration","title":"Migrate configuration","text":"

The command below will migrate a KUTTL test suite file to the corresponding Chainsaw Configuration.

chainsaw migrate kuttl config path/to/kuttl/testsuite --save --cleanup\n

This will generate a .chainsaw.yaml configuration file.

"},{"location":"more/lint/","title":"Lint tests","text":""},{"location":"more/lint/#overview","title":"Overview","text":"

Chainsaw comes with a lint command to detect ill-formated tests.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/lint/#usage","title":"Usage","text":"

To build the docs of a test, Chainsaw provides the chainsaw lint test -f path/to/chainsaw-test.yaml command.

chainsaw lint test -f - <<EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: assertion-tree\nspec:\n  steps:\n  - try:\n    - assert:\n        file: assert.yaml\nEOF\n
Processing input...\nThe document is valid\n
"},{"location":"more/test-docs/","title":"Building test docs","text":""},{"location":"more/test-docs/#overview","title":"Overview","text":"

Chainsaw makes it simple to build the documentation of your tests.

As test suites grow, it becomes important to document what a test does and how it is supposed to work.

Going through the implementation of a test to understand its purpose is not an efficient strategy.

Reference documentation

You can view the full command documentation here.

"},{"location":"more/test-docs/#usage","title":"Usage","text":"

To build the docs of a test, Chainsaw provides the chainsaw build docs command.

chainsaw build docs --test-dir path/to/chainsaw/tests\n

This will automatically discover tests and document steps and operations in try, catch and finally statements.

"},{"location":"more/test-docs/#the-description-field","title":"The description field","text":"

Additionally, you can set the description field in:

Chainsaw will output them nicely in the built docs.

"},{"location":"more/test-docs/#example","title":"Example","text":"

See below for an example test and the corresponding built docs.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: basic\nspec:\n  description: This is a very simple test that creates a configmap and checks the content is as expected.\n  steps:\n  - description: This steps applies the configmap in the cluster and checks the configmap content.\n    try:\n    - description: Create the configmap.\n      apply:\n        file: configmap.yaml\n    - description: Check the configmap content.\n      assert:\n        file: configmap-assert.yaml\n
"},{"location":"more/test-docs/#test-basic","title":"Test: basic","text":"

This is a very simple test that creates a configmap and checks the content is as expected.

"},{"location":"more/test-docs/#steps","title":"Steps","text":"# Name Try Catch Finally 1 step-1 2 0 0"},{"location":"more/test-docs/#step-step-1","title":"Step: step-1","text":"

This step applies the configmap in the cluster and checks the configmap content.

"},{"location":"more/test-docs/#try","title":"Try","text":"# Operation Description 1 apply Create the configmap. 2 assert Check the configmap content."},{"location":"operations/","title":"Operations","text":"

While tests are made of test steps, test steps can be considered made of operations.

Every operation in a test step runs sequentially.

Only one action per operation

Every operation consists of a single action. While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.

The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed in case an operation consists of multiple actions. For this reason, operations consisting of multiple actions are disallowed.

"},{"location":"operations/#common-fields","title":"Common fields","text":"

All operations share some configuration fields.

Reference documentation

The full structure of the Operation is documented here.

"},{"location":"operations/#continueonerror","title":"ContinueOnError","text":"

Determines whether a test step should continue or not in case the operation is not successful.

Even if the test continues executing, it will still be reported as failed.

"},{"location":"operations/#available-operations","title":"Available operations","text":""},{"location":"operations/#non-resource-assertions","title":"Non-resource assertions","text":"

It is possible to evaluate assertions that do not depend on resources.

See Non-resource assertions for details.

"},{"location":"operations/#operation-checks","title":"Operation checks","text":"

Some operations support checking the operation execution result against specific expectations.

See Operation checks for use case details and supported operations.

"},{"location":"operations/#resource-templating","title":"Resource templating","text":"

Chainsaw can apply transformations to the resources before they are processed by the operation.

See Resource templating for use case details and supported operations.

"},{"location":"operations/apply/","title":"Apply","text":"

The apply operation lets you define resources that should be applied to the Kubernetes cluster during the test step.

These can be configurations, deployments, services, or any other Kubernetes resource.

Reference documentation

The full structure of the Apply is documented here.

"},{"location":"operations/apply/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using apply in a Test resource.

Using a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: my-configmap.yaml\n    # ...\n

Using file path expressions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: \"configs/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/step/configmap.yaml\n    # ...\n

Using an inline resource

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n    # ...\n
"},{"location":"operations/apply/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- apply:\n    file: my-configmap.yaml\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n

With check

# ...\n- apply:\n    resource:\n      apiVersion: v1\n      kind: ConfigMap\n      metadata:\n        name: chainsaw-quick-start\n      data:\n        foo: bar\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/assert/","title":"Assert","text":"

The assert operation allows you to specify conditions that should hold true for a successful test.

For example, after applying resources, you might want to ensure that a particular pod is running or a service is accessible.

Assertion trees

Assertions in Chainsaw are based on assertion trees.

Assertion trees are a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Reference documentation

The full structure of the Assert is documented here.

"},{"location":"operations/assert/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using assert in a Test resource.

Using a specific file for assertions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: ../resources/deployment-assert.yaml\n    # ...\n

Using file path expressions for assertions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: \"../assertions/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/resource/valid.yaml\n    # ...\n

Using an inline assertion tree

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - assert:\n        resource:\n          apiVersion: v1\n          kind: Deployment\n          metadata:\n            name: foo\n          spec:\n            (replicas > 3): true\n    # ...\n
"},{"location":"operations/check/","title":"Operation checks","text":"

Considering an operation success or failure is not always as simple as checking an error code.

To support those kind of use cases, some operations support an additional check field to evaluate the operation result against an assertion tree.

Assertion trees

Assertions in Chainsaw are based on assertion trees.

Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Checked model

Different operation have a different model passed through the assertion tree.

The object passed to the assertion tree is the output object of the operation. Additional data like error or standard logs are passed using bindings ($error, $stdout, $stderr)

"},{"location":"operations/check/#expect-vs-check","title":"Expect vs Check","text":"

While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply and create operations. Those operations can operate on files containing multiple manifests and every manifest can have a different result.

To support more granular checks we use the expect field that contains an array of Expectation. Every expectation is made of an optional match and a check statement.

This way it is possible to control the scope of a check.

Null match

If the match statement is null, the check statement applies to all manifests in the operation.

If no expectation matches a given manifest, the default expectation will be used, checking that no error occured.

"},{"location":"operations/check/#apply","title":"Apply","text":"

apply supports expect and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#command","title":"Command","text":"

command supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string $stdout The content of the standard console output (if any) at the end of the operation string $stderr The content of the standard console error output (if any) at the end of the operation string @ Always null"},{"location":"operations/check/#create","title":"Create","text":"

create supports expect and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#delete","title":"Delete","text":"

delete supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string @ The state of the resource (if any) at the end of the operation object"},{"location":"operations/check/#script","title":"Script","text":"

script supports check and has the following elements to be checked:

Name Purpose Type $values Values provided when invoking chainsaw with --values flag object $namespace Name of the current test namespace string $client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object $error The error message (if any) at the end of the operation string $stdout The content of the standard console output (if any) at the end of the operation string $stderr The content of the standard console error output (if any) at the end of the operation string @ Always null"},{"location":"operations/command/","title":"Command","text":"

The command operation provides a means to execute a specific command during the test step.

Reference documentation

The full structure of the Command is documented here.

"},{"location":"operations/command/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using command in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - command:\n        entrypoint: echo\n        args:\n        - hello chainsaw\n    # ...\n
"},{"location":"operations/command/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- command:\n    entrypoint: echo\n    args:\n    - hello chainsaw\n    check:\n      # an error is expected, this will:\n      # - succeed if the operation failed\n      # - fail if the operation succeeded\n      ($error != null): true\n# ...\n
"},{"location":"operations/create/","title":"Create","text":"

The create operation lets you define resources that should be created in the Kubernetes cluster during the test step. These can be configurations, deployments, services, or any other Kubernetes resource.

Warning

If the resource to be created already exists in the cluster, the step will fail.

Reference documentation

The full structure of the Create is documented here.

"},{"location":"operations/create/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using create in a Test resource.

Using a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        file: my-configmap.yaml\n    # ...\n

Using file path expressions

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - apply:\n        file: \"configs/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/resource/valid.yaml\n    # ...\n

Using an inline resource

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - create:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n    # ...\n
"},{"location":"operations/create/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- apply:\n    file: my-configmap.yaml\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n

With check

# ...\n- apply:\n    resource:\n      apiVersion: v1\n      kind: ConfigMap\n      metadata:\n        name: chainsaw-quick-start\n      data:\n        foo: bar\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: ConfigMap\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/delete/","title":"Delete","text":"

The delete operation allows you to specify resources that should be deleted from the Kubernetes cluster before a particular test step is executed.

Reference documentation

The full structure of the Delete is documented here.

"},{"location":"operations/delete/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using delete in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - delete:\n        ref:\n          apiVersion: v1\n          kind: Pod\n          namespace: default\n          name: my-test-pod\n    # ...\n
"},{"location":"operations/delete/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- delete:\n    ref:\n      apiVersion: v1\n      kind: Pod\n      namespace: default\n      name: my-test-pod\n    expect:\n    - match:\n        # this check applies only if the match\n        # statement below evaluates to `true`\n        apiVersion: v1\n        kind: Pod\n        metadata:\n          namespace: default\n          name: my-test-pod\n      check:\n        # an error is expected, this will:\n        # - succeed if the operation failed\n        # - fail if the operation succeeded\n        ($error != null): true\n# ...\n
"},{"location":"operations/error/","title":"Error","text":"

The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.

Assertion trees

Errors in Chainsaw are based on assertion trees.

Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.

Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more. Please see the assertion trees documentation in kyverno-json for details.

Reference documentation

The full structure of the Error is documented here.

"},{"location":"operations/error/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using error in a Test resource.

Expecting an error from a specific file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: ../resources/deployment-error.yaml\n    # ...\n

Using file path expressions to expect errors from multiple files

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example-multi\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: \"../errors/*.yaml\"\n    # ...\n

Using an URL

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        file: https://raw.githubusercontent.com/user/repo/branch/path/to/deployment-error.yaml\n    # ...\n

Using an inline assertion tree

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - error:\n        resource:\n          apiVersion: v1\n          kind: Deployment\n          metadata:\n            name: foo\n          spec:\n            (replicas > 3): true\n    # ...\n
"},{"location":"operations/non-resource-assert/","title":"Non-resource assertions","text":"

Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.

"},{"location":"operations/non-resource-assert/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using assert in a Test resource.

Using a file

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: non-resource-assertion\nspec:\n  steps:\n  - try:\n    - assert:\n        resource:\n          (x_k8s_list($client, 'v1', 'Node')):\n            (length(@)): 1\n    - error:\n        resource:\n          (x_k8s_list($client, 'v1', 'Node')):\n            (length(@)): 2\n
"},{"location":"operations/script/","title":"Script","text":"

The script operation provides a means to run a script during the test step.

Reference documentation

The full structure of the Script is documented here.

"},{"location":"operations/script/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using script in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - script:\n        content: |\n          echo \"hello chainsaw\"\n    # ...\n
"},{"location":"operations/script/#operation-check","title":"Operation check","text":"

Below is an example of using an operation check.

With check

# ...\n- script:\n    content: |\n      echo \"hello chainsaw\"\n    check:\n      # an error is expected, this will:\n      # - succeed if the operation failed\n      # - fail if the operation succeeded\n      ($error != null): true\n# ...\n
"},{"location":"operations/sleep/","title":"Sleep","text":"

The sleep operation provides a means to sleep for a configured duration.

Reference documentation

The full structure of the Sleep is documented here.

"},{"location":"operations/sleep/#usage-in-test","title":"Usage in Test","text":"

Below is an example of using sleep in a Test resource.

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    # ...\n    - sleep:\n        duration: 30s\n    # ...\n
"},{"location":"operations/templating/","title":"Resource templating","text":"

Experimental status

This is an experimental feature, and implementation could change slightly in the next versions.

Chainsaw can apply transformations to the resources before they are processed by the operation.

This is useful when a resource needs some runtime configuration.

Templating must be enabled at the configuration, test, step, or operation level for the templating process to kick in. Alternatively, templating can be enabled using the --template flag when invoking chainsaw from the command line.

Note

Unlike assertion trees, templating can only be specified in leave nodes of the YAML tree.

"},{"location":"operations/templating/#supported-operations","title":"Supported operations","text":"

Resource templating is supported in the following operations:

"},{"location":"operations/templating/#assert-and-error","title":"Assert and Error","text":"

When templating assert or error operations, the content is already an assertion tree.

For this reason, only the elements used for looking up the resources to be processed by the operation will be considered for templating. That is, only apiVersion, kind, name, namespace and labels are considered for templating. Other fields are not, they are part of the assertion tree only.

assert and error example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - assert:\n      resource:\n        # apiVersion, kind, name, namespace and labels are considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        # other fields are not (they are part of the assertion tree)\n        data:\n          foo: ($namespace)\n  - error:\n      resource:\n        # apiVersion, kind, name, namespace and labels are considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        # other fields are not (they are part of the assertion tree)\n        data:\n          bar: ($namespace)\n
"},{"location":"operations/templating/#apply-and-create","title":"Apply and Create","text":"

When templating apply or create operations, the whole content is considered for templating.

apply and create example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - apply:\n      resource:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        data:\n          foo: ($namespace)\n  - create:\n      resource:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        metadata:\n          name: ($namespace)\n        data:\n          foo: ($namespace)\n
"},{"location":"operations/templating/#delete","title":"Delete","text":"

When templating delete operations, the whole content is considered for templating.

apply and create example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: template\nspec:\n  template: true\n  steps:\n  - delete:\n      ref:\n        # the whole content is considered for templating\n        apiVersion: v1\n        kind: ConfigMap\n        name: ($namespace)\n
"},{"location":"steps/","title":"What is a test step","text":"

A test step is made of three main components used to dermine the actions Chainsaw will perform when executing the step.

  1. The try statement (required)
  2. The catch statement (optional)
  3. The finally statement (optional)

Reference documentation

The full structure of the TestStep is documented here.

"},{"location":"steps/#test-step-lifecycle","title":"Test step lifecycle","text":"

Test step lifecycle

  1. The step starts executing operations in the try statement
  2. If an operation fails in the try statement
    1. If a catch statement is present, all operations and collectors are executed
  3. If a finally statement is present, all operations and collectors are executed
"},{"location":"steps/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    - apply:\n        file: path/to/apply.yaml\n    - assert:\n        file: path/to/assert.yaml\n    catch: []\n    finally: []\n
"},{"location":"steps/catch/","title":"Catch","text":"

A catch statement is also a sequence of operations or collectors.

Operations and collectors contained in a catch statement will be executed only if the step failed when executing the operations in the step's try statement.

Tip

All operations and collectors of a catch statement will be executed regardless of the success or failure of each of them.

"},{"location":"steps/catch/#operations","title":"Operations","text":"

A catch statement supports only the following operations:

"},{"location":"steps/catch/#collectors","title":"Collectors","text":"

A catch statement supports all collectors:

"},{"location":"steps/catch/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try: []\n    catch:\n    - command:\n        # ...\n    - script:\n        # ...\n    - events:\n        # ...\n    - podLogs:\n        # ...\n    - describe:\n        # ...\n    - sleep:\n        # ...\n    finally: []\n
"},{"location":"steps/finally/","title":"Finally","text":"

A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.

Tip

All operations and collectors of a finally statement will be executed regardless of the success or failure of each of them.

"},{"location":"steps/finally/#operations","title":"Operations","text":"

A finally statement supports only the following operations:

"},{"location":"steps/finally/#collectors","title":"Collectors","text":"

A finally statement supports all collectors:

"},{"location":"steps/finally/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try: []\n    catch: []\n    finally:\n    - command:\n        # ...\n    - script:\n        # ...\n    - events:\n        # ...\n    - podLogs:\n        # ...\n    - describe:\n        # ...\n    - sleep:\n        # ...\n
"},{"location":"steps/try/","title":"Try","text":"

A try statement is a sequence of operations executed in the same order they are declared. If an operation fails the entire step is considered failed.

The try statement is at the heart of a test step, it represents what the step is supposed to be about.

catch and finally statements should be viewed as complementary to the try statement.

Continue on error

By default a test step stops executing when an operation fails and the following operations are not executed.

This behaviour can be changed using the continueOnError field, if continueOnError is set to true the step will still be considered failed but execution will continue with the next operations.

"},{"location":"steps/try/#operations","title":"Operations","text":"

A try statement supports all operations:

"},{"location":"steps/try/#example","title":"Example","text":"

Example

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: example\nspec:\n  steps:\n  - try:\n    - command:\n        # ...\n    - sleep:\n        # ...\n    - script:\n        # ...\n    - apply:\n        # ...\n    - assert:\n        # ...\n    - error:\n        # ...\n    - create:\n        # ...\n    catch: []\n    finally: []\n
"},{"location":"tests/","title":"What is a test","text":"

To put it simply, a test can be represented as an ordered sequence of test steps.

Test steps within a test are run sequentially: if any of the test steps fail, the entire test is considered failed.

A test step can consist in one or more operations:

"},{"location":"tests/#different-syntaxes","title":"Different syntaxes","text":"

Chainsaw supports two different test definition syntaxes. Each syntax has pros and cons, see the descriptions below for more details on advantages and pitfalls.

"},{"location":"tests/#manifests-based-syntax","title":"Manifests based syntax","text":"

This is the simplest and less verbose syntax.

You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.

While this syntax is extremly simple, not all operations are supported. For example, delete, command, script and sleep operations are not.

Another strong limitation is that it is not possible to specify additional configuration per test, step or operation.

Finally, this syntax relies heavily on file naming conventions, it can be error prone and makes it hard to reuse files across multiple tests.

"},{"location":"tests/#test-based-syntax","title":"Test based syntax","text":"

The Test based syntax is a more verbose and explicit syntax.

It does not rely on file naming conventions to determine test steps order and allows to easily reuse files accross multiple tests.

This syntax also comes with no limitations to provide additional configuration at the test, step or operation level.

Making a choice

Choosing one syntax over the other is not a trivial choice, every one will have its own preference and/or constraints.

It's usually easier to start with the manifests based syntax. However, as test suites grow and tests become more complex, it is often necessary to configure options on a per test, step or operation basis and the Test based syntax becomes necessary.

Fortunately Chainsaw has a command to automatically migrate from manifest based to Test based syntax.

"},{"location":"tests/#namespaced-resources","title":"Namespaced resources","text":"

Kubernetes organizes resources into two primary scopes: namespaced and cluster-scoped.

While namespaced resources belong to a specific namespace, cluster-scoped resources span across the entire Kubernetes cluster.

"},{"location":"tests/#purpose-of-the-namespacer-interface","title":"Purpose of the Namespacer Interface","text":"

The Namespacer interface ensures automated and consistent namespace assignment to Kubernetes resources.

"},{"location":"tests/#cleanup","title":"Cleanup","text":"

Unless configured differently, by default Chainsaw will automatically cleanup the resources it created after a test finishes. Cleanup happens in reverse order of creation (created last, cleaned up first).

Note that Chainsaw performs a blocking deletion, that is, it will wait the resource is actually not present anymore in the cluster before proceeding with the next resource cleanup.

This is important, especially when the controller being tested makes use of finalizers.

Overriding cleanup timeout

A global cleanup timeout can be defined at the configuration level or using command line flags.

It can also be overriden on a per test or per test step basis but not at the operation level.

"},{"location":"tests/#parallel-execution-of-tests","title":"Parallel Execution of Tests","text":"

While Chainsaw ensures that the steps within a test are executed sequentially, it is designed to run multiple tests in parallel to each other. This parallel execution helps in significantly reducing the overall time required to run an entire suite of tests, making the testing process more efficient, especially in scenarios with a large number of tests.

Parallel execution of tests

By default, Chainsaw will run tests in parallel.

This can be configured at the configuration level or using command line flags. However, individual tests can be configured to run concurrently by setting Concurrent: true in their TestSpec.

All non-concurrent tests are executed first, followed by the concurrent tests in parallel.

"},{"location":"tests/manifests-based/","title":"Manifests based syntax","text":"

This is the simplest and less verbose supported syntax.

You provide bare Kubernetes resource manifests and Chainsaw will use those manifests to create, update, or assert expectations against a cluster.

While this syntax is simple, it doesn't support deletion operations and doesn't allow specifying additional configuration per test or step.

It also relies a lot on file naming conventions and makes it hard to reuse files across multiple tests.

"},{"location":"tests/manifests-based/#file-naming-convention","title":"File naming convention","text":"

Manifest files must follow the naming convention <step index>-<name|assert|errors>.yaml.

As an example 00-configmap.yaml, 01-assert.yaml and 02-errors.yaml can all be considered valid file names.

It's also perfectly valid to have multiple files for the same step. Let's imagine we have the following files 00-resources.yaml, 00-more-resources.yaml, 00-assert.yaml and 00-errors.yaml:

With the four files above, Chainsaw will assemble a test step made of the combination of all those files.

"},{"location":"tests/manifests-based/#manifests-loading-process","title":"Manifests loading process","text":"

The logic to determine the content of a step is always:

"},{"location":"tests/manifests-based/#example","title":"Example","text":""},{"location":"tests/manifests-based/#01-configmapyaml","title":"01-configmap.yaml","text":"

The manifest below contains a config map in a file called 01-configmap.yaml. Chainsaw will associate this manifest with an apply operation in step 01.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\n
"},{"location":"tests/manifests-based/#02-assertyaml","title":"02-assert.yaml","text":"

The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  foo: bar\n
"},{"location":"tests/manifests-based/#03-errorsyaml","title":"03-errors.yaml","text":"

The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: chainsaw-quick-start\ndata:\n  lorem: ipsum\n
"},{"location":"tests/manifests-based/#conclusion","title":"Conclusion","text":"

This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the configmap does not contain the lorem: ipsum data.

Assert and error statements are very similar to standard resource definitions, they share the same structure but can be partial. Chainsaw will process only the elements present in the statements against the actual resource definition.

"},{"location":"tests/test-based/","title":"Test based syntax","text":"

The Test based syntax is more verbose than the manifests based syntax but offers more flexibility and features:

"},{"location":"tests/test-based/#the-test-resource","title":"The Test resource","text":"

A Test resource, like any Kubernetes resource has an apiVersion, kind and metadata section.

It also comes with a spec section used to declaratively represent the steps of a test and other configuration elements belonging to the test being defined.

Reference documentation

The full structure of the Test resource is documented here.

"},{"location":"tests/test-based/#test-steps","title":"Test steps","text":"

A Test is mostly made of test steps. Test steps are detailed in a dedicated documentation.

"},{"location":"tests/test-based/#example","title":"Example","text":""},{"location":"tests/test-based/#chainsaw-testyaml","title":"chainsaw-test.yaml","text":"

The manifest below contains a Test in a file called chainsaw-test.yaml.

Chainsaw will load the Test and steps defined in its spec section.

The Test uses a custom timeouts for the whole test. Note that timeouts could have been overridden in specific steps if needed.

apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  # these timeouts are applied per operation\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step\n  # apply a configmap to the cluster\n  # the path to the configmap is relative to the folder\n  # containing the test, hence allow reusing manifests\n  # across multiple tests\n  - try:\n    - apply:\n        file: ../resources/configmap.yaml\n  # second step\n  # execute assert statements against existing resources\n  # in the cluster\n  - try:\n    - assert:\n        file: ../resources/configmap-assert.yaml\n  # third step\n  # execute error statements against existing resources\n  # in the cluster\n  - try:\n    - error:\n        file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#specifying-a-custom-test-file","title":"Specifying a Custom Test File","text":"

If you have your test defined in a different file, you can specify it when running Chainsaw:

chainsaw test . --test-file=<custom-test-file-name>.yaml\n
"},{"location":"tests/test-based/#raw-resource-support","title":"Raw Resource Support","text":"

Chainsaw now includes the raw resource feature, allowing direct specification of Kubernetes resources within the test definitions.

This feature offers a more streamlined approach for defining resources, especially useful for simpler test scenarios or for cases where resource definitions need to be reused or slightly modified across different tests.

"},{"location":"tests/test-based/#example-raw-resource","title":"Example Raw Resource","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step applies a configmap directly to the cluster\n  - try:\n    - apply:\n        resource:\n          apiVersion: v1\n          kind: ConfigMap\n          metadata:\n            name: chainsaw-quick-start\n          data:\n            foo: bar\n  # second step executes assert statements against existing resources\n  - try:\n    - assert:\n        file: ../resources/configmap-assert.yaml\n  # third step executes error statements against existing resources\n  - try:\n    - error:\n        file: ../resources/configmap-error.yaml\n
"},{"location":"tests/test-based/#url-support-for-file-references","title":"URL Support for File References","text":"

Chainsaw has support for URLs in file references for assert, apply, error and similar operations.

This feature enhances the reach of Chainsaw by allowing users to reference files hosted on remote locations, such as GitHub raw URLs or other web URLs, directly within their test definitions.

"},{"location":"tests/test-based/#example-url-file-reference","title":"Example URL File Reference","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n  name: test-name\nspec:\n  skip: false\n  concurrent: false\n  skipDelete: false\n  timeouts:\n    apply: 10s\n    assert: 10s\n    error: 10s\n  steps:\n  # first step\n  # apply a Kubernetes manifest hosted at a GitHub raw URL\n  - try:\n    - apply:\n        file: https://raw.githubusercontent.com/username/repo/branch/path/to/configmap.yaml\n  # second step\n  # execute assert statements against existing resources\n  # using a file hosted on another web URL\n  - try:\n    - assert:\n        file: https://example.com/path/to/configmap-assert.yaml\n  # third step\n  # execute error statements against existing resources\n  - try:\n    - error:\n        file: https://mywebsite.com/path/to/configmap-error.yaml\n
"},{"location":"blog/archive/2023/","title":"2023","text":""},{"location":"blog/category/writing-tests/","title":"Writing Tests","text":""},{"location":"blog/category/announcements/","title":"announcements","text":""}]} \ No newline at end of file diff --git a/main/sitemap.xml.gz b/main/sitemap.xml.gz index 54c07e995649df97ea7e69f785241ad427e807c0..ea2c303378eb506d63998707a277e0c26336bd0b 100644 GIT binary patch delta 15 Wcmcb}c9D%uzMF$1S8gMlH!}bv?F0V+ delta 15 Wcmcb}c9D%uzMF$XRdOSnH!}buBm=Sl