diff --git a/.docs/LABELS.md b/.docs/LABELS.md index 43c6192f..f9a74ed8 100644 --- a/.docs/LABELS.md +++ b/.docs/LABELS.md @@ -1,6 +1,6 @@ # Labels docs -This docs contains informations on how we use GitHub labels on issues and pull requests. +This docs contains information on how we use GitHub labels on issues and pull requests. ## Labels diff --git a/.docs/README.md b/.docs/README.md index 87d9d3ac..b5e2de53 100644 --- a/.docs/README.md +++ b/.docs/README.md @@ -1,6 +1,6 @@ # Release docs -This docs contains informations for releasing releasing. +This docs contains information for releasing. ## Create a release diff --git a/.github/ISSUE_TEMPLATE/bug.yaml b/.github/ISSUE_TEMPLATE/bug.yaml index 8a910b6a..8e244b53 100644 --- a/.github/ISSUE_TEMPLATE/bug.yaml +++ b/.github/ISSUE_TEMPLATE/bug.yaml @@ -14,7 +14,7 @@ body: description: >- What version of the Kyverno JSON are you running? options: - - 1.0.0 + - 0.1.0 validations: required: true - type: textarea diff --git a/.github/workflows/ct-lint.yaml b/.github/workflows/ct-lint.yaml index 42ca8f0c..f7831b40 100644 --- a/.github/workflows/ct-lint.yaml +++ b/.github/workflows/ct-lint.yaml @@ -28,7 +28,7 @@ jobs: with: python-version: 3.7 - name: Set up chart-testing - uses: helm/chart-testing-action@e8788873172cb653a90ca2e819d79d65a66d4e76 # v2.4.0 + uses: helm/chart-testing-action@b43128a8b25298e1e7b043b78ea6613844e079b1 # v2.6.0 - name: Run chart-testing (lint) run: | set -e diff --git a/README.md b/README.md index 0341f44f..d150363e 100644 --- a/README.md +++ b/README.md @@ -1,428 +1,56 @@ -# kyverno-json +# Kyverno JSON [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=Kyverno%20everywhere%21%20%0A%0AEasily%20validate%20any%20JSON%20or%20YAML%20payload%20using%20Kyverno.%0A%0A&url=https://github.com/kyverno/kyverno-json/%0A%0A&hashtags=kubernetes,devops) -This CLI tool is very similar to the [Kyverno CLI](https://github.com/kyverno/kyverno/tree/main/cmd/cli/kubectl-kyverno) tool. +**Kyverno everywhere! 🎉** -The difference is that this CLI tool can apply policies to abitrary json or yaml payloads. +`kyverno-json` applies Kyverno policies to any JSON or YAML payload. -Policy definition syntax is looks a lot like the [Kyverno policy](https://kyverno.io/docs/kyverno-policies/) definition syntax but is more generic and flexible. -This was needed to allow working with arbitrary payloads, not just [Kubernetes](https://kubernetes.io) ones. -Those differences are detailed in the [section below](#differences-with-with-kyverno-policy-definition-syntax). +![logo](website/docs/static/kyverno-json-horizontal.png) -Additionally, you can provide preprocessing queries in [jmespath](https://jmespath.site) format to preprocess the input payload before evaluating *resources* against policies. -This is necessary if the input payload is not what you want to directly analyse. -Preprocessing is detailed in the following [section](#preprocessing). -## Differences with with Kyverno policy definition syntax +Use Kyverno's powerful, declarative, low-code policies to validate any runtime or configuration data that can be converted to JSON including: +* Terraform files +* Dockerfiles +* Cloud configurations +* Service authorization requests -Sections below highlight the main differences between polcies used by this tool and [Kyverno policies](https://kyverno.io/docs/kyverno-policies/). +Run `kyverno-json` as a CLI, or a web application with a REST API. Or, integrate as a Golang library. -### Different `apiVersion` and `kind` +## 📙 Documentation -Both [Kyverno policies](https://kyverno.io/docs/kyverno-policies/) and policies used by this tool are defined using [Kubernetes](https://kubernetes.io) manifests. +Documentation is available at: https://kyverno.github.io/kyverno-json -They don't use the same `apiVersion` and `kind` though. +👉 **[Quick Start](https://kyverno.github.io/kyverno-json/quick-start/)** -[Kyverno policies](https://kyverno.io/docs/kyverno-policies/) belong to the `kyverno.io` group, exist in multiple versions (`v1`, `v2beta1`) and can be of kind `Policy` or `ClusterPolicy`. +👉 **[Sample Policies](https://kyverno.github.io/kyverno-json/catalog/)** -Policies for this tool belong to the `json.kyverno.io` group, exist only in `v1alpha1` version and can only be of kind `Policy`. +👉 **[Playground](https://kyverno.github.io/kyverno-json/playground/)** -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar-4 - validate: - assert: - all: - - foo: - bar: 4 -``` +## 🙋‍♂️ Getting Help -The concept of clustered vs namespaced resources exist only in the [Kubernetes](https://kubernetes.io) world and it didn't make sense to reproduce the same pattern in this tool. +We are here to help! -### Different `match` and `exclude` statements +👉 For feature requests and bugs, file an [issue](https://github.com/kyverno/kyverno-json/issues). -Both [Kyverno policies](https://kyverno.io/docs/kyverno-policies/) and policies used by this tool can match and exclude *resources* when being evaluated. +👉 For discussions or questions, join the [Kyverno Slack channel](https://slack.k8s.io/#kyverno). -[Kyverno policies](https://kyverno.io/docs/kyverno-policies/) use [Kubernetes](https://kubernetes.io) specific constructs for that matter that didn't map well with arbitrary payloads. +👉 For community meeting access, join the [mailing list](https://groups.google.com/g/kyverno). -This tool uses [assertion trees](#assertion-trees-replace-pattern-matching) to implement `match` and `exclude` statements: +👉 To get notified ⭐️ [star this repository](https://github.com/kyverno/kyverno-json/stargazers). -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: required-s3-tags -spec: - rules: - - name: require-team-tag - match: - any: - - type: aws_s3_bucket - exclude: - any: - - name: bypass-me - validate: - assert: - all: - - values: - tags: - Team: ?* -``` +## ➕ Contributing -In the example above, every *resource* having `type: aws_s3_bucket` will match, and *resources* having `name: bypass-me` will be excluded. +Thanks for your interest in contributing to Kyverno! Here are some steps to help get you started: -### Different `jmesPath` implementation +✔ Read and agree to the [Contribution Guidelines](/CONTRIBUTING.md). -This tool uses [jmespath-community/go-jmespath](https://github.com/jmespath-community/go-jmespath), a more modern implementation than the one used in [Kyverno](https://kyverno.io). +✔ Check out the [good first issues](https://github.com/kyverno/kyverno-json/labels/good%20first%20issue) list. Add a comment with `/assign` to request assignment of the issue. -This implementation supports the `let` feature and this tool leverages it to implement context entries: +✔ Check out the Kyverno [Community page](https://kyverno.io/community/) for other ways to get involved. -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: required-s3-tags -spec: - rules: - - name: require-team-tag - match: - any: - - type: aws_s3_bucket - context: - - name: expectedTeam - variable: Kyverno - validate: - message: Bucket `{{ name }}` ({{ address }}) does not have the required Team tag {{ $expectedTeam }} - assert: - all: - - values: - tags: - Team: ($expectedTeam) -``` +## Developer Documentation -Note that all context entries are lazily evaluated, a context entry will only be evaluated once. They can be used in all [assertion trees](#assertion-trees-replace-pattern-matching), including `match` and `exclude` statements. +Developer documentation can be found in the [.docs](./.docs/) folder. -### No preconditions, pattern operators, anchors or wildcards +## License -Policies used by this tool don't support `preconditions`, pattern operators, anchors or wildcards. - -Most of the time `preconditions` can be replaced by the more flexible `match` and `exclude` statements. - -Pattern operators, anchors and wildcards can be replaced with an improved pattern matching system. -The new pattern matching system is called *assertion trees*, this is detailed [below](#assertion-trees-replace-pattern-matching). - -### Assertion trees replace pattern matching - -[Kyverno policies](https://kyverno.io/docs/kyverno-policies/) started with a declarative approach but slowly adopted the imperative approach too, because of the limitations in the implemented declarative approach. - -This tool tries to be as declarative as possible, for now `forEach`, pattern operators, anchors and wildcards are not supported are not supported. -Hopefully we won't need to adopt an imperative approach anymore. - -Instead, assertion trees can now be used to express complex and dynamic conditions by using [jmespath](https://jmespath.site) expressions. Those expressions represent projections of the being analysed *resource* and the result of this projection is passed to descendants for further analysis. - -All comparisons happen in the leaves of the assertion tree. - -Given the input payload below: - -```yaml -foo: - baz: true - bar: 4 - bat: 6 -``` - -It is now possible to write a validation tree like this: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar-4 - validate: - assert: - all: - - - # project field `foo` onto itself, the content of `foo` becomes the current object for descendants - foo: - - # evaluate expression `(bar > `3`)`, the result becomes the current object for descendants (in this case the result will be a simple boolean) - # then we hit the `true` leaf, comparison happens and we expect the current value to be `true` - (bar > `3`): true - - # evaluate expression `(!baz)`, the result becomes the current object for descendants (in this case the result will be a simple boolean) - # then we hit the `true` leaf, comparison happens and we expect the current value to be `false` - (!baz): false - - # evaluate expression `(bar + bat)`, the result becomes the current object for descendants (in this case the result will be a number) - # then we hit the `10` leaf, comparison happens and we expect the current value to be `10` - (bar + bat): 10 -``` - -#### Projection modifiers - -Assertion tree expressions support modifiers to influence the way projected values are processed. - -The `~` modifier applies to arrays and maps, it mean the input array or map elements will be processed individually by descendants. -When the `~` modifier is not used, descendants receive the whole array, not individual elements. - -Consider the following input document: - -```yaml -foo: - bar: - - 1 - - 2 - - 3 -``` - -The policy below does not use the `~` modifier and `foo.bar` array is compared against the expected array: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar - validate: - assert: - all: - - foo: - # the content of the `bar` field will be compared against `[1, 2, 3]` - bar: - - 1 - - 2 - - 3 -``` - -With the `~` modifier, we can apply descendants to all elements in the array individually. -The policy below ensures that all elements in the input array are `< 5`: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar - validate: - assert: - all: - - foo: - # with the `~` modifier all elements in the `[1, 2, 3]` array are processed individually and passed to descendants - ~.bar: - # the expression `(@ < `5`)` is evaluated for every element and the result is expected to be `true` - (@ < `5`): true -``` - -The `~` modifier supports binding the index of the element being processed to a named binding with the following syntax `~index_name.bar`. When this is used, we can access the element index in descendants with `$index_name`. - -When used with a map, the named binding receives the key of the element being processed. - -#### Explicit bindings - -Sometimes it can be useful to refer to a parent node in the assertion tree. - -This is possible to add an explicit binding at every node in the tree by appending the `@binding_name` to the key. - -Given the input document: - -```yaml -foo: - bar: 4 - bat: 6 -``` - -The following policy will compute a sum and bind the result to the `sum` binding. A descendant can then use `$sum` and use it: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar - validate: - assert: - all: - - foo: - # evaluate expression `(bar + bat)` and bind it to `sum` - (bar + bat)@sum: - # get the `$sum` binding and compare it against `10` - ($sum): 10 -``` - -All binding are available to descendants, if a descendant creates a binding with a name that already exists the binding will be overriden for descendants only and it doesn't affect the bindings at upper levels in the tree. - -In other words, a node in the tree always sees bindings that are definied in the parents and if a name is reused, the first binding with the given name wins when winding up the tree. - -As a consequence, the policy below is perfectly valid: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar - validate: - assert: - all: - - foo: - (bar + bat)@sum: - ($sum + $sum)@sum: - ($sum): 20 - ($sum): 10 -``` - -Note that all context entries are made available to the rule via bindings: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: required-s3-tags -spec: - rules: - - name: require-team-tag - match: - any: - - type: aws_s3_bucket - context: - # creates a `expectedTeam` binding automatically - - name: expectedTeam - variable: Kyverno - validate: - message: Bucket `{{ name }}` ({{ address }}) does not have the required Team tag {{ $expectedTeam }} - assert: - all: - - values: - tags: - # use the `$expectedTeam` binding coming from the context - Team: ($expectedTeam) -``` - -Finally, we can always access the current payload, policy and rule being evaluated using the builtin `$payload`, `$policy` and `$rule` bindings. No protection is made to prevent you from overriding those bindings though. - -#### Escaping projections - -It can be necessary to prevent a projection under certain circumstances. - -Consider the following document: - -```yaml -foo: - (bar): 4 - (baz): - - 1 - - 2 - - 3 -``` - -Here the `(bar)` key conflict with the projection syntax used. -To workaround this issue, you can escape a projection by surrounding it with `\` characters like this: - -```yaml -apiVersion: json.kyverno.io/v1alpha1 -kind: ValidatingPolicy -metadata: - name: test -spec: - rules: - - name: foo-bar - validate: - assert: - all: - - foo: - \(bar)\: 10 -``` - -In this case, the leading and trailing `\` characters will be erased and the projection won't be applied. - -Note that it's still possible to use the `~` modifier or to create a named binding with and escaped projection. - -Keys like this are perfectly valid: -- `~index.\baz\` -- `\baz\@foo` -- `~index.\baz\@foo` - -## SDK - -This CLI tool contains an initial implementation of an SDK to allow flexible creation of dedicated policy engines. - -The [json-engine](./pkg/json-engine/) at the heart of this tool is built by assembling blocks provided by the [engine](./pkg/engine/) SDK. - -## Build kyverno-json - -To build this tool locally, simply run: - -```console -make build -``` - -## Preprocessing - -When the input payload is not what you want to analyse directly, you can provide one or more [jmespath](https://jmespath.site) expressions to preprocess the data. -The policies will be evaluated against the result of the preprocessing step. - -Traditionnally, policies apply to *resources* and this is how this tool implements policy evaluation: - -``` -loop through all resources { - loop through all policies { - loop through all rules in the policy { - evaluate the resource against the rule - } - } -} -``` - -Note that if you provide a single payload, the tool will internally wrap it in an array of one element. - -So imagine an input payload similar to this: - -```yaml -version: 1.2.3 -creationDate: '2023-09-29' -resources: -- type: something - name: foo - spec: - # ... -- type: something else - name: bar - spec: - # ... -``` - -The *resources* you want to analyse are located under the `resources` stanza, and your policies are probably written to work on those *resources*. -In order to extract the data under the `resources` stanza before processing happens you can specify the `--pre-process "resources"` when invoking the tool. - -You can chain mutliple preprocessing queries by specifying the `--pre-process` flag multiple times. -There is no limitation in a preprocessing [jmespath](https://jmespath.site) expression. - -## Invoke kyverno-json - -```console -# with yaml payload -./kyverno-json scan --payload ./testdata/foo-bar/payload.yaml --policy ./testdata/foo-bar/policy.yaml - -# with json payload (and pre processing) -./kyverno-json scan --payload ./testdata/tf-plan/tf.plan.json --pre-process "planned_values.root_module.resources" --policy ./testdata/tf-plan/policy.yaml -``` - -## Documentation - -- User documentation can be found in [docs/user](./docs/user/README.md) -- Dev documentation can be found in [docs/dev](./docs/dev/README.md) +Copyright 2023, the Kyverno project. All rights reserved. kyverno-json is licensed under the [Apache License 2.0](LICENSE). diff --git a/charts/kyverno-json/Chart.yaml b/charts/kyverno-json/Chart.yaml index afd3f39e..a69e335a 100644 --- a/charts/kyverno-json/Chart.yaml +++ b/charts/kyverno-json/Chart.yaml @@ -13,4 +13,5 @@ sources: maintainers: - name: Nirmata url: https://kyverno.io/ + email: cncf-kyverno-maintainers@lists.cncf.io kubeVersion: ">=1.16.0-0" diff --git a/charts/kyverno-json/README.md b/charts/kyverno-json/README.md index 1ec95255..afdfc3a8 100644 --- a/charts/kyverno-json/README.md +++ b/charts/kyverno-json/README.md @@ -85,7 +85,7 @@ Kubernetes: `>=1.16.0-0` | Name | Email | Url | | ---- | ------ | --- | -| Nirmata | | | +| Nirmata | | | ---------------------------------------------- Autogenerated from chart metadata using [helm-docs v1.11.0](https://github.com/norwoodj/helm-docs/releases/v1.11.0) diff --git a/pkg/commands/scan/command.go b/pkg/commands/scan/command.go index 0e29a2a9..4a20502b 100644 --- a/pkg/commands/scan/command.go +++ b/pkg/commands/scan/command.go @@ -15,9 +15,9 @@ func Command() *cobra.Command { RunE: command.run, } cmd.Flags().StringVar(&command.payload, "payload", "", "Path to payload (json or yaml file)") - cmd.Flags().StringSliceVar(&command.preprocessors, "pre-process", nil, "JmesPath expression used to pre process payload") + cmd.Flags().StringSliceVar(&command.preprocessors, "pre-process", nil, "JMESPath expression used to pre process payload") cmd.Flags().StringSliceVar(&command.policies, "policy", nil, "Path to kyverno-json policies") cmd.Flags().StringSliceVar(&command.selectors, "labels", nil, "Labels selectors for policies") - cmd.Flags().StringVar(&command.identifier, "identifier", "", "JmesPath expression used to identify a resource") + cmd.Flags().StringVar(&command.identifier, "identifier", "", "JMESPath expression used to identify a resource") return cmd } diff --git a/pkg/commands/serve/options.go b/pkg/commands/serve/options.go index 89b99007..a205a4de 100644 --- a/pkg/commands/serve/options.go +++ b/pkg/commands/serve/options.go @@ -39,7 +39,7 @@ type clusterFlags struct { } func (c *options) Run(_ *cobra.Command, _ []string) error { - // initialise gin framework + // initialize gin framework gin.SetMode(c.ginFlags.mode) tonic.SetBindHook(tonic.DefaultBindingHookMaxBodyBytes(int64(c.ginFlags.maxBodySize))) // create router diff --git a/pkg/json-engine/engine.go b/pkg/json-engine/engine.go index 7a5a00f5..586acc4a 100644 --- a/pkg/json-engine/engine.go +++ b/pkg/json-engine/engine.go @@ -63,9 +63,9 @@ func New() engine.Engine[JsonEngineRequest, JsonEngineResponse] { } errs, err := assert.MatchAssert(ctx, nil, r.rule.Assert, r.value, r.bindings) if err != nil { - response.Failure = err - } else if err := multierr.Combine(errs...); err != nil { response.Error = err + } else if err := multierr.Combine(errs...); err != nil { + response.Failure = err } return response }). diff --git a/pkg/server/linux.go b/pkg/server/linux.go index 67d615ce..852bdabe 100644 --- a/pkg/server/linux.go +++ b/pkg/server/linux.go @@ -5,6 +5,7 @@ package server import ( "context" "fmt" + "log" "net/http" "time" ) @@ -21,5 +22,6 @@ func Run(_ context.Context, s Server, host string, port int) Shutdown { panic(err) } }() + log.Default().Printf("listening to requests on %s:%d", host, port) return srv.Shutdown } diff --git a/pkg/server/scan/handler.go b/pkg/server/scan/handler.go index 8ae84525..4118738b 100644 --- a/pkg/server/scan/handler.go +++ b/pkg/server/scan/handler.go @@ -56,6 +56,10 @@ func newHandler(policyProvider PolicyProvider) (gin.HandlerFunc, error) { Resources: resources, Policies: pols, }) - return makeResponse(results...), nil + resp, _ := makeResponse(results...) + // if status != http.StatusOK { + // // TODO: handle HTTP status codes + // } + return resp, nil }, http.StatusOK), nil } diff --git a/pkg/server/scan/response.go b/pkg/server/scan/response.go index d6697ab6..b9dfd36b 100644 --- a/pkg/server/scan/response.go +++ b/pkg/server/scan/response.go @@ -1,7 +1,8 @@ package scan import ( - "github.com/kyverno/kyverno-json/pkg/apis/v1alpha1" + "net/http" + jsonengine "github.com/kyverno/kyverno-json/pkg/json-engine" ) @@ -9,18 +10,60 @@ type Response struct { Results []Result `json:"results"` } +type PolicyResult string + type Result struct { - Policy *v1alpha1.ValidatingPolicy `json:"policy"` - Rule v1alpha1.ValidatingRule `json:"rule"` - Resource interface{} `json:"resource"` - Failure error `json:"failure"` - Error error `json:"error"` + PolicyName string `json:"policy"` + RuleName string `json:"rule"` + Result PolicyResult `json:"status"` + Message string `json:"message"` } -func makeResponse(responses ...jsonengine.JsonEngineResponse) *Response { +// Status specifies state of a policy result +const ( + StatusPass PolicyResult = "pass" + StatusFail PolicyResult = "fail" + StatusWarn PolicyResult = "warn" + StatusError PolicyResult = "error" + StatusSkip PolicyResult = "skip" +) + +func makeResponse(responses ...jsonengine.JsonEngineResponse) (*Response, int) { var response Response - for _, result := range responses { - response.Results = append(response.Results, Result(result)) + failCount := 0 + errorCount := 0 + for _, r := range responses { + status, msg := getStatusAndMessage(r) + if status == StatusError { + errorCount++ + } else if status == StatusFail { + failCount++ + } + + response.Results = append(response.Results, Result{ + PolicyName: r.Policy.Name, + RuleName: r.Rule.Name, + Result: status, + Message: msg, + }) + } + + httpStatus := http.StatusOK + if failCount > 0 { + httpStatus = http.StatusForbidden + } else if errorCount > 0 { + httpStatus = http.StatusNotAcceptable + } + + return &response, httpStatus +} + +func getStatusAndMessage(r jsonengine.JsonEngineResponse) (PolicyResult, string) { + if r.Error != nil { + return StatusError, r.Error.Error() + } + if r.Failure != nil { + return StatusFail, r.Failure.Error() } - return &response + return StatusPass, "" } diff --git a/pkg/server/scan/routes.go b/pkg/server/scan/routes.go index d52dd39f..03c248bb 100644 --- a/pkg/server/scan/routes.go +++ b/pkg/server/scan/routes.go @@ -1,6 +1,8 @@ package scan import ( + "log" + "github.com/gin-gonic/gin" ) @@ -10,5 +12,6 @@ func AddRoutes(group *gin.RouterGroup, policyProvider PolicyProvider) error { return err } group.POST("/scan", handler) + log.Default().Printf("configured route %s/%s", group.BasePath(), "scan") return nil } diff --git a/test/commands/scan/dockerfile/out.txt b/test/commands/scan/dockerfile/out.txt index 4a4a78e7..cb306c06 100644 --- a/test/commands/scan/dockerfile/out.txt +++ b/test/commands/scan/dockerfile/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- check-dockerfile / deny-external-calls / (unknown) FAILED: HTTP calls are not allowed: all[0].check.~.(Stages[].Commands[].Args[].Value)[0].(contains(@, 'https://') || contains(@, 'http://')): Invalid value: true: Expected value: false; wget is not allowed: all[3].check.~.(Stages[].Commands[].CmdLine[])[0].(contains(@, 'wget')): Invalid value: true: Expected value: false +- check-dockerfile / deny-external-calls / (unknown) ERROR: HTTP calls are not allowed: all[0].check.~.(Stages[].Commands[].Args[].Value)[0].(contains(@, 'https://') || contains(@, 'http://')): Invalid value: true: Expected value: false; wget is not allowed: all[3].check.~.(Stages[].Commands[].CmdLine[])[0].(contains(@, 'wget')): Invalid value: true: Expected value: false Done diff --git a/test/commands/scan/payload-yaml/out.txt b/test/commands/scan/payload-yaml/out.txt index 9a806e35..ce39dff0 100644 --- a/test/commands/scan/payload-yaml/out.txt +++ b/test/commands/scan/payload-yaml/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- required-s3-tags / require-team-tag / aws_s3_bucket.example FAILED: Bucket `example` (aws_s3_bucket.example) does not have the required tags {"Team":"Kyverno"}: all[0].check.values.tags: Invalid value: map[string]interface {}{"Environment":"Dev", "Name":"My bucket"}: Expected value: map[string]interface {}{"Team":"Kyverno"} +- required-s3-tags / require-team-tag / aws_s3_bucket.example ERROR: Bucket `example` (aws_s3_bucket.example) does not have the required tags {"Team":"Kyverno"}: all[0].check.values.tags: Invalid value: map[string]interface {}{"Environment":"Dev", "Name":"My bucket"}: Expected value: map[string]interface {}{"Team":"Kyverno"} Done diff --git a/test/commands/scan/pod-no-latest/out.txt b/test/commands/scan/pod-no-latest/out.txt index 2e8b82bc..cbd06991 100644 --- a/test/commands/scan/pod-no-latest/out.txt +++ b/test/commands/scan/pod-no-latest/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- test / pod-no-latest / webserver FAILED: [all[0].check.spec.~foo.containers->foos[0].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false, all[0].check.spec.~foo.containers->foos[1].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false, all[0].check.spec.~foo.containers->foos[2].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false]; [all[1].check.spec.~.containers->foo[0].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[1].check.spec.~.containers->foo[1].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[1].check.spec.~.containers->foo[2].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false]; [all[2].check.~index.(spec.containers[*].image)->images[0].(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[2].check.~index.(spec.containers[*].image)->images[1].(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[2].check.~index.(spec.containers[*].image)->images[2].(ends_with(@, ':latest')): Invalid value: true: Expected value: false] +- test / pod-no-latest / webserver ERROR: [all[0].check.spec.~foo.containers->foos[0].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false, all[0].check.spec.~foo.containers->foos[1].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false, all[0].check.spec.~foo.containers->foos[2].(at($foos, $foo).image)->foo.(ends_with($foo, $tag)): Invalid value: true: Expected value: false]; [all[1].check.spec.~.containers->foo[0].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[1].check.spec.~.containers->foo[1].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[1].check.spec.~.containers->foo[2].image.(ends_with(@, ':latest')): Invalid value: true: Expected value: false]; [all[2].check.~index.(spec.containers[*].image)->images[0].(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[2].check.~index.(spec.containers[*].image)->images[1].(ends_with(@, ':latest')): Invalid value: true: Expected value: false, all[2].check.~index.(spec.containers[*].image)->images[2].(ends_with(@, ':latest')): Invalid value: true: Expected value: false] Done diff --git a/test/commands/scan/tf-plan/out.txt b/test/commands/scan/tf-plan/out.txt index 9a806e35..ce39dff0 100644 --- a/test/commands/scan/tf-plan/out.txt +++ b/test/commands/scan/tf-plan/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- required-s3-tags / require-team-tag / aws_s3_bucket.example FAILED: Bucket `example` (aws_s3_bucket.example) does not have the required tags {"Team":"Kyverno"}: all[0].check.values.tags: Invalid value: map[string]interface {}{"Environment":"Dev", "Name":"My bucket"}: Expected value: map[string]interface {}{"Team":"Kyverno"} +- required-s3-tags / require-team-tag / aws_s3_bucket.example ERROR: Bucket `example` (aws_s3_bucket.example) does not have the required tags {"Team":"Kyverno"}: all[0].check.values.tags: Invalid value: map[string]interface {}{"Environment":"Dev", "Name":"My bucket"}: Expected value: map[string]interface {}{"Team":"Kyverno"} Done diff --git a/test/commands/scan/tf-s3/out.txt b/test/commands/scan/tf-s3/out.txt index de8bef93..76f5a078 100644 --- a/test/commands/scan/tf-s3/out.txt +++ b/test/commands/scan/tf-s3/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- s3 / check-tags / (unknown) FAILED: all[0].check.planned_values.root_module.~.resources[0].values.(keys(tags_all)).(contains(@, 'Team')): Invalid value: false: Expected value: true +- s3 / check-tags / (unknown) ERROR: all[0].check.planned_values.root_module.~.resources[0].values.(keys(tags_all)).(contains(@, 'Team')): Invalid value: false: Expected value: true Done diff --git a/test/commands/scan/wildcard/out.txt b/test/commands/scan/wildcard/out.txt index 0f416f3c..bc4dae44 100644 --- a/test/commands/scan/wildcard/out.txt +++ b/test/commands/scan/wildcard/out.txt @@ -2,5 +2,5 @@ Loading policies ... Loading payload ... Pre processing ... Running ( evaluating 1 resource against 1 policy ) ... -- required-s3-tags / require-team-tag / bucket1 FAILED: all[0].check.tags.(wildcard('?*', Team)): Invalid value: true: Expected value: false +- required-s3-tags / require-team-tag / bucket1 ERROR: all[0].check.tags.(wildcard('?*', Team)): Invalid value: true: Expected value: false Done diff --git a/website/docs/cli/commands/kyverno-json_scan.md b/website/docs/cli/commands/kyverno-json_scan.md index ed3a701d..e5829940 100644 --- a/website/docs/cli/commands/kyverno-json_scan.md +++ b/website/docs/cli/commands/kyverno-json_scan.md @@ -14,11 +14,11 @@ kyverno-json scan [flags] ``` -h, --help help for scan - --identifier string JmesPath expression used to identify a resource + --identifier string JMESPath expression used to identify a resource --labels strings Labels selectors for policies --payload string Path to payload (json or yaml file) --policy strings Path to kyverno-json policies - --pre-process strings JmesPath expression used to pre process payload + --pre-process strings JMESPath expression used to pre process payload ``` ### SEE ALSO diff --git a/website/docs/cli/index.md b/website/docs/cli/index.md index ce137f78..93862c89 100644 --- a/website/docs/cli/index.md +++ b/website/docs/cli/index.md @@ -1,9 +1,92 @@ -# Usage +# Overview -tbd... +The `kyverno-json` Command Line Interface (CLI) can be used to: -## Pre-processing +* scan JSON or YAML files +* launch a web application with a REST API +* launch a playground -Additionally, you can provide preprocessing queries in [jmespath](https://jmespath.site) format to pre-process the input payload before evaluating *resources* against policies. +Here is an example of scanning an Terraform plan that creates an S3 bucket: -This is necessary if the input payload is not what you want to directly analyse. +```sh +./kyverno-json scan --policy test/commands/scan/tf-s3/policy.yaml --payload test/commands/scan/tf-s3/payload.json +``` + +The output looks like: + +```sh +Loading policies ... +Loading payload ... +Pre processing ... +Running ( evaluating 1 resource against 1 policy ) ... +- s3 / check-tags / (unknown) FAILED: all[0].check.planned_values.root_module.~.resources[0].values.(keys(tags_all)).(contains(@, 'Team')): Invalid value: false: Expected value: true +Done +``` + +## Installation + +See [Install](../install.md) for the available options to install the CLI. + +## Pre-processing payloads + +You can provide preprocessing queries in [jmespath](https://jmespath.site) format to pre-process the input payload before evaluating *resources* against policies. + +This is necessary if the input payload is not what you want to directly analyze. + +For example, here is a partial JSON which was produced by converting a Terraform plan that creates an EC2 instance: + +[kyverno/kyverno-json/main/test/commands/scan/tf-ec2/payload.json](https://github.com/kyverno/kyverno-json/blob/main/test/commands/scan/tf-ec2/payload.json) + +```json +{ + "format_version": "1.2", + "terraform_version": "1.5.7", + "planned_values": { + "root_module": { + "resources": [ + { + "address": "aws_instance.app_server", + "mode": "managed", + "type": "aws_instance", + "name": "app_server", + "provider_name": "registry.terraform.io/hashicorp/aws", + "schema_version": 1, + "values": { + "ami": "ami-830c94e3", + "credit_specification": [], + "get_password_data": false, + "hibernation": null, + "instance_type": "t2.micro", + "launch_template": [], + "source_dest_check": true, + "tags": { + "Name": "ExampleAppServerInstance" + }, + "tags_all": { + "Name": "ExampleAppServerInstance" + }, + "timeouts": null, + "user_data_replace_on_change": false, + "volume_tags": null + }, + + ... + +``` + +To directly scan the `resources` element use `--pre-process planned_values.root_module.resources` as follows: + +```sh +./kyverno-json scan --policy test/commands/scan/tf-ec2/policy.yaml --payload test/commands/scan/tf-ec2/payload.json --pre-process planned_values.root_module.resources +``` + +This command will produce the output: + +```sh +Loading policies ... +Loading payload ... +Pre processing ... +Running ( evaluating 1 resource against 1 policy ) ... +- required-ec2-tags / require-team-tag / (unknown) PASSED +Done +``` diff --git a/website/docs/install.md b/website/docs/install.md index bfcb6f49..eb8190eb 100644 --- a/website/docs/install.md +++ b/website/docs/install.md @@ -2,7 +2,7 @@ You can install the pre-compiled binary (in several ways), or compile from source. -## Install using `go install` +## Using `go install` You can install with `go install` with: @@ -10,7 +10,7 @@ You can install with `go install` with: go install github.com/kyverno/kyverno-json@latest ``` -## Manually +## Download binary Download the pre-compiled binaries from the [releases page](https://github.com/kyverno/kyverno-json/releases) and copy them to the desired location. diff --git a/website/docs/intro.md b/website/docs/intro.md index 0a21d5f3..1ee62fc6 100644 --- a/website/docs/intro.md +++ b/website/docs/intro.md @@ -1,11 +1,11 @@ # Introduction -`kyverno-json` allows any data in JSON (or YAML) format data to be validated with Kyverno policies. For example, you can now use Kyverno policies to validate: +`kyverno-json` extends Kyverno policies to perform simple and efficient validation of data in JSON or YAML format. With `kyverno-json`, you can now use Kyverno policies to validate: - Terraform files - Dockerfiles - Cloud configurations -- Service authorization requests +- Authorization requests Simply convert your runtime or configuration data to JSON, and use Kyverno to audit or enforce policies for security and best practices compliance. @@ -14,4 +14,3 @@ Simply convert your runtime or configuration data to JSON, and use Kyverno to au 1. [A Command Line Interface (CLI)](./cli/index.md) 2. [A web application with a REST API](./webapp/index.md) 3. [A Golang library](./go-library/index.md) - diff --git a/website/docs/policies/policies.md b/website/docs/policies/policies.md index 8db33333..3d18dec1 100644 --- a/website/docs/policies/policies.md +++ b/website/docs/policies/policies.md @@ -2,7 +2,7 @@ Kyverno policies are [Kubernetes resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and can be easily managed via Kubernetes APIs, GitOps workflows, and other existing tools. -However, policies that apply to JSON payload have a few differences from Kyverno policies that are applied to Kubernetes resources at admission controls. +Policies that apply to JSON payload have a few differences from Kyverno policies that are applied to Kubernetes resources at admission controls. ## Resource Scope @@ -65,7 +65,7 @@ A policy rule can contain `context` entries are made available to the rule via b ```yaml apiVersion: json.kyverno.io/v1alpha1 -kind: Policy +kind: ValidatingPolicy metadata: name: required-s3-tags spec: diff --git a/website/docs/static/kyverno-json-horizontal.png b/website/docs/static/kyverno-json-horizontal.png new file mode 100644 index 00000000..db5e739c Binary files /dev/null and b/website/docs/static/kyverno-json-horizontal.png differ diff --git a/website/docs/static/kyverno-json-logo.pptx b/website/docs/static/kyverno-json-logo.pptx index d7f2effb..f991ebde 100644 Binary files a/website/docs/static/kyverno-json-logo.pptx and b/website/docs/static/kyverno-json-logo.pptx differ diff --git a/website/docs/webapp/index.md b/website/docs/webapp/index.md index b878927e..1ce24b16 100644 --- a/website/docs/webapp/index.md +++ b/website/docs/webapp/index.md @@ -1,4 +1,83 @@ # Usage +`kyverno-json` can be deployed as a web application with a REST API. This is useful for deployments when a long running service that processes policy requests is desired. -tbd... \ No newline at end of file +## Managing Policies + +With `kyverno-json` policies are managed as Kubernetes resources. This means that you can use Kubernetes APIs, `kubectl`, GitOps, or any other Kubernetes management tool to manage policies. + +## Usage + +Here is a complete demonstration of how to use `kyverno-json` as an web application: + +**Install CRDs** + +Install the CRD for `kyverno-json`: + +```sh +kubectl apply -f .crds/json.kyverno.io_validatingpolicies.yaml +``` + +**Install policies:** + +Install a sample policy: + +```sh +kubectl apply -f test/commands/scan/dockerfile/policy.yaml +``` + +**Prepare the payload** + +The payload is a JSON object with two fields: + +| Name | Type | Required | +| --------------- | ---------------- | ------------ | +| `payload` | Object | Y | +| `preprocessors` | Array of Strings | N | + + +You can construct a sample payload for the Dockerfile policy using: + +```sh +cat test/commands/scan/dockerfile/payload.json | jq '{"payload": .}' > /tmp/webapp-payload.json +``` + +Run the web application + +```sh +./kyverno-json serve +``` + +This will show the output: + +```sh +2023/10/29 23:46:11 configured route /api/scan +2023/10/29 23:46:11 listening to requests on 0.0.0.0:8080 +``` + +Send the REST API request + +```sh +curl http://localhost:8080/api/scan -X POST -H "Content-Type: application/json" -d @/tmp/webapp-payload.json | jq +``` + +The configured policies will be applied to the payload and the results will be returned back: + +```sh +{ + "results": [ + { + "policy": "check-dockerfile", + "rule": "deny-external-calls", + "status": "fail", + "message": "HTTP calls are not allowed: all[0].check.~.(Stages[].Commands[].Args[].Value)[0].(contains(@, 'https://') || contains(@, 'http://')): Invalid value: true: Expected value: false; wget is not allowed: all[3].check.~.(Stages[].Commands[].CmdLine[])[0].(contains(@, 'wget')): Invalid value: true: Expected value: false" + } + ] +} +``` + +## Helm Chart + +The web application can be installed and managed in a Kubernetes cluster using Helm. + +See details at: https://github.com/kyverno/kyverno-json/tree/main/charts/kyverno-json