Policies being enforced despite being in Audit mode

[kirkpabk]

Kyverno Version

1.9.x

Kubernetes Version

1.22.x

Kubernetes Platform

K3d

Description

Understanding that 1.9.2 is not fully "supported", would there be any reason to cause a basic Audit only policy from your default kyverno/policies samples to be Enforced? In other words, it seems that policies set to Audit are Enforced regardless. Example: disallow-privilege-escalation

Steps to reproduce

1. Enabled disallow-privilege-escalation policy as is.
2. Helm install containing a busybox for inits (e.g., sonarqube)
3. Postgres DB failed to init due to securityContext.allowPrivilegeEscallation must be set to 'false'
4. But it shows a Warning PolicyViolation, but is still blocking as an validationFailureAction: Audit??

Expected behavior

Allowed action, but showing the warning.

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[chipzoller]

The policy must be in enforce mode as it's not possible for an audit-mode policy to produce this result. You could have multiple policies in play here. Check across the entire cluster with the command kubectl get cpol,pol -A -o wide

[kirkpabk]

@chipzoller -- absolutely agree. I saw that they were audits but had to re-apply via source control since I had removed them temporarily to get through the current install.

clusterpolicy.kyverno.io/disallow-privilege-escalation true Audit true 16s 1 0 0 0
clusterpolicy.kyverno.io/disallow-privileged-containers true Audit true 18s 1 0 0 0

But had the same result on the install. Was being blocked on privileged-containers, check-for-labels, require-drop-all, check-seccomp-strict, run-as-non-root, and require-ro-rootfs. We don't have all of the policies enabled, but everything is in audit for now. I verified in source control. I verified by the command you provided. Strange, I know, but SonarQube certainly will not run the init pods with those policies applied in Audit.

Here's the default one "described" but includes the excludes for kube-system and kyverno which we secure differently:

The policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privilege-escalation
  annotations:
    policies.kyverno.io/title: Disallow Privilege Escalation
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    kyverno.io/kyverno-version: 1.6.0
    kyverno.io/kubernetes-version: "1.22-1.23"
    policies.kyverno.io/description: >-
      Status - for review 
            
      Privilege escalation...
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: privilege-escalation
      match:
        any:
        - resources:
            kinds:
              - Pod
      validate:
        message: >-
          Privilege escalation is disallowed. The fields
          spec.containers[*].securityContext.allowPrivilegeEscalation,
          spec.initContainers[*].securityContext.allowPrivilegeEscalation,
          and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
          must be set to false.
        pattern:
          spec:
            =(ephemeralContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            =(initContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            containers:
            - securityContext:
                allowPrivilegeEscalation: "false"
      # Adding an explicit exclude for certain namespaces 
      # until we have the desired outcomes and expectations with new policies.
      exclude: 
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno

The described result:

kubectl describe cpol disallow-privilege-escalation
Name:         disallow-privilege-escalation
Namespace:
Labels:       <none>
Annotations:  kyverno.io/kubernetes-version: 1.22-1.23
              kyverno.io/kyverno-version: 1.6.0
              policies.kyverno.io/category: Pod Security Standards (Restricted)
              policies.kyverno.io/description:
                Status - for review

                Privilege escalation....
              policies.kyverno.io/severity: medium
              policies.kyverno.io/subject: Pod
              policies.kyverno.io/title: Disallow Privilege Escalation
API Version:  kyverno.io/v1
Kind:         ClusterPolicy
Metadata:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    API Version:  kyverno.io/v1
    Fields Type:  FieldsV1
    Manager:         kyverno
    Operation:       Update
    Subresource:     status
Spec:
  Background:  true
  Rules:
    Exclude:
      Any:
        Resources:
          Namespaces:
            kube-system
            kyverno
    Match:
      Any:
        Resources:
          Kinds:
            Pod
    Name:  privilege-escalation
    Validate:
      Message:  Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation, spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation must be set to 'false'.
      Pattern:
        Spec:
          =(ephemeralContainers):
            Security Context:
              Allow Privilege Escalation:  false
          =(initContainers):
            Security Context:
              Allow Privilege Escalation:  false
          Containers:
            Security Context:
              Allow Privilege Escalation:  false
  Validation Failure Action:               Audit
Status:
  Autogen:
    Rules:
      Exclude:
        Any:
          Resources:
            Namespaces:
              kube-system
              kyverno
        Resources:
      Generate:
        Clone:
        Clone List:
      Match:
        Any:
          Resources:
            Kinds:
              DaemonSet
              Deployment
              Job
              StatefulSet
              ReplicaSet
              ReplicationController
        Resources:
      Mutate:
      Name:  autogen-privilege-escalation
      Validate:
        Message:  Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation, spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation must be set to 'false'.
        Pattern:
          Spec:
            Template:
              Spec:
                =(ephemeralContainers):
                  Security Context:
                    Allow Privilege Escalation:  false
                =(initContainers):
                  Security Context:
                    Allow Privilege Escalation:  false
                Containers:
                  Security Context:
                    Allow Privilege Escalation:  false
      Exclude:
        Any:
          Resources:
            Namespaces:
              kube-system
              kyverno
        Resources:
      Generate:
        Clone:
        Clone List:
      Match:
        Any:
          Resources:
            Kinds:
              CronJob
        Resources:
      Mutate:
      Name:  autogen-cronjob-privilege-escalation
      Validate:
        Message:  Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation, spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation must be set to `false`.
        Pattern:
          Spec:
            Job Template:
              Spec:
                Template:
                  Spec:
                    =(ephemeralContainers):
                      Security Context:
                        Allow Privilege Escalation:  false
                    =(initContainers):
                      Security Context:
                        Allow Privilege Escalation:  false
                    Containers:
                      Security Context:
                        Allow Privilege Escalation:  false
  Conditions:
    Message:
    Reason:                Succeeded
    Status:                True
    Type:                  Ready
  Ready:                   true
  Rulecount:
    Generate:      0
    Mutate:        0
    Validate:      1
    Verifyimages:  0
Events:
  Type     Reason           Age                     From          Message
  ----     ------           ----                    ----          -------
  .
  .
  Warning  PolicyViolation  7m59s                   kyverno-scan  Pod a1/busybox: [privilege-escalation] fail
  Warning  PolicyViolation  7m55s                   kyverno-scan  Pod sonarqube/sonarqube-postgresql-0: [privilege-escalation] fail
  .
  .

So, even though it only shows Warning. The failed actions could not continue until the policy was removed. Which I thought was strange.

[chipzoller]

How do I reproduce this on my end?

[kirkpabk]

1. Applied the policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privilege-escalation
  annotations:
    policies.kyverno.io/title: Disallow Privilege Escalation
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    kyverno.io/kyverno-version: 1.6.0
    kyverno.io/kubernetes-version: "1.22-1.23"
    policies.kyverno.io/description: >-
      Status - for review 
            
      Privilege escalation...
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: privilege-escalation
      match:
        any:
        - resources:
            kinds:
              - Pod
      validate:
        message: >-
          Privilege escalation is disallowed. The fields
          spec.containers[*].securityContext.allowPrivilegeEscalation,
          spec.initContainers[*].securityContext.allowPrivilegeEscalation,
          and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
          must be set to false.
        pattern:
          spec:
            =(ephemeralContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            =(initContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            containers:
            - securityContext:
                allowPrivilegeEscalation: "false"
      # Adding an explicit exclude for certain namespaces 
      # until we have the desired outcomes and expectations with new policies.
      exclude: 
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno

2. Follow sonarqube install instructions, here
3. Instructions are explicit regarding Pod Security Levels for Basic and Restricted. Even though they're marked as Audit, it seems to still be blocked.
4. Check with

kubectl describe cpol disallow-privilege-escalation

Strange if I'm the only one seeing this--perhaps I'll rebuild on another machine to duplicate...

[chipzoller]

Followed instructions as provided, the chart was successfully installed and Sonarqube came into a running state. As expected, violations were produced from the installation not being compliant with the policy. Here's the Policy Report for that policy and in the sonarqube Namespace:

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
  creationTimestamp: "2023-08-04T16:33:47Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kyverno
    cpol.kyverno.io/disallow-privilege-escalation: "17169075"
  name: cpol-disallow-privilege-escalation
  namespace: sonarqube
  resourceVersion: "17169445"
  uid: 159bc772-aa9c-4575-9dbd-61741ce0b6a5
results:
- category: Pod Security Standards (Restricted)
  message: 'validation error: Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation,
    spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
    must be set to false. rule autogen-privilege-escalation failed at path /spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation/'
  policy: disallow-privilege-escalation
  resources:
  - apiVersion: apps/v1
    kind: StatefulSet
    name: sonarqube-postgresql
    namespace: sonarqube
    uid: 331db20f-1ec8-4384-a0cd-6bb20fc662dc
  result: fail
  rule: autogen-privilege-escalation
  scored: true
  severity: medium
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1691166827
- category: Pod Security Standards (Restricted)
  message: 'validation error: Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation,
    spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
    must be set to false. rule autogen-privilege-escalation failed at path /spec/template/spec/initContainers/0/securityContext/'
  policy: disallow-privilege-escalation
  resources:
  - apiVersion: apps/v1
    kind: StatefulSet
    name: sonarqube-sonarqube
    namespace: sonarqube
    uid: 6fb41240-b9a1-4612-be70-053ae9fd51e2
  result: fail
  rule: autogen-privilege-escalation
  scored: true
  severity: medium
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1691166827
- category: Pod Security Standards (Restricted)
  message: 'validation error: Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation,
    spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
    must be set to false. rule privilege-escalation failed at path /spec/containers/0/securityContext/allowPrivilegeEscalation/'
  policy: disallow-privilege-escalation
  resources:
  - apiVersion: v1
    kind: Pod
    name: sonarqube-postgresql-0
    namespace: sonarqube
    uid: d6d93d14-2492-4fd7-961d-53e6b33d74c9
  result: fail
  rule: privilege-escalation
  scored: true
  severity: medium
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1691166797
- category: Pod Security Standards (Restricted)
  message: 'validation error: Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation,
    spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
    must be set to false. rule privilege-escalation failed at path /spec/initContainers/0/securityContext/'
  policy: disallow-privilege-escalation
  resources:
  - apiVersion: v1
    kind: Pod
    name: sonarqube-sonarqube-0
    namespace: sonarqube
    uid: f375d1de-4c2c-43d1-93ef-972b543c6e99
  result: fail
  rule: privilege-escalation
  scored: true
  severity: medium
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1691166797
summary:
  error: 0
  fail: 4
  pass: 0
  skip: 0
  warn: 0

[kirkpabk]

Hey @chipzoller, thanks for taking the time to double-check. Something must be skewed on my side. While I get the fail, I SonarQube fails to go into the running state until the policies are removed. I'll keep looking. The only thing I can point to this is a breaking change in 1.9.2 in regards to K8S 1.22. So, in order to go further (confirm), I may have to move down to 1.7.0... which really bugs me. Do you have additional insight on this? Confirm/Deny sort of answer? Thanks again!

[chipzoller]

From what you've told me, no, and for what it's worth, you are the only person who has ever reported some sort of "phantom enforcement" behavior despite matching validate policies in Audit mode. So, given that, I think there is something environment on your side and it's probably not related to Kubernetes v1.22.

[chipzoller]

Converted to discussion as this is not a policy issue.

[chipzoller]

Is this question resolved?