Namespace label to indicate a namespace managed by kyma

[a-thaler]

Created on 2024.07.22 by Andreas Thaler (@a-thaler ).

Decision log

Name	Description
Title	Align on a label to mark a namespace as managed by kyma
Due date	ASAP
Status	Proposed on 2024-07-22
Decision type	Binary
Affected decisions	-

Context

The different tools for Kyma like the CLI, Dashboard and also telemetry like to treat resources in a managed namespace differently. For example they might to show them "locked" so that novices will not modify them, or as in the case of telemetry, users like to exclude log collection from them as workloads running there are not in the interest of there operational duty.

The goal is to have a simple namespaceSelector in place to identify all managed namespaces, like:

  namespaceSelector:
    matchExpressions:
    - key: operator.kyma-project.io/managed-by:
      operator: NotIn
      values:
      - kyma
    - key: kubernetes.io/metadata.name:
      operator: NotIn
      values:
      - kube-system

Currently there is no easy way to figure out these managed namespaces, modules are spread across different namespaces, modules might be installed in a managed way or custom.

The goal is to have a unified way to know which namespaces are managed by the kyma lifecycle manager in order to smooth the user experience and have a better distinction between reponsibilities.

At the moment, the Kyma Lifecycle Manager already labels all applied resources (incl namespaces) with these labels:

operator.kyma-project.io/managed-by: declarative-v2
operator.kyma-project.io/watched-by: module-manager

Furthermore, it adds an addition to all resources indicating to not change them.

These labels not documented anywhere and probably not really part of any contract. Anyway, these should be re-used, condensed to one meaningful label and written down to make use of them by clients.

Furthermore, it should be mentioned that modules typically manage further resources which are then potentially become a "managed" resource as well. For these resources, we agreed already to have a consistent labelling as decided here.
However, these labels are defined by the modules itself and the module will not differ between if the module was installed in a managed way by the lifecycle manager or not. The "managed" flavor anyway is not so relevant for all kind of resources but mainly namespaces a a grouping element.
To overcome the problem with modules like "Istio" where a namespace gets created dynamically, a contract should be established that all namespaces owned by a module should be part of the moduleTemplate, so that they are known to the lifecycle manager and are getting labeled accordingly.

Gardener is labelling the kube-system namespace using:

resources.gardener.cloud/managed-by: gardener

Decision

Have a consistent labelling on namespace level which indicates that this namespace is under control of the Kyma Lifecycle Manager and with that is a kyma managed namespace. The user do not need to care about the operational health of components running inside this namespaces. The label is called operator.kyma-project.io/managed-by: kyma. It will replace the two labels already used by the lifecycle manager. The kyma-system and kube-system namespace will be labeled always.

All namespaces owned by a module must be defined as part of the ModuleTemplate so that it becomes a "managed" one automatically, incl the istio-system namespace.

Consequences

The approach will allow the different tools to dynamically search or exclude for namespaces having the proposed label. Determined namespaces will be treated differently by the tooling..

[a-thaler]

Relates to #864

[a-thaler]

Very good example is: Users are asked to not auto-instrument namesapces not owned by them. But how can they know that list? Example of a dynatrace CR configuration:

namespaceSelector:
  matchExpressions:
  - key: kubernetes.io/metadata.name
   operator: NotIn
   values:
   - kube-system
   - kyma-system

That list is not complete at all. Now imagine that this list gets replaced by something like:

namespaceSelector:
  matchExpressions:
  - key: kyma-project.io/namespace
   operator: In
   values:
   - managed

[pbochynski]

+1 for having a label added by KLM.
Just reuse the existing labeling (operator.kyma-project.io/managed-by: declarative-v2) but replace it with something more intuitive like:

operator.kyma-project.io/managed-by: kyma

KLM has to make sure that kyma-system namespace will get that label also. Modules that run outside of kyma-system have the namespaces in the deployment yaml , so it will work automatically. The only missing part is istio-system, but we can probably easily fix it by adding istio-system namespace to the deployment yaml of istio module (@strekm please confirm).

[a-thaler]

Even if the kube-system is managed by gardener, for end users the kube-system namespace is just part of the managed kyma offering and it will not be understandable why it should be treated separate in the namespaceSelector:

namespaceSelector:
    matchExpressions:
    - key: [kubernetes.io/metadata.name](http://kubernetes.io/metadata.name)
      operator: NotIn
      values:
      - kube-system
    - key: [operator.kyma-project.io/managed-by](http://operator.kyma-project.io/managed-by):
      operator: NotIn
      values:
      - kyma

I suggest that KLM not only enriches the kyma-system namespace always but also kube-system. I adjusted the proposed decision to it.

[a-thaler]

Issue was discussed with all relevant parties and no one objected, will close it as accepted

[a-thaler]

The work items in KLM are:

• Improve the transparency of managed-by label: Make managed-by label more transparent lifecycle-manager#1821
• Address the watched-by label: Make watched-by label more transparent lifecycle-manager#1822
• Label the kube-system namespace: Label the kube-system namespace with managed-by label lifecycle-manager#1824
  The work items for istio:
• Add istio-system namespace to the module template: Include istio-system namespace in the deployment yaml of istio module istio#997