Δ-controller is a framework to simplify the design, implementation and maintenance of Kubernetes operators. The main goal is to reduce the mental overhead of writing Kubernetes operators by providing simple automations to eliminate some of the repetitive code that must be written when coding against the Kubernetes controller-runtime project. The final goal is to let anyone with minimal Go skills to write complex Kubernetes operators in the NoCode/LowCode style.
The main design philosophy behind Δ-controller is to view the Kubernetes API as a gigantic NoSQL database and implement a custom query language that can be used to process Kubernetes objects into either a simplified internal representation that we call a view, or back into actual native Kubernetes resources. This is much akin to MongoDB, so users familiar with document-oriented databases and declarative queries will feel at home when using Δ-controller.
The Δ in the name stands for declarative and delta.
First, Δ-controller implements a declarative query language (inspired by MongoDB aggregators) that can be used to combine, filter and transform Kubernetes API resources into a shape that better fits the operators' internal logic, and from there back to native Kubernetes resources. This is done by registering a processing pipeline to map the Kubernetes API resources into a view of interest to the controller. Views are dynamically maintained by Δ-controller by running the aggregation pipeline on the watch events automatically installed for the source Kubernetes API resources of the aggregation pipeline.
The literal delta in the name connotes that operators in Δ-controller use incremental reconciliation style control loops. This means that the controllers watch the incremental changes to Kubernetes API resources and internal views and act only on the deltas. This simplifies writing edge-triggered conrollers.
Deploy Δ-controller with Helm:
helm repo add dcontroller https://hsnlab.github.io/dcontroller/
helm repo update
helm install dcontroller dcontroller/dcontrollerThis will deploy the Δ-controller operator, a Kubernetes operator that can take custom resources describing a declarative controller and implement the logics on the Kubernetes API.
We will create a simple operator below that will write into each pod the number of containers in the pod as a custom annotation. To simplify the operations, we will restrict the operator to the default namespace. Whenever the container number is updated, the annotation will be updated too.
For here for more complex examples.
First we create the operator that will implement the reconcile logic:
kubectl apply -f - <<EOF
apiVersion: dcontroller.io/v1alpha1
kind: Operator
metadata:
name: pod-container-num-annotator
spec:
controllers:
- name: pod-container-num-annotator
sources:
- apiGroup: ""
kind: Pod
namespace: default
pipeline:
"@aggregate":
- "@project":
metadata:
name: "$.metadata.name"
namespace: "$.metadata.namespace"
annotations:
"dcontroller.io/container-num":
'@string':
'@len': ["$.spec.containers"]
target:
apiGroup: ""
kind: Pod
type: Patcher
EOFThe operator defines a single controller. The controller name indicates a unique identifier that
is used to refer to individual controllers inside the operator. The sources field specifies the
Kubernetes API resource(s) the operator watches, and the target spec describes the API resource the
operator will write. This time, both the source and target will be Pods. The target type is
Patcher, which means that the controller will use the processing pipeline output to patch the
target. (In contrast, Updater targets will simply overwrite the target.)
The most important field is the pipeline, which describes a declarative pipeline to process the
source API resource(s) into the target resource. The pipeline operates on the fields of the source
and the target resources using standard JSONPath
notation.
"@aggregate":
- "@project":
metadata:
name: "$.metadata.name"
namespace: "$.metadata.namespace"
annotations:
"dcontroller.io/container-num":
'@string':
'@len': ["$.spec.containers"]Our sample pipeline comprises a single aggregation operation, namely a projection. This @project
op will create a patch by copying the pod's name and namespace and adding the length of the
containers list in the Pod spec as an annotation to the metadata. The @string op is included to
explicitly cast the numeric length value into a string.
Suppose the following pod resource:
apiVersion: v1
kind: Pod
metadata:
name: my-pod
namespace: default
spec:
containers:
- name: container-1
image: my-container-1:latest
...
- name: container-2
image: my-container-2:latest
...
- name: container-3
image: my-container-3:latest
...Then, our pipeline will generate the below patch, which the controller will then readily apply to patch the target pod.
metadata:
name: my-pod
namespaace: default
annotations:
"dcontroller.io/container-num": "3"And that's the whole idea: you specify one or more Kubernetes API resources to watch, a declarative pipeline that will process the input resources into a patch that is then automatically applied to the target resource. The Δ-controller operators are completely dynamic so you can add, delete and modify them anytime, they are fully described in a single custom resource with the entire logics, there can be any number of operators running in parallel (but make sure they do not create infinite update cycles!), and the entire framework involves zero line of code. This is NoCode/LowCode at its best!
We will test the operator with a sample pod:
kubectl run net-debug --image=docker.io/l7mp/net-debug:latestCheck that the operator status indicates no errors:
kubectl get operators.dcontroller.io pod-container-num-annotator -o yaml
apiVersion: dcontroller.io/v1alpha1
kind: Operator
metadata:
name: pod-container-num-annotator
spec:
...
status:
controllers:
- name: pod-container-num-annotator
conditions:
- lastTransitionTime: "2024-10-03T13:30:11Z"
message: Controller is up and running
observedGeneration: 1
reason: Ready
status: "True"
type: ReadyThen check whether the container number actually appears as an annotation on the pod:
kubectl get pod net-debug -o jsonpath='{.metadata.annotations}'
{"dcontroller.io/container-num":"1"}So far so good. Now create another pod with 3 containers:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: net-debug-2
spec:
containers:
- name: net-debug-1
image: docker.io/l7mp/net-debug:latest
- name: net-debug-2
image: docker.io/l7mp/net-debug:latest
- name: net-debug-3
image: docker.io/l7mp/net-debug:latest
EOFThe annotation should now indicate 3 containers:
kubectl get pod net-debug-2 -o jsonpath='{.metadata.annotations}'
{"dcontroller.io/container-num":"3"}Note that pods in other namespaces will not have the container-num annotation:
kubectl -n kube-system get pod etcd -o jsonpath='{.metadata.annotations}'
...Remove all resources we have created:
kubectl delete pod net-debug
kubectl delete pod net-debug-2
kubectl delete operators.dcontroller.io pod-container-num-annotator- The operator runs with full RBAC permissions to operate on the entire Kubernetes API. This is required at this point to make sure that the declarative operators loaded at runtime can access any Kubernetes API resource they intend to watch or modify. Later we may implement a dynamic RBAC scheme that would minimize the required permissions to the minimal set of API resources the running operators may need.
- The strategic merge patch implementation does not handle lists. Since Kubernetes does not
implement native strategic merge patching for schemaless unstructured resources that Δ-controller
uses internally, currently patches must be implemented a simplified local JSON patch code that
does not handle lists. Use
Patchertargets carefully or, whenever possible, opt forUpdatertargets.
Copyright 2024 by its authors. Some rights reserved. See AUTHORS.
Apache License - see LICENSE for full text.