Skip to content

Commit

Permalink
V1.2.0 dev (#16)
Browse files Browse the repository at this point in the history
* adding init 1.2.0 changes

* controller changes

* controller rewrite to support pod labels

* agent now supports podlables, configurable prometheus, plugin auto discover

* chart changes to support new features

* updating CONTRIBUTING doc with accurate info

* restapi changes - wip

* latest dev images in chart

* restapi changes

* restapi fix

* bug fix for controller; restapi fix; remove binary

* restapi chages for new ui; bug fixes; version in configs

* refactor Makefiles, move to alpine

* more dockerfile changes

* small redis refactor, restapi detect redis.nil er

* bug fixes and prep for ui, using protojson instead of json

* remove multi-platform build

* Fix SelectRandomAgent

* Remove all buildfiles before building docker-all

* fix bug with controller, export CRD status to redis

* plugin discovery chagnes

* fix controller not updating redis if there are errs

* label and ns matching for agents, beter prom labels, py test

* move to buf for proto generation, add plugin back to summary

* python text experiment

* working python test, bug fixes

* python module

* remove python cache

* adding pyproject.toml

* update pyproject

* more python development, bug fixes

* fix python wrapper

* fix python wrapper

* fully working python plugin - json-ping

* push.sh file

* Create README.md

* merge Dockerfiles, use build stages

* chore: create missing directories

* fix bug where agent wont purge deleted test correctly

* chart chages and local testing configs

* chart image fix

* more docs - deployment models, logo, local debugging

* Markdown linting

* mnore Markdown linting

* mnore Markdown linting

---------

Co-authored-by: bakshi41c <bakshi41c@gmail.com>
Co-authored-by: BlitzQuiche <kieran.shave@live.com>
Co-authored-by: Marwa Ouled El Hadj Ali <mouledel@cisco.com>
  • Loading branch information
4 people authored Jul 11, 2024
1 parent d6bf0fb commit 081c80c
Show file tree
Hide file tree
Showing 87 changed files with 3,900 additions and 1,536 deletions.
File renamed without changes.
6 changes: 4 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ any real-time space e.g., Slack, Discord, etc.

### Setting up a build environment

#### Using a cluster (kind or any other Kubernetes cluster)

Prerequisites:

- A MacOS or Linux dev machine.
Expand All @@ -35,11 +37,11 @@ Once you have a cluster to develop on, install the Helm chart to that cluster, a

Note: The Controller has special commands to make changes to CRD files etc. Please refer to the [controller README](./controller/README.md).

Note 2: The Agent also has special commands to make changes to proto files. Please refer to the [agent README](./agent/README.md).
Note 2: The Agent and the common folders also has special commands to generate new plugins and building proto files. Please refer to the [agent README](agent/README.md) and [common README](agent/README.md).

### Writing new synthetic test plugins for the agent

Synthetic-heart welcomes the community to contribute new plugins for the agents. Please refer to the [agent README](./agent/README.md) to see how to write new plugins.
Synthetic-heart welcomes the community to contribute new plugins for the agents. Please refer to the [agent README](agent/README.md) to see how to write new plugins.

## Reporting Issues

Expand Down
71 changes: 12 additions & 59 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,79 +1,32 @@
default: all
GOFLAGS=-ldflags=-w -ldflags=-s
SYNTEST_PLUGIN_SUBDIRS = $(notdir $(wildcard ./agent/plugins/syntests/*))

LOCAL_BUILD_PATH=./bin

## Proto
build-go-proto:
@echo Building Golang proto files...
protoc --go_out=plugins=grpc:common/proto/ common/proto/syntest.proto --proto_path=common/proto/

## Agent
build-go-syntest-plugins:
@echo "Building Go syntest plugins"
cd agent && \
for dir in $(SYNTEST_PLUGIN_SUBDIRS); do \
echo "Building plugin: $$dir" && \
CGO_ENABLED=0 go build $(GOFLAGS) -o $(LOCAL_BUILD_PATH)/plugins/test-$$dir ./plugins/syntests/$$dir/ && \
echo "$$dir compiled!" || { echo "$$dir failed!"; exit 1; }; \
done

build-agent-only:
@echo "Building agent"
cd agent && CGO_ENABLED=0 go build $(GOFLAGS) -o $(LOCAL_BUILD_PATH)/agent .

.PHONY: build-agent
build-agent: build-agent-only build-go-syntest-plugins
default: docker-all

.PHONY: docker-agent
docker-agent:
@echo "Building agent container image"
cd agent && podman build -f Dockerfile -t synheart-agent:dev_latest ..
cd agent && podman build -f Dockerfile --target base -t synheart-agent:dev-latest-no-plugins ..
cd agent && podman build -f Dockerfile --target base-with-go-plugins -t synheart-agent:dev-latest ..

## Rest Api
build-restapi:
@echo "Building restapi binary"
cd restapi && CGO_ENABLED=0 go build $(GOFLAGS) -o $(LOCAL_BUILD_PATH)/restapi .
.PHONY: docker-agent-py
docker-agent-py:
@echo "Building python agent container image (Experimental)"
cd agent && podman build -f Dockerfile --target base-with-python-plugins -t synheart-agent:dev-latest-with-py ..

.PHONY: docker-restapi
docker-restapi:
@echo "Building restapi container image"
cd restapi && podman build -f Dockerfile -t synheart-restapi:dev_latest ..
cd restapi && podman build -f Dockerfile -t synheart-restapi:dev-latest ..

## Controller
.PHONY: docker-controller
docker-controller:
@echo "Building controller container image"
cd controller && podman build -f Dockerfile -t synheart-controller:dev_latest ..
cd controller && podman build -f Dockerfile -t synheart-controller:dev-latest ..

.PHONY : docker-all
docker-all: docker-agent docker-restapi docker-controller

#.PHONY : test
#test: build-dummy-test-plugins build-syn-heart
# go test -race -v synthetic-heart/agent
# go test -race -v synthetic-heart/broadcaster


## SDK - Creating new syntest plugin
.PHONY : new-go-test
new-go-test:
@echo Creating Directory plugins/syntests/$(name)
mkdir agent/plugins/syntests/$(name)

@echo Creating file plugins/syntests/$(name)/$(name).go
cp agent/plugins/templates/go/test.tpl.go plugins/syntests/$(name)/$(name).go

sed -i .backup s/{{TestName}}/$(call capitalize,$(name))/g plugins/syntests/$(name)/$(name).go
rm agent/plugins/syntests/$(name)/$(name).go.backup

docker-all: clean docker-agent docker-agent-py docker-restapi docker-controller

.PHONY : clean
clean:
rm -rf agent/bin
rm -rf controller/bin
rm -rf restapi/bin

define capitalize
$(shell echo $(1) | awk '{$$1=toupper(substr($$1,0,1))substr($$1,2); print $$0}')
endef
rm -rf restapi/bin
44 changes: 33 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# Synthetic Heart - Kubernetes synthetic testing and monitoring framework

![Synthethic-heart-banner](./docs/synheart-banner.png)

## Synthetic Heart - Kubernetes synthetic testing and monitoring framework

Synthetic Heart is a synthetic testing and monitoring framework for Kubernetes clusters. It allows operators to run synthetic tests, then check and monitor the results through an API or through metrics (via Prometheus).

Expand Down Expand Up @@ -44,19 +47,31 @@ spec:
The Agent is responsible for watching the storage (Redis) for any new or updated test configs, and running them.
There are two main components in the agent:
There are three main components in the agent:
- Plugin Manager - responsible for managing lifecycle of plugins
- Plugins - Synthetic tests written as separate binary
The tests run as plugins ([hashicorp go-plugins](https://github.com/hashicorp/go-plugin)), and communicate to the plugin-manager via gRPC.
- Prometheus Exporter - Exports metrics to Prometheus (can use push gateway)
- Plugins - Test logic written as separate binaries/scripts.
The plugin manager is responsible for handling the lifecycle of these plugins. It will start/stop, check health and restart them on crash/error. The plugin manager also exports the test results to redis, as well as metrics to Prometheus.
#### The Plugins
Plugins run the actual tests. They are separate binaries/scripts that are run by the agent.
This is what makes Synthetic Heart extensible. The plugins can be written in any language, as long as they can communicate with the plugin manager via gRPC. Currently only Golang and Python plugins are tested.
The plugins use [hashicorp go-plugins](https://github.com/hashicorp/go-plugin).
## Rest Api
Synthetic Heart also comes with a Rest API which lets external services query test results etc. It is essentially a shim for the Redis storage.
## UI (Experimental)
A UI is currently being worked on. It will allow users to view test results, logs, and agent details. You can check it out the [UI repo here](https://github.com/bakshi41c/synthetic-heart-ui).
![Synthetic Heart Agent Architecture](./docs/ui-screenshot.png)
## Installation
### With Helm Chart
Expand All @@ -71,20 +86,20 @@ Prerequisites:
Steps to follow:

- Install the [synthetic-heart](./chart/synthetic-heart) Helm chart
- Run: `helm upgrade -i synthetic-heart -n synthetic-heart .` in `charts/` directory.
- Run: `helm upgrade -i synthetic-heart-system -n synthetic-heart-system .` in `charts/` directory.
- This installs the RestAPI, Agents (as DaemonSet), Controller, and Redis.
- Install the [synthetic-tests](./chart/synthetic-tests) Helm chart
- Run `helm upgrade -i synthetic-tests -n synthetic-heart .` in `charts/` directory.
- Run `helm upgrade -i synthetic-tests -n synthetic-heart-tests .` in `charts/` directory.
- This installs two example synthetic tests.
- You may then port forward the rest api to query results.
- Run `kubectl port-forward svc/synheart-api-svc 8080:8080 -n synthetic-heart`
- and then query the tests `curl http://localhost:8080/api/v1/tests`
- and then query the tests `curl http://localhost:8080/api/v1/test-configs`

### Without Helm Chart

It might be a bit more complex to install the components manually. Please explore the [Helm chart](./chart/synthetic-heart) to check the example configurations for different Kubernetes resources. The dependencies, and major components are described below.

A few Kubenretes resources need to be installed:
A few Kubernetes resources need to be installed:

- CustomResourceDefinition - The `SyntheticTest` CRD needs to be installed. [Link to CRD.](./controller/config/crd/bases/synheart.infra.webex.com_synthetictests.yaml)
- Redis - Redis v7 needs to be installed so the test configs and results can be stored.
Expand All @@ -98,6 +113,10 @@ A few Kubenretes resources need to be installed:
- A `Service` for the Restapi is needed.
- Optionally an `Ingress` can be added to the Restapi to make the API accessible from outside the cluster.

## Deployment strategies

Please check the [Deployment](./docs/Deployment.md) document for different deployment strategies.

## Credits and Acknowledgements

Thank you to the following folks for contributing to synthetic-heart project:
Expand All @@ -113,14 +132,17 @@ Thank you to the following folks for contributing to synthetic-heart project:
- Meibao Wang: <meibwang@cisco.com>
- Mercion Wilathgamuwage: <mwilathg@cisco.com>
- Shaz Balakumar: <shbalaku@cisco.com>
- Kieran Shave <kshave@cisco.com>

## Roadmap

There's a lot of features that are still being worked on. [Get involved!](./CONTRIBUTING.md)

- UI (currently being worked on)
- Re-design agent deployment model (allow agents per node, per namespace etc.)
- Allow agents to be deployed in different namespace, mount secrets etc.
- ~~Re-design agent deployment model (allow agents per node, per namespace etc.)~~
~~- Allow agents to be deployed in different namespace, mount secrets etc.~~
- ~~Python plugins~~
- Agent CRD (to allow for easier agent deployment)
- Plugin versions
- Dynamically add or remove plugins at build time (to reduce binary size)
- More plugins
Expand Down
59 changes: 34 additions & 25 deletions agent/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,14 +1,9 @@
############################
# STEP 1 build the image for creating the executable
############################
FROM golang:1.22-alpine3.19 as builder
FROM docker.io/library/golang:1.22-alpine3.19 as builder

# Install git + SSL ca certificates + make
# Git is required for fetching the dependencies.
# Ca-certificates is required to call HTTPS endpoints.
# Make to build go application
ARG TARGETARCH
RUN if [ "$TARGETARCH" = "arm64" ]; then ln -s /bin/run-parts /usr/bin/run-parts; fi
RUN apk update && apk upgrade && apk add --no-cache git ca-certificates make unzip g++ && update-ca-certificates && apk --no-cache add openssl wget && rm -rf /var/cache/apk/*

# Create appuser
Expand All @@ -25,8 +20,10 @@ RUN go install github.com/golang/protobuf/protoc-gen-go@v1.5.2

WORKDIR /app

RUN mkdir agent
COPY agent/go.mod agent/.
COPY agent/go.sum agent/.
RUN mkdir common
COPY common/go.mod common/.
COPY common/go.sum common/.
RUN cd agent && go mod download
Expand All @@ -38,38 +35,50 @@ RUN mkdir -p /app/synthetic-heart/
RUN touch /app/synthetic-heart/.emptyfile

# Compile the binary
RUN make build-agent
RUN cd agent/ && make build-agent

# Move everything to the final location
RUN mv agent/bin/* /app/synthetic-heart
RUN mv agent/plugins/syntests-python/ /app/synthetic-heart/plugins-python


############################
# STEP 2 build a small image with only the executable
############################
FROM docker.io/alpine:3.19
FROM docker.io/library/alpine:3.19 as base

RUN apk update
# Copy over just the agent binary
COPY --from=builder /app/synthetic-heart/agent /app/synthetic-heart/agent

# Add curl for curl tests
RUN apk add --no-cache curl
# Create a /tmp/ diretctory (required for go plugin for Unix Domain Socket)
COPY --from=builder /app/synthetic-heart/.emptyfile /tmp/.emptyfile

# Import from builder.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
COPY --from=builder /bin/sh /bin/sh
WORKDIR /app/synthetic-heart

# Copy our static executables
COPY --from=builder --chown=appuser:appuser /app/synthetic-heart /app/synthetic-heart
# Run the binary.
ENTRYPOINT ["./agent"]

# Create a /tmp/ diretctory (required for go plugin for Unix Domain Socket)
COPY --from=builder --chown=appuser:appuser /app/synthetic-heart/.emptyfile /tmp/.emptyfile
############################
# STEP 3 build image with go plugins
############################
From base as base-with-go-plugins

# Copy over the go plugins
COPY --from=builder /app/synthetic-heart/plugins/* /app/synthetic-heart/plugins/

# Use an unprivileged user.
USER appuser
# Install CURL for syntest
RUN apk add curl

WORKDIR /app/synthetic-heart
############################
# STEP 4 build image with python plugins
############################
FROM base-with-go-plugins as base-with-python-plugins

# Run the hello binary.
ENTRYPOINT ["./agent"]
# Copy over python plugins
COPY --from=builder /app/synthetic-heart/plugins-python /app/synthetic-heart/plugins-python

# Install Python
ENV PYTHONUNBUFFERED=1
RUN apk add --update --no-cache python3 py3-pip git
RUN pip3 install --no-cache --upgrade pip setuptools --break-system-packages
RUN cd /app/synthetic-heart/plugins-python && sh install_requirements.sh --break-system-packages
45 changes: 41 additions & 4 deletions agent/Makefile
Original file line number Diff line number Diff line change
@@ -1,15 +1,52 @@
GOFLAGS=-ldflags=-w -ldflags=-s
SYNTEST_PLUGIN_SUBDIRS = $(notdir $(wildcard ./plugins/syntests/*))
LOCAL_BUILD_PATH=./bin

## Agent
build-go-syntest-plugins:
@echo "Building Go syntest plugins"
for dir in $(SYNTEST_PLUGIN_SUBDIRS); do \
echo "Building plugin: $$dir" && \
CGO_ENABLED=0 go build $(GOFLAGS) -o $(LOCAL_BUILD_PATH)/plugins/test-$$dir ./plugins/syntests/$$dir/ && \
echo "$$dir compiled!" || { echo "$$dir failed!"; exit 1; }; \
done

build-agent-only:
@echo "Building agent"
CGO_ENABLED=0 go build $(GOFLAGS) -ldflags="-X main.Version=$(SYNHEART_VERSION)" -o $(LOCAL_BUILD_PATH)/agent .

.PHONY: build-agent
build-agent: build-agent-only build-go-syntest-plugins

## SDK - Creating new syntest plugin
.PHONY : new-go-syntest
new-go-syntest:
@echo Creating Directory plugins/syntests/$(name)
mkdir agent/plugins/syntests/$(name)
mkdir plugins/syntests/$(name)

@echo Creating file plugins/syntests/$(name)/$(name).go
cp agent/plugins/templates/go/test.tpl.go plugins/syntests/$(name)/$(name).go
cp plugins/templates/go/test.tpl.go plugins/syntests/$(name)/$(name).go

sed -i s/{{TestNameRaw}}/$(name)/g plugins/syntests/$(name)/$(name).go
sed -i s/{{TestName}}/$(call capitalize,$(name))/g plugins/syntests/$(name)/$(name).go

sed -i .backup s/{{TestName}}/$(call capitalize,$(name))/g plugins/syntests/$(name)/$(name).go
rm agent/plugins/syntests/$(name)/$(name).go.backup
.PHONY : new-python-syntest
new-python-syntest:
@echo Creating Directory plugins/syntests-python/$(name)
mkdir -p plugins/syntests-python/$(name)

@echo Creating file plugins/syntests/$(name)/$(name).py
cp plugins/templates/python/test.tpl.py plugins/syntests-python/$(name)/test-$(name).py
cp plugins/templates/python/requirements.txt plugins/syntests-python/$(name)/requirements.txt

echo $(call camelcase,$(name))

sed -i s/{{TestName}}/$(call camelcase,$(name))/g plugins/syntests-python/$(name)/test-$(name).py

define capitalize
$(shell echo $(1) | awk '{$$1=toupper(substr($$1,0,1))substr($$1,2); print $$0}')
endef

define camelcase
$(shell echo $(1) | awk 'BEGIN{FS="";RS="-";ORS=""} {$$0=toupper(substr($$0,1,1)) substr($$0,2)} 1')
endef
Loading

0 comments on commit 081c80c

Please sign in to comment.