To launch the end-2-end (E2E) services in containers, simply run ../gradlew :dockerSolution:composeUp from the
TrafficCapture directory (parent of this directory). That will build all java artifacts and create the necessary images
for each service. ../gradlew :dockerSolution:composeDown will tear everything (volumes, networks, containers) back
down again.
Notice that most of the Dockerfiles are dynamically constructed in the build hierarchy. Some efforts have been made to ensure that changes will make it into containers to be launched.
While in the TrafficCapture directory, run the following command:
../gradlew :dockerSolution:composeUp
A sample source cluster (running Elasticsearch 7.10) is set up with the Capture Proxy running in front of it. To route
calls through the Capture Proxy, send requests to localhost:9200 with the default credentials (admin, admin), e.g.:
curl --insecure https://localhost:9200/_cat/indices -u 'admin:admin'You can send the same calls to the source cluster while bypassing the Capture Proxy (calls will not be relayed to the
target cluster) via localhost:19200, and to the target cluster directly at localhost:29200.
For sample data that exercises various endpoints with a range of datatypes, you can execute a shell in the Migration
Console (docker exec -i -t $(docker ps -aqf "ancestor=migrations/migration_console:latest") bash or via the Docker
console) and run ./runTestBenchmarks.sh. By default, this runs four short test workloads from
Opensearch-Benchmark against the Capture Proxy endpoint
(9200). The Migration Console contains other utility functions (./catIndices.sh, kafka-tools, etc.) to interact
with the various containers of the solution.
With the default docker-compose configuration launched with :dockerSolution:composeUp, instrumentation containers
will be started (see below for other options). You can access the metrics generated by the solution in Grafana.
While the solution is running, go to http://localhost:3000 and enter the default credentials
(admin, admin). Connections to Jaeger and Prometheus are automatically provisioned (see them under
Connections->Data sources), so you can go directly to Explore and define a query using the supplied data
from either data source. Traces for the capture proxy and replayer are available via Jaeger at
http://localhost:16686.
By default, composeUp will run an otel-collector that exports instrumentation to other local containers within the
migrations network. However, the collector's export configuration can be overridden via the otel-collector property:
TrafficCapture % ../gradlew :dockerSolution:composeUp -Potel-collector=otel-aws.yml
The otel-aws.yml will use that docker-compose extension. That extension uses the collector configurations (from the container's base image) and wires the ~/.aws/credentials file into the container to provide the collector credentials that it needs to push metrics and traces to CloudWatch and X-Ray. In addition to the default configuration to use local containers, a third option will use BOTH local containers AND the AWS services.
The migrations components use OpenTelemetry for instrumentation so that different systems can be utilized across (Prometheus, CloudWatch, Zipkin, etc) and across different types of infrastructure. The docker solutions vended in this directory try to provide flexibility and consistency between different environments. Base images may change in the future as otel matures and configurations will also need to be updated over time due to external changes (e.g. debug/logging exporter) or internal ones (buffering parameters). To manage the 5 starting configurations that we produce for one purpose or another, support code is within otelConfigs.
The otel-collector configuration is contained within one yaml file. That single file configures the collector to interface with many different systems. Maintaining consistency of configurations even as they're copy-pasted between each other isn't scalable. Complicating matters more is that the base otel-collector from otel and the AWS distro both lack a posix base system. That makes cons-ing any configurations within the container challenging. The compromise struck here is to do the construction of configuration files as a preprocessing step BEFORE docker builds. That preprocessing logic is within dockerSolution/otelConfigs.
A python script creates individual (but complete) otel-collector configurations (consConfigSnippets.py). A shell script (makeConfigFiles.sh) runs 5 configuration sets to output the otel-config-*.yaml files that are used by the otelCollector image and the compose configurations. The compose configurations override the original otel collector configurations with new ones. Those compose files also vary by specifying different ports depending upon which services have been configured (Prometheus, zpages, etc ports).
Those configurations are created by merging a set of YAML snippets into a final file. Within the
dependencies.yml file, parents (and their respective snippets) are dependencies of their children.
As consConfigSnippets.py is invoked, all ancestors' snippets are included before the children that
were specified. To further simplify management of dependencies, snippets may have multiple dependencies.
Upon hitting each dependency for the first time, all of its dependencies are also found and included
within the final yaml configuration that is being output.
The tools in this directory can only be built if you have Java version 11 installed.
The version is specified in TrafficCapture/build.gradle using a Java toolchain, which allows us
to decouple the Java version used by Gradle itself from Java version used by the tools here.
Any attempt to use a different version will cause the build to fail and will result in the following error (or similar) depending on which tool/project is being built. Below is an example error when attempting to build with an incompatible Java version.
../gradlew trafficCaptureProxyServer:build
* What went wrong:
A problem occurred evaluating project ':trafficCaptureProxyServer'.
> Could not resolve all dependencies for configuration ':trafficCaptureProxyServer:opensearchSecurityPlugin'.
> Failed to calculate the value of task ':trafficCaptureProxyServer:compileJava' property 'javaCompiler'.
> No matching toolchains found for requested specification: {languageVersion=10, vendor=any, implementation=vendor-specific}.
> No locally installed toolchains match (see https://docs.gradle.org/8.0.2/userguide/toolchains.html#sec:auto_detection) and toolchain download repositories have not been configured (see https://docs.gradle.org/8.0.2/userguide/toolchains.html#sub:download_repositories).
When updating Pipfiles, run the following command to update all Pipfile.lock in subdirectories
find . -type f -name Pipfile -execdir bash -c 'PIPENV_IGNORE_VIRTUALENVS=true pipenv lock --python 3.11' \;