Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add info about interacting with the docker solution & configure grafana data sources automatically #552

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions TrafficCapture/dockerSolution/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,33 @@ While in the TrafficCapture directory, run the following command:

`./gradlew :dockerSolution:composeUp`

### Interacting with the Docker Solution

A sample source cluster (running Elasticsearch 7.10) is set up with the Capture Proxy running in front of it. To route
calls through the Capture Proxy, send requests to `localhost:9200` with the default credentials (admin, admin), e.g.:
```sh
curl --insecure https://localhost:9200/_cat/indices -u 'admin:admin'
```

You can send the same calls to the source cluster while bypassing the Capture Proxy (calls will not be relayed to the
target cluster) via `localhost:19200`, and to the target cluster directly at `localhost:29200`.

For sample data that exercises various endpoints with a range of datatypes, you can execute a shell in the Migration
Console (`docker exec -i -t $(docker ps -aqf "ancestor=migrations/migration_console:latest") bash` or via the Docker
console) and run `./runTestBenchmarks.sh`. By default, this runs four short test workloads from
[Opensearch-Benchmark](https://github.com/opensearch-project/opensearch-benchmark) against the Capture Proxy endpoint
(9200). The Migration Console contains other utility functions (`./catIndices.sh`, `kafka-tools`, etc.) to interact
with the various containers of the solution.

With the default docker-compose configuration launched with :dockerSolution:composeUp, instrumentation containers
will be started (see below for other options). You can access the metrics generated by the solution in Grafana.
While the solution is running, go to [http://localhost:3000](http://localhost:3000/) and enter the default credentials
(admin, admin). Connections to Jaeger and Prometheus are automatically provisioned (see them under
`Connections->Data sources`), so you can go directly to `Explore` and define a query using the supplied data
from either data source. Traces for the capture proxy and replayer are available via Jaeger at
[http://localhost:16686](http://localhost:16686).


### Running with different telemetry flavors.

By default, composeUp will run an otel-collector that exports instrumentation to other local containers within the
Expand Down
4 changes: 3 additions & 1 deletion TrafficCapture/dockerSolution/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ dependencies {
def dockerFilesForExternalServices = [
"elasticsearchWithSearchGuard": "elasticsearch_searchguard",
"migrationConsole": "migration_console",
"otelCollector": "otel_collector"
"otelCollector": "otel_collector",
"grafana": "grafana"
]
// Create the static docker files that aren't hosting migrations java code from this repo
dockerFilesForExternalServices.each { projectName, dockerImageName ->
Expand Down Expand Up @@ -74,6 +75,7 @@ task buildDockerImages {
dependsOn buildDockerImage_elasticsearchWithSearchGuard
dependsOn buildDockerImage_migrationConsole
dependsOn buildDockerImage_otelCollector
dependsOn buildDockerImage_grafana

dependsOn buildDockerImage_trafficCaptureProxyServer
dependsOn buildDockerImage_trafficReplayer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ services:
- COLLECTOR_OTLP_ENABLED=true

grafana:
image: grafana/grafana:latest
image: 'migrations/grafana:latest'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes a lot more sense since the datasources are REALLY going to be specific for the compose environment. Thanks!

networks:
- migrations
ports:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
FROM grafana/grafana:latest

COPY datasources.yaml /usr/share/grafana/conf/provisioning/datasources/
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: 1
datasources:
- name: Jaeger
type: jaeger
url: http://jaeger:16686
basicAuth: false
isDefault: false
readOnly: false
jsonData: {}
- name: Prometheus
type: prometheus
url: http://prometheus:9090
access: proxy
basicAuth: false
isDefault: true
jsonData:
httpMethod: POST
prometheusType: Prometheus
exemplarTraceIdDestinations:
- name: trace_id
urlDisplayLabel: Jaeger
readOnly: false