-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add telemetry stack to local playground #2804
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
d883612
feat: add telemetry stack to localenv under command
mkurapov 4f0dad8
chore: remove tempo
mkurapov 84643af
chore(localenv): update prometheus scrape interval
mkurapov 093bc8e
chore: explicitly set otel collector endpoint
mkurapov 58c5b85
chore: change prometheus scrape interval to 15s
mkurapov 51c1c26
chore: update dashboard queries
mkurapov 0bc78f7
chore: add readme
mkurapov bd88c3a
chore: set auto-refresh within grafana dashboard
mkurapov 0a91ca4
chore: add psql command
mkurapov 2c26549
Merge branch 'main' into 2800/mk/local-telemetry
mkurapov 56bd60b
chore: formatting
mkurapov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Local Playground Telemetry | ||
|
|
||
| This folder contains telemetry components that can be run as part of the local playground. | ||
|
|
||
| ## Overview | ||
|
|
||
| The telemetry components include: | ||
|
|
||
| - **OpenTelemetry Collector**: Collects and processes telemetry data from `cloud-nine-backend` and `happy-life-backend` services. | ||
| - **Prometheus**: Scrapes metrics from the OpenTelemetry collector, and stores them. | ||
| - **Grafana**: Visualizes metrics from Prometheus. | ||
|
|
||
| ## Usage | ||
|
|
||
| From the root of the repository, run: | ||
|
|
||
| ``` | ||
| pnpm localenv:compose:telemetry up | ||
| ``` | ||
|
|
||
| If wanting to use Postgres instead of TigerBeetle for the accounting database: | ||
|
|
||
| ``` | ||
| pnpm localenv:compose:psql:telemetry up | ||
| ``` | ||
|
|
||
| Once the components are running, you can access Grafana at http://localhost:4500. Grafana comes with an example dashboard of some of our basic metrics. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| services: | ||
| cloud-nine-backend: | ||
| environment: | ||
| ENABLE_TELEMETRY: true | ||
| LIVENET: false | ||
| OPEN_TELEMETRY_COLLECTOR_URLS: http://otel-collector:4317 | ||
|
|
||
| happy-life-backend: | ||
| environment: | ||
| ENABLE_TELEMETRY: true | ||
| LIVENET: false | ||
| OPEN_TELEMETRY_COLLECTOR_URLS: http://otel-collector:4317 | ||
|
|
||
| otel-collector: | ||
| hostname: otel-collector | ||
| image: otel/opentelemetry-collector:latest | ||
| command: "--config=/etc/otel-collector-config.yaml" | ||
| networks: | ||
| - rafiki | ||
| volumes: | ||
| - ../telemetry/otel-collector-config.yaml:/etc/otel-collector-config.yaml | ||
|
|
||
| prometheus: | ||
| image: prom/prometheus:latest | ||
| command: "--config.file=/etc/prometheus/prometheus.yaml --log.level=debug" | ||
| networks: | ||
| - rafiki | ||
| volumes: | ||
| - ../telemetry/prometheus.yaml:/etc/prometheus/prometheus.yaml | ||
| ports: | ||
| - "9090:9090" | ||
|
|
||
| grafana: | ||
| image: grafana/grafana:latest | ||
| networks: | ||
| - rafiki | ||
| ports: | ||
| - '4500:3000' | ||
| volumes: | ||
| - grafana_storage:/var/lib/grafana | ||
| - ../telemetry/grafana/provisioning:/etc/grafana/provisioning | ||
| environment: | ||
| - GF_AUTH_ANONYMOUS_ENABLED=true | ||
| - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin | ||
|
|
||
| volumes: | ||
| grafana_storage: |
7 changes: 7 additions & 0 deletions
7
localenv/telemetry/grafana/provisioning/dashboards/default.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| apiVersion: 1 | ||
|
|
||
| providers: | ||
| - name: Example | ||
| type: file | ||
| options: | ||
| path: /etc/grafana/provisioning/dashboards |
248 changes: 248 additions & 0 deletions
248
localenv/telemetry/grafana/provisioning/dashboards/example.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| { | ||
| "annotations": { | ||
| "list": [ | ||
| { | ||
| "builtIn": 1, | ||
| "datasource": { | ||
| "type": "grafana", | ||
| "uid": "-- Grafana --" | ||
| }, | ||
| "enable": true, | ||
| "hide": true, | ||
| "iconColor": "rgba(0, 211, 255, 1)", | ||
| "name": "Annotations & Alerts", | ||
| "type": "dashboard" | ||
| } | ||
| ] | ||
| }, | ||
| "editable": true, | ||
| "fiscalYearStartMonth": 0, | ||
| "graphTooltip": 0, | ||
| "links": [], | ||
| "panels": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "PBFA97CFB590B2093" | ||
| }, | ||
| "fieldConfig": { | ||
| "defaults": { | ||
| "color": { | ||
| "mode": "palette-classic" | ||
| }, | ||
| "custom": { | ||
| "axisBorderShow": false, | ||
| "axisCenteredZero": false, | ||
| "axisColorMode": "text", | ||
| "axisLabel": "", | ||
| "axisPlacement": "auto", | ||
| "barAlignment": 0, | ||
| "drawStyle": "line", | ||
| "fillOpacity": 0, | ||
| "gradientMode": "none", | ||
| "hideFrom": { | ||
| "legend": false, | ||
| "tooltip": false, | ||
| "viz": false | ||
| }, | ||
| "insertNulls": false, | ||
| "lineInterpolation": "linear", | ||
| "lineWidth": 1, | ||
| "pointSize": 5, | ||
| "scaleDistribution": { | ||
| "type": "linear" | ||
| }, | ||
| "showPoints": "auto", | ||
| "spanNulls": false, | ||
| "stacking": { | ||
| "group": "A", | ||
| "mode": "none" | ||
| }, | ||
| "thresholdsStyle": { | ||
| "mode": "off" | ||
| } | ||
| }, | ||
| "mappings": [], | ||
| "thresholds": { | ||
| "mode": "absolute", | ||
| "steps": [ | ||
| { | ||
| "color": "green", | ||
| "value": null | ||
| }, | ||
| { | ||
| "color": "red", | ||
| "value": 80 | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "overrides": [] | ||
| }, | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 12, | ||
| "x": 0, | ||
| "y": 0 | ||
| }, | ||
| "id": 2, | ||
| "interval": "15s", | ||
| "options": { | ||
| "legend": { | ||
| "calcs": [], | ||
| "displayMode": "list", | ||
| "placement": "bottom", | ||
| "showLegend": true | ||
| }, | ||
| "tooltip": { | ||
| "mode": "single", | ||
| "sort": "none" | ||
| } | ||
| }, | ||
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "PBFA97CFB590B2093" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "(increase(transactions_amount_total[30s]) * 0.5 * 0.0001)", | ||
| "hide": false, | ||
| "instant": false, | ||
| "interval": "", | ||
| "legendFormat": "__auto", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Transaction Amount", | ||
| "type": "timeseries" | ||
| }, | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "PBFA97CFB590B2093" | ||
| }, | ||
| "fieldConfig": { | ||
| "defaults": { | ||
| "color": { | ||
| "mode": "palette-classic" | ||
| }, | ||
| "custom": { | ||
| "axisBorderShow": false, | ||
| "axisCenteredZero": false, | ||
| "axisColorMode": "text", | ||
| "axisLabel": "", | ||
| "axisPlacement": "auto", | ||
| "barAlignment": 0, | ||
| "drawStyle": "line", | ||
| "fillOpacity": 0, | ||
| "gradientMode": "none", | ||
| "hideFrom": { | ||
| "legend": false, | ||
| "tooltip": false, | ||
| "viz": false | ||
| }, | ||
| "insertNulls": false, | ||
| "lineInterpolation": "linear", | ||
| "lineWidth": 1, | ||
| "pointSize": 5, | ||
| "scaleDistribution": { | ||
| "type": "linear" | ||
| }, | ||
| "showPoints": "auto", | ||
| "spanNulls": false, | ||
| "stacking": { | ||
| "group": "A", | ||
| "mode": "none" | ||
| }, | ||
| "thresholdsStyle": { | ||
| "mode": "off" | ||
| } | ||
| }, | ||
| "mappings": [], | ||
| "thresholds": { | ||
| "mode": "absolute", | ||
| "steps": [ | ||
| { | ||
| "color": "green", | ||
| "value": null | ||
| }, | ||
| { | ||
| "color": "red", | ||
| "value": 80 | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "overrides": [] | ||
| }, | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 12, | ||
| "x": 0, | ||
| "y": 8 | ||
| }, | ||
| "id": 1, | ||
| "interval": "15s", | ||
| "options": { | ||
| "legend": { | ||
| "calcs": [], | ||
| "displayMode": "list", | ||
| "placement": "bottom", | ||
| "showLegend": true | ||
| }, | ||
| "tooltip": { | ||
| "mode": "single", | ||
| "sort": "none" | ||
| } | ||
| }, | ||
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "PBFA97CFB590B2093" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "sum by (source) (round(increase(transactions_total[30s]) * 0.5))", | ||
| "hide": false, | ||
| "instant": false, | ||
| "interval": "", | ||
| "legendFormat": "__auto", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Transaction Count", | ||
| "type": "timeseries" | ||
| } | ||
| ], | ||
| "refresh": "15s", | ||
| "schemaVersion": 39, | ||
| "tags": [], | ||
| "templating": { | ||
| "list": [] | ||
| }, | ||
| "time": { | ||
| "from": "now-15m", | ||
| "to": "now" | ||
| }, | ||
| "timepicker": { | ||
| "refresh_intervals": [ | ||
| "15s", | ||
| "30s", | ||
| "1m", | ||
| "2m", | ||
| "5m", | ||
| "10m", | ||
| "15m", | ||
| "30m", | ||
| "1h" | ||
| ] | ||
| }, | ||
| "timezone": "browser", | ||
| "title": "Example Dashboard", | ||
| "uid": "fdr58stwkr6yof", | ||
| "version": 1, | ||
| "weekStart": "" | ||
| } |
11 changes: 11 additions & 0 deletions
11
localenv/telemetry/grafana/provisioning/datasources/datasources.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| apiVersion: 1 | ||
|
|
||
| datasources: | ||
| - name: Prometheus | ||
| type: prometheus | ||
| access: proxy | ||
| orgId: 1 | ||
| url: http://prometheus:9090 | ||
| isDefault: true | ||
| version: 1 | ||
| editable: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| receivers: | ||
| otlp: | ||
| protocols: | ||
| grpc: | ||
| endpoint: 0.0.0.0:4317 | ||
|
|
||
| processors: | ||
| batch: | ||
|
|
||
| exporters: | ||
| debug: | ||
| verbosity: detailed | ||
| prometheus: | ||
| endpoint: 0.0.0.0:8491 | ||
|
|
||
| service: | ||
| pipelines: | ||
| metrics: | ||
| receivers: [otlp] | ||
| processors: [batch] | ||
| exporters: [prometheus, debug] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| scrape_configs: | ||
| - job_name: prometheus | ||
| scrape_interval: 15s | ||
| static_configs: | ||
| - targets: ['otel-collector:8491'] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
beyond scope of this task, but yet another variation of
localenv:composedefinitely suggests we should make a start script that handles these options as discussed in slack. would be nice to be able to mix and match as needed without a bespoke command