Specify that counters are monotonic #442
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sums must be declared as monotonic in order to be represented in prometheus as counters. Note that this causes the otel collector to rename metrics when using the prometheus exporter. Counters have `_total` appended to the name. There are cases where counters aren't monotonic, in particular when counters are emitted only as deltas (V1) and the counter totals are reset in the Forwarder Agent. Signed-off-by: Rebecca Roberts <robertsre@vmware.com>
- Envelopes that are emitted only with a delta are decorated by the counter aggregator within Forwarder Agent to have both a delta and a total. - The envelope totals are not monotonic as the Forwarder Agent will discard totals once the map of totals grows over a set size. - Envelopes that are emitted only with totals will not have a delta present. Signed-off-by: Andrew Crump <crumpan@vmware.com>
rroberts2222
approved these changes
Dec 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sums must be declared as monotonic in order to be represented in prometheus as counters.
Note that this causes the otel collector to rename metrics when using the prometheus exporter. Counters have
_totalappended to the name.There are cases where loggregator counters aren't monotonic, in particular when counters are emitted only as deltas (V1) and the counter totals are reset in the Forwarder Agent. In this case we mark the otel sum as not monotonic and it will be represented as a gauge when using the prometheus exporter.
Type of change
Testing performed?
Checklist:
mainbranch, or relevant version branchIf you have any questions, or want to get attention for a PR or issue please reach out on the #logging-and-metrics channel in the cloudfoundry slack