Skip to content

Commit

Permalink
EDU-3257: Updates Task-Queue metrics coverage to reduce confusion
Browse files Browse the repository at this point in the history
- "Describe" returns info that isn't metrics
- METRIC ACCURACY warnings are unnecessary clutter and scary
  • Loading branch information
fairlydurable committed Oct 15, 2024
1 parent 18efd1a commit d152606
Showing 1 changed file with 13 additions and 37 deletions.
50 changes: 13 additions & 37 deletions docs/develop/worker-performance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ The Workflow Cache is created and shared between all the workers. It's designed

These options limit the resource consumption of the in-memory Workflow cache. Workflow cache options are shared between all Workers, because the Workflow cache is something that has to do with the resource consumption of the whole host, like memory and the total amount of threads, and should be limited per JVM.

## Monitor Task Queue backlog metrics {#task-queue-metrics}
## Monitor Task Queue backlog data {#task-queue-metrics}

A [Task Queue](https://docs.temporal.io/workers#task-queue) is a lightweight, dynamically allocated queue.
[Worker Entities](https://docs.temporal.io/workers#worker-entity) poll the queue for [Tasks](https://docs.temporal.io/workers#task).
Expand All @@ -48,8 +48,8 @@ With an accurate estimate of backlog Tasks, you can determine the optimal number
Balance your Worker count with the number of Tasks to achieve the best performance.
This approach minimizes Task backlog saturation and reduces idle Workers.

Task Queue metrics provide numerical insights into your Task Queue activity and backlog characteristics.
Use these metrics to tune your production deployments.
Task Queue data provide numerical insights into your Task Queue activity and backlog characteristics.
Use these numbers to tune your production deployments.
Evaluate your Worker loads and assess whether you need to scale up or reduce your Worker deployment.

### Task Queue types {#task-queue-types}
Expand All @@ -76,49 +76,31 @@ The Temporal Service reports information separately for each Task Queue type (no
Use the following metrics to retrieve detailed information about Task Queue health and performance.
Available metrics include:

- [`ApproximateBacklogCount`](#ApproximateBacklogCount)
- [`ApproximateBacklogAge`](#ApproximateBacklogAge)
- [`ApproximateBacklogCount`](#ApproximateBacklogCountAndAge) and [`ApproximateBacklogAge`](#ApproximateBacklogCountAndAge)
- [`TasksAddRate`](#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](#TasksAddRate-and-TasksDispatchRate)
- [`BacklogIncreaseRate`](#BacklogIncreaseRate) (derived from [`TasksAddRate`](#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](#TasksAddRate-and-TasksDispatchRate))

#### `ApproximateBacklogCount` {#ApproximateBacklogCount}
#### `ApproximateBacklogCount` and `ApproximateBacklogAge` {#ApproximateBacklogCountAndAge}

Represents the approximate count of Tasks currently backlogged in this Task Queue.
`ApproximateBacklogCount` represents the approximate count of Tasks currently backlogged in this Task Queue.
The number may include expired Tasks as well as active Tasks, but it will eventually converge to the correct count over time.

You can rely on this count when making scaling decisions.

:::info Metric Accuracy

Workflow Task Queue types provide partial information due to performance optimizations.
Tasks sent to [Sticky](https://docs.temporal.io/workers#sticky-execution) queues are not included in the returned values for this metric.
Since Tasks remain valid for only a few seconds in Sticky Queues, this inaccuracy diminishes over time, especially as the backlog grows.
:::

#### `ApproximateBacklogAge` {#ApproximateBacklogAge}

Returns the approximate age of the oldest Task in the backlog.
`ApproximateBacklogAge` returns the approximate age of the oldest Task in the backlog.
The age is based on the creation time of the Task at the head of the queue.

You can rely on this count when making scaling decisions.

:::info Metric Accuracy

Workflow Task Queue types provide partial information due to performance optimizations.
Tasks sent to [Sticky](https://docs.temporal.io/workers#sticky-execution) queues are not included in the returned values.
Since Tasks remain valid for only a few seconds in Sticky Queues, this inaccuracy diminishes over time, especially when the backlog is older than a few seconds.
You can rely on both these counts when making scaling decisions.

:::
Please note: [Sticky queues](https://docs.temporal.io/workers#sticky-execution) will affect these metrics, but only for a few seconds as the Worker begins polling.
Inaccuracies diminish over time, especially as the backlog grows.
That's because Tasks sent to Sticky queues are not included in the returned values for `ApproximateBacklogCount` and `ApproximateBacklogAge` so only partial information is provided when that happens.

#### `TasksAddRate` and `TasksDispatchRate` {#TasksAddRate-and-TasksDispatchRate}

Reports the approximate Tasks-per-second added to or dispatched from a Task Queue.
This rate is averaged over the most recent 30-second time interval.
The calculations include Tasks that were added to or dispatched from the backlog as well as Tasks that were immediately dispatched and bypassed the backlog (sync-matched).

:::info Metric Accuracy

The actual Task delivery count may be significantly higher than the number reported by these metrics:
The actual Task delivery count may be significantly higher than the number reported by these twometrics:

- Eager dispatch refers to a Temporal feature where Activities can be requested by an SDK using one Workflow Task completion response.
Tasks using Eager dispatch do not pass through Task Queues.
Expand All @@ -127,8 +109,6 @@ The actual Task delivery count may be significantly higher than the number repor
Normally, only the first Workflow Task of each Workflow is placed on a Workflow Task Queue.
Subsequent Tasks are passed to the Sticky Task Queue for performance improvement.

:::

#### `BacklogIncreaseRate` {#BacklogIncreaseRate}

Approximates the _net_ Tasks per second added to the backlog, averaged over the most recent 30 seconds.
Expand All @@ -141,13 +121,9 @@ TasksAddRate - TasksDispatchRate
- Positive values of `X` indicate the backlog is growing by about `X` Tasks per second.
- Negative values of `X` indicate the backlog is shrinking by about `X` Tasks per second.

:::info Metric Accuracy

While individual `add` and `dispatch` rates may be inaccurate due to Eager and Sticky Task Queues, the `BacklogIncreaseRate` reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds.

:::

### Fetch Worker metrics {#retrieve-worker-info}
### Fetch Worker info {#retrieve-worker-info}

The Temporal CLI helps you monitor and evaluate Worker performance.
Issue the following command to display a list of active Workers that have recently polled a Task Queue:
Expand Down

0 comments on commit d152606

Please sign in to comment.