Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion documentation/architecture/query-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ to process data in table page frames for better CPU use.
thread. Other queries, like those involving `GROUP BY` and `SAMPLE BY`, execute a pipeline with some single-threaded stages and some multi-threaded stages to avoid slow downs when groups are unbalanced.

- **Worker pools:** QuestDB allows to configure different pools for specialized functions, like
parsing incoming data, applying WAL file changes, handling PostgreSQL-Wire protocol, or responding to HTTP connections. By default, most tasks are handled by a shared worker pool.
parsing incoming data, applying WAL file changes, handling PostgreSQL-Wire protocol, or responding to HTTP connections. By default, work is spread across three shared pools: network I/O, query
execution, and writes. See [shared workers](/docs/configuration/shared-workers/).

- **Query plan caching:**
The system caches query plans for reuse within the same connection. (Query results are not
Expand Down
67 changes: 67 additions & 0 deletions documentation/configuration/shared-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,30 @@ server, each pool gets 62 threads. Adjust these values when you need to shift
CPU capacity toward a specific workload, for example reducing the network pool
and increasing the query pool on a read-heavy system.

The pools run at different thread priorities so the server stays responsive
under load: the network pool is one step above normal, the query pool is at
normal, and the write pool is one step below.

## Settings inherited from the legacy shared pool

Before the split into three pools, QuestDB ran a single shared pool configured
through the `shared.worker.*` keys. Those keys still work and act as the
defaults for all three pools.

:::note

`shared.worker.count` sets the default size of **each** pool, not a total to be
divided between them. On a configuration carried over from a single-pool
release, `shared.worker.count=8` yields 8 network threads, 8 query threads and
8 write threads, so 24 in total. Set `shared.network.worker.count`,
`shared.query.worker.count` and `shared.write.worker.count` explicitly when you
need to cap the total.

:::

The remaining `shared.worker.*` settings below have no per-pool equivalent:
they apply to all three pools at once.

## shared.network.worker.affinity

- **Default**: none
Expand Down Expand Up @@ -63,6 +87,16 @@ operations such as filters and group-by. Increasing this value raises query
parallelism at the expense of CPU resources available to network I/O and
writes.

## shared.worker.count

- **Default**: see table above
- **Reloadable**: no

Default number of worker threads for each of the three pools. Overridden per
pool by `shared.network.worker.count`, `shared.query.worker.count` and
`shared.write.worker.count`. Kept for compatibility with single-pool
configurations.

## shared.worker.haltOnError

- **Default**: `false`
Expand All @@ -72,6 +106,39 @@ When enabled, a worker thread stops if it encounters an unexpected error.
Intended for debugging. In production, leave this disabled so that transient
errors do not reduce the size of a thread pool.

## shared.worker.nap.threshold

- **Default**: `7000`
- **Reloadable**: no

Number of idle cycles after which a worker thread yields to other threads with
a short pause. Applies to all three pools.

## shared.worker.sleep.threshold

- **Default**: `10000`
- **Reloadable**: no

Number of idle cycles after which a worker thread goes to sleep, waking on
`shared.worker.sleep.timeout`. Lower values reduce idle CPU use at the cost of
a slower response to the next task. Applies to all three pools.

## shared.worker.sleep.timeout

- **Default**: `10`
- **Reloadable**: no

Time in milliseconds a sleeping worker thread waits before checking for work
again. Applies to all three pools.

## shared.worker.yield.threshold

- **Default**: `10`
- **Reloadable**: no

Number of idle cycles after which a worker thread yields its CPU time slice.
Applies to all three pools.

## shared.write.worker.affinity

- **Default**: none
Expand Down
5 changes: 4 additions & 1 deletion documentation/getting-started/capacity-planning.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,10 @@ for other database processes to use.

### CPU cores

By default, QuestDB tries to use all available CPU cores.
By default, QuestDB tries to use all available CPU cores. Work is spread across
three pools, sized independently: network I/O, query execution, and writes. Each
one defaults to roughly the core count, so the total thread count is the sum of
the three rather than a single pool of that size.
[The guide on shared worker configuration](/docs/configuration/shared-workers/)
explains how to change the default settings. Assuming that the disk is not
bottlenecked on IOPS, the throughput of read-only queries scales proportionally
Expand Down
Loading