From e88cea833cdaf6e07da56f1799557a617eaa35de Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Mon, 15 Jun 2026 13:06:24 +0100 Subject: [PATCH 01/14] docs: add EXPIRE ROWS materialized-view documentation Deep-dive concept page (concepts/deep-dive/expire-rows.md) for passthrough materialized views with EXPIRE ROWS: all modes, worked examples, read-filter/cleanup mechanics, NULL/ties/monotonicity semantics, and limitations. Adds an ALTER MATERIALIZED VIEW SET EXPIRE reference page, an EXPIRE ROWS section in CREATE MATERIALIZED VIEW, and sidebar entries. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../concepts/deep-dive/expire-rows.md | 416 ++++++++++++++++++ .../query/sql/alter-mat-view-set-expire.md | 112 +++++ documentation/query/sql/create-mat-view.md | 39 +- documentation/sidebars.js | 2 + 4 files changed, 568 insertions(+), 1 deletion(-) create mode 100644 documentation/concepts/deep-dive/expire-rows.md create mode 100644 documentation/query/sql/alter-mat-view-set-expire.md diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md new file mode 100644 index 0000000000..e2e05478a6 --- /dev/null +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -0,0 +1,416 @@ +--- +title: Expiring rows (EXPIRE ROWS) +sidebar_label: Expiring rows +description: + EXPIRE ROWS is a row-level retention policy for passthrough materialized + views. Keep the latest row per key, the top-N per group, or rows matching a + predicate — recomputed continuously, with expired rows hidden immediately and + reclaimed in the background. +--- + +`EXPIRE ROWS` is a row-level retention policy for +[materialized views](/docs/concepts/materialized-views/). Where +[TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE +ROWS` decides retention **row by row** — keep the latest row per key, the top-N +per group, rows matching a predicate, and so on — and recomputes the result +continuously as the view refreshes. + +Expired rows disappear from query results **immediately**; their on-disk storage +is reclaimed afterwards by a background job. + +:::note + +`EXPIRE ROWS` is **materialized-view-only**, and the view must be a +**passthrough** (non-aggregating) view: `SELECT * FROM base` with no +`SAMPLE BY` / `GROUP BY`. `CREATE TABLE ... EXPIRE ROWS` and aggregating views +are rejected. For base-table retention use [TTL](/docs/concepts/ttl/) or, on +Enterprise, [storage policies](/docs/concepts/storage-policy/). + +::: + +## When to use EXPIRE ROWS + +Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a +continuously-maintained, pruned copy of a base table: + +- **Current-state-per-key tables** — keep only the latest row per device, + symbol, or session (`KEEP LATEST`). +- **Per-group extremes or leaderboards** — keep the highest/lowest value per + group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). +- **Rolling row-level windows** — keep rows newer than a cutoff, or matching any + predicate, at finer granularity than TTL's whole-partition drops + (`WHEN predicate`). + +Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based +retention on a base table is enough — it is cheaper and has no passthrough-view +requirement. + +## Requirements + +`EXPIRE ROWS` requires a **passthrough materialized view**: + +- The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter + are allowed; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, `DISTINCT`, + `UNION`, and `JOIN` are not — they make the view non-passthrough and are + rejected with + `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views`. +- The view inherits the base table's + [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. + +A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it +is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that +replica down to the rows you want to keep — without touching the base table. + +## The modes + +Every mode keeps a defined set of rows and expires the rest. A row is expired +only when the rule selects it for removal. + +| Mode | What it keeps | Syntax | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a +window predicate, so the window `WHEN` is the general escape hatch. + +The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if present), or set later with +[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +| Element | Meaning | +| ------------------ | ---------------------------------------------------------------------------------------- | +| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view. Defaults to `1h` if omitted. | + +## Worked examples + +The following walks through every mode on a small fixed dataset so you can see +exactly which rows each policy keeps. + +### Setup + +```questdb-sql title="Base table and sample data" +CREATE TABLE trades ( + symbol SYMBOL, + price DOUBLE, + amount DOUBLE, + ts TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY WAL; + +INSERT INTO trades VALUES + ('BTC', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), + ('BTC', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), + ('BTC', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), + ('ETH', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), + ('ETH', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); +``` + +:::note + +A materialized view starts an **asynchronous initial refresh** on creation, so +it may briefly return no rows. Check progress with +`SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` +— the view is up to date when `refresh_base_table_txn = base_table_txn`. With +this small dataset that is effectively instant. + +::: + +### Per-row predicate: `WHEN` + +A per-row predicate expires a row when it evaluates `TRUE`. Here, expire small +trades (`amount < 1.5`): + +```questdb-sql title="Expire rows where amount < 1.5" +CREATE MATERIALIZED VIEW trades_sized AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN amount < 1.5; + +SELECT * FROM trades_sized; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | + +The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is +`FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, +not `TRUE` — see [NULLs](#nulls)). + +A predicate on the designated timestamp gives a **rolling retention window**, +re-evaluated on every read so the visible set rolls forward with the clock: + +```questdb-sql title="Keep the last 1 day" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < dateadd('d', -1, now()); +``` + +(With the 2024 sample timestamps above, every row is already older than a day +and would be hidden; use recent data to see rows retained.) + +### Keep latest per key: `KEEP LATEST` + +Keep only the most recent row per key — turning the passthrough view into a +live, current-state-per-symbol table: + +```questdb-sql title="Keep the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +SELECT * FROM trades_latest; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | + +One row per symbol — the latest by the designated timestamp `ts`. As new trades +arrive, the kept row advances automatically. `PARTITION BY` may list multiple key +columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but the `ON` +column must be the view's designated timestamp. + +### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` + +Keep the rows tied at the group maximum (or minimum) of a column: + +```questdb-sql title="Keep the highest-priced trade per symbol" +CREATE MATERIALIZED VIEW trades_peak AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_peak; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | + +`KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, +ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows +are kept (a `NULL` is never less than the max). + +### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` + +Keep a per-group leaderboard — the `N` highest (or lowest) rows: + +```questdb-sql title="Keep the 2 highest-priced trades per symbol" +CREATE MATERIALIZED VIEW trades_top2 AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_top2 ORDER BY symbol, price DESC; +``` + +| symbol | price | amount | ts | +| ------ | ----- | ------ | --------------------------- | +| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | + +BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, +so both survive. Ties are broken by the designated timestamp, so the N-th +boundary is deterministic. + +### Window predicate: the escape hatch + +`KEEP HIGHEST/LOWEST` and `KEEP N` are shorthand for window predicates. When you +need a rule they do not cover, write the window predicate directly in `WHEN`. +For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` +expands to: + +```questdb-sql title="Equivalent to KEEP HIGHEST, written as a window predicate" +CREATE MATERIALIZED VIEW trades_peak_win AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN price < max(price) OVER (PARTITION BY symbol); +``` + +A row expires when its price is below its symbol's maximum, so only the peak per +symbol survives — the same result as `trades_peak` above. From here you can +express richer rules, for example keeping rows within 5% of the peak +(`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window +(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). + +## How it works + +`EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a +best-effort background cleanup. + +### Read-time filter (authoritative) + +Every query against a policied view is transparently rewritten so that only the +kept rows are visible — **immediately, and regardless of whether cleanup has +run**. This is what makes results correct at all times: + +- **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB + filtering is three-valued, so `FALSE` **and** `NULL` are kept (see + [NULLs](#nulls)). +- **`KEEP LATEST`** returns the latest row per key using the designated + timestamp. +- **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a + window function over the whole view. + +Because the filter is applied at query time, a freshly-refreshed row that should +be expired is hidden the moment it lands, and a row that should reappear (under a +time-based predicate) reappears on the next read. + +### Physical cleanup (best-effort) + +A background job reclaims disk for non-active partitions: a fully-expired +partition is removed, and a partially-expired one is compacted down to its +survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is +**best-effort** — the read filter is authoritative, so deferred or skipped +reclamation only affects disk usage, never query results. + +On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation +still replicates: the compaction commits are ordinary WAL transactions, so +replicas reclaim the identical rows by applying them. A read-only replica neither +runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` +in `server.conf` (reads stay filtered; only reclamation stops). + +To observe reclamation, compare the physical row count per partition before and +after a sweep: + +```questdb-sql title="Physical rows still on disk per partition" +SELECT name, numRows FROM table_partitions('trades_latest'); +``` + +Reclamation **defers while a view is being refreshed continuously** and resumes +on a quiet sweep. + +## Semantics + +### NULLs + +The keep-set is computed with three-valued logic, so a `NULL` value is never +*less than* a group maximum (the comparison is `UNKNOWN`, not `TRUE`). +Therefore: + +- **`KEEP HIGHEST/LOWEST` and value-based `WHEN`** predicates **keep** rows whose + value is `NULL`. +- **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. +- **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB + has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under + `DESC` a floating-point `NULL` (NaN) sorts first (kept while there is room in + `N`), while an integer/timestamp `NULL` sorts last (expired first). Use + `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of + type. + +### Ties and determinism + +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by +construction. `KEEP N` makes the order total by appending the designated +timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base +table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` +is not already unique). + +### Monotonicity and cleanup safety + +Physical deletion is only safe when expiry is **monotonic**: a row that is +expired now must stay expired forever. All the relative modes (`KEEP LATEST`, +`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction, as is a +designated-timestamp predicate such as `WHEN ts < now()` (a row only gets +older). A scalar `WHEN predicate` is arbitrary SQL, so **monotonicity is the +author's responsibility**. + +:::warning + +A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that +**un-expire** as `now()` advances. The read filter recomputes `now()` on every +read and stays correct, but the cleanup job assumes monotonicity and may +physically delete a row that a later read would otherwise show (recoverable only +by a full refresh). Write `WHEN` predicates that expire things in the **past** +(`ts < now()`) or against fixed thresholds — never rows that the passage of time +will later keep. + +::: + +## Inspecting a policy + +`SHOW CREATE MATERIALIZED VIEW` renders the clause as written: + +```questdb-sql +SHOW CREATE MATERIALIZED VIEW trades_latest; +-- ... EXPIRE ROWS KEEP LATEST PARTITION BY symbol +``` + +The [`materialized_views()`](/docs/query/functions/meta/) function exposes the +policy in the `expire_predicate` and `expire_cleanup_every` columns (both +`NULL` when no policy is set): + +```questdb-sql title="List EXPIRE ROWS policies" +SELECT view_name, expire_predicate, expire_cleanup_every +FROM materialized_views(); +``` + +| view_name | expire_predicate | expire_cleanup_every | +| ------------- | ------------------------------- | -------------------- | +| trades_sized | amount < 1.5 | 1h | +| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | +| trades_top2 | KEEP 2 HIGHEST price ... | 1h | + +## Changing or removing a policy + +Set, change, or drop a policy on an existing passthrough view — see +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): + +```questdb-sql +-- set or replace the policy +ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +-- remove it (keeps all rows again) +ALTER MATERIALIZED VIEW trades_latest DROP EXPIRE; +``` + +`SET EXPIRE ROWS` validates the new policy against the view's columns before +applying it, so an invalid predicate or an unknown column is rejected up front +rather than breaking subsequent reads. + +## Limitations and operational notes + +- **Reads recompute the keep-set.** A relative/window policy computes its + keep-set over the whole physical view on every read. `KEEP LATEST` on an + [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window + modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` + keeps the physical residue — and therefore the read cost — small. +- **Cleanup defers under continuous refresh.** Reclamation only proceeds when the + view is quiescent and fully applied, so a view being refreshed continuously + defers reclamation to a quiet sweep. The read filter stays authoritative + meanwhile. +- **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but + the view's designated timestamp is always used; naming a different column is + rejected. +- **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see + [monotonicity](#monotonicity-and-cleanup-safety) above. + +## Related documentation + +- [Materialized views](/docs/concepts/materialized-views/) — the view type + `EXPIRE ROWS` runs on +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create + syntax, including the `EXPIRE ROWS` clause +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) + — set, change, or drop a policy +- [Time To Live (TTL)](/docs/concepts/ttl/) — partition-level retention by age +- [Storage policy](/docs/concepts/storage-policy/) — graduated partition + lifecycle (Enterprise) diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md new file mode 100644 index 0000000000..8c3d4216b8 --- /dev/null +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -0,0 +1,112 @@ +--- +title: ALTER MATERIALIZED VIEW SET EXPIRE +sidebar_label: SET EXPIRE +description: + ALTER MATERIALIZED VIEW SET EXPIRE ROWS / DROP EXPIRE SQL keyword reference + documentation. +--- + +Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) +row-retention policy on a **passthrough** materialized view. Expired rows are +hidden from queries immediately and reclaimed on disk in the background. + +## Syntax + +``` +ALTER MATERIALIZED VIEW viewName SET EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] + +ALTER MATERIALIZED VIEW viewName DROP EXPIRE +``` + +## Parameters + +| Parameter | Description | +| ---------------- | --------------------------------------------------------------------------------- | +| `viewName` | Name of the passthrough materialized view to modify | +| `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | +| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | +| `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | + +For the full description of each mode and its semantics, see the +[Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. + +## When to use + +- Add a retention policy to a passthrough view created without one +- Switch a view between modes (e.g. from a value predicate to `KEEP LATEST`) +- Tune the `CLEANUP EVERY` cadence +- Remove a policy with `DROP EXPIRE` so the view keeps all rows again + +## How it works + +`SET EXPIRE ROWS` validates the new policy against the view's columns first +(compiling the predicate / checking the key columns), so an invalid predicate or +an unknown column is rejected immediately rather than breaking later reads. Once +set, queries against the view are filtered to the kept rows immediately; physical +reclamation follows in the background. See +[How it works](/docs/concepts/deep-dive/expire-rows/#how-it-works). + +## Examples + +```questdb-sql title="Per-row predicate, with a tighter cleanup cadence" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS WHEN amount < 1.5 CLEANUP EVERY 30m; +``` + +```questdb-sql title="Keep the latest row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +```questdb-sql title="Keep the highest-priced row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Keep the 2 highest-priced rows per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Remove the policy" +ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; +``` + +## Behavior + +| Aspect | Description | +| ----------------- | ---------------------------------------------------------------------------- | +| Passthrough only | The view must be `SELECT * FROM base`; aggregating views are rejected | +| Validation | The policy is checked against the view's columns before it is applied | +| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | +| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | + +## Permissions (Enterprise) + +Changing the policy requires the `ALTER MATERIALIZED VIEW` permission: + +```questdb-sql title="Grant alter permission" +GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `materialized view does not exist` | View with the specified name doesn't exist | +| `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views` | View aggregates (e.g. `SAMPLE BY`) | +| `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | +| `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | + +## See also + +- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/deep-dive/expire-rows/) +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) +- [Materialized views concept](/docs/concepts/materialized-views/) +- [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 0dd9739535..19593d34cf 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -22,13 +22,16 @@ AS [ ( ] query [ ) ] [ TIMESTAMP ( columnRef ) ] [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) [ TTL n timeUnit ] ] +[ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] [ OWNED BY ownerName ] ``` Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` +- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the + passthrough views used by `EXPIRE ROWS` — see below) +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — passthrough views only ## Parameters @@ -43,6 +46,7 @@ Where: | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | +| `EXPIRE ROWS` | Row-level retention for passthrough views (see below) | | `OWNED BY` | Assign ownership (Enterprise) | ## Rules and defaults @@ -278,6 +282,39 @@ Time units: `HOURS`, `DAYS`, `WEEKS`, `MONTHS`, `YEARS` The view's TTL is independent of the base table's TTL. See [TTL documentation](/docs/concepts/ttl/) for details. +## EXPIRE ROWS + +Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops +whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the latest +per key, the top-N per group, or rows matching a predicate — recomputed +continuously as the view refreshes. + +`EXPIRE ROWS` applies **only to passthrough (non-aggregating) views** — the query +is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`: + +```questdb-sql title="Passthrough view that keeps the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +The clause goes after the query (and after `PARTITION BY` if present): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +Expired rows are hidden from queries immediately and reclaimed on disk in the +background (`CLEANUP EVERY`, default `1h`). Change or remove a policy with +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). + +See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for +all modes, worked examples, and semantics (NULLs, ties, monotonicity). + ## Complete example Putting it all together: diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 9955129082..d9fa245edb 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -311,6 +311,7 @@ module.exports = { "query/sql/alter-mat-view-set-refresh", "query/sql/alter-mat-view-set-refresh-limit", "query/sql/alter-mat-view-set-ttl", + "query/sql/alter-mat-view-set-expire", ], }, { @@ -546,6 +547,7 @@ module.exports = { items: [ "concepts/deep-dive/indexes", "concepts/deep-dive/posting-index", + "concepts/deep-dive/expire-rows", "concepts/deep-dive/interval-scan", "concepts/deep-dive/jit-compiler", "concepts/deep-dive/query-tracing", From 3fd5d8d7c4066784ce8a1b606ebe84efc36211d4 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 21 Jul 2026 15:35:59 +0200 Subject: [PATCH 02/14] docs: align EXPIRE ROWS pages with the implementation Corrections and additions ported from the core repo's in-repo design doc (docs/row-expiry.md, now removed there): - Aggregating views are ACCEPTED with a logged advisory, not rejected (the previously-cited error message does not exist); reworded the concept note, Requirements, the ALTER behavior and errors tables, and the CREATE section accordingly. - materialized_views() exposes the policy as expire_clause, not expire_predicate. - Monotonicity: list the exact clock shapes the cleanup job proves monotonic (bare clock, clock minus non-negative constant, fixed-unit look-back dateadd) and that calendar units, look-forward offsets, compound arithmetic, and window predicates skip cleanup; the job skips rather than risks deleting rows. - No-policied-chains rule: CREATE rejects a defining query reading a policied view (base or join); SET EXPIRE is rejected with dependent views; corresponding errors added to the ALTER page. - Reserved __qdb_re_keep column name; no line comments inside the clause; CLEANUP EVERY strict grammar (s/m/h/d/w). - Kill switch is read at startup (restart required); failing sweeps back off from 1s up to a 10-minute cap. - Parquet side effect: compacting a partially-expired Parquet partition rewrites it as native storage until re-conversion. Co-Authored-By: Claude Fable 5 --- .../concepts/deep-dive/expire-rows.md | 85 +++++++++++++------ .../query/sql/alter-mat-view-set-expire.md | 22 +++-- documentation/query/sql/create-mat-view.md | 9 +- 3 files changed, 80 insertions(+), 36 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index e2e05478a6..70cd194159 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -20,11 +20,15 @@ is reclaimed afterwards by a background job. :::note -`EXPIRE ROWS` is **materialized-view-only**, and the view must be a -**passthrough** (non-aggregating) view: `SELECT * FROM base` with no -`SAMPLE BY` / `GROUP BY`. `CREATE TABLE ... EXPIRE ROWS` and aggregating views -are rejected. For base-table retention use [TTL](/docs/concepts/ttl/) or, on -Enterprise, [storage policies](/docs/concepts/storage-policy/). +`EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is +rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT +* FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base +rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a +logged advisory**: reads stay filtered, but a later refresh can regenerate +reclaimed rows from base rows that still exist (see +[Requirements](#requirements)). For base-table retention use +[TTL](/docs/concepts/ttl/) or, on Enterprise, +[storage policies](/docs/concepts/storage-policy/). ::: @@ -47,13 +51,11 @@ requirement. ## Requirements -`EXPIRE ROWS` requires a **passthrough materialized view**: +`EXPIRE ROWS` is designed for a **passthrough materialized view**: - The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter - are allowed; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, `DISTINCT`, - `UNION`, and `JOIN` are not — they make the view non-passthrough and are - rejected with - `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views`. + keep the view passthrough; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, + `DISTINCT`, `UNION`, and `JOIN` make it non-passthrough. - The view inherits the base table's [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. @@ -61,6 +63,17 @@ A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that replica down to the rows you want to keep — without touching the base table. +A **non-passthrough (aggregating) view is accepted with a logged advisory** +rather than rejected: the read filter still hides expired rows, but physical +reclamation only sticks when base-table retention is aligned with the expiry +horizon — a later incremental or full refresh can regenerate a reclaimed row +from base rows that still exist. + +A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a +defining query that reads a policied view (as its base or in a join), and +`ALTER ... SET EXPIRE` is rejected on a view that other materialized views +derive from — otherwise those views would copy expired rows on refresh. + ## The modes Every mode keeps a defined set of rows and expires the rest. A row is expired @@ -95,7 +108,7 @@ EXPIRE ROWS | `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | | `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | | `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | -| `CLEANUP EVERY` | How often the background reclamation job runs for this view. Defaults to `1h` if omitted. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | ## Worked examples @@ -286,7 +299,9 @@ On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` -in `server.conf` (reads stay filtered; only reclamation stops). +in `server.conf` (reads stay filtered; only reclamation stops); the setting is +read at startup, so changing it requires a restart. A failing sweep retries +after one second, doubling the per-view retry gap up to a 10-minute cap. To observe reclamation, compare the physical row count per partition before and after a sweep: @@ -328,20 +343,32 @@ is not already unique). Physical deletion is only safe when expiry is **monotonic**: a row that is expired now must stay expired forever. All the relative modes (`KEEP LATEST`, -`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction, as is a -designated-timestamp predicate such as `WHEN ts < now()` (a row only gets -older). A scalar `WHEN predicate` is arbitrary SQL, so **monotonicity is the -author's responsibility**. +`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction. A scalar +`WHEN predicate` is arbitrary SQL, so the cleanup job reclaims disk only for +predicates it can **prove** monotonic: + +- clock-free predicates (`WHEN amount < 1.5`), and +- designated-timestamp thresholds of a proven advancing-clock shape: a bare + clock (`ts < now()`), a bare clock minus a non-negative constant + (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare + clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). + +Anything else **skips cleanup**: calendar units (`dateadd('M', -1, now())` — a +month is a variable amount), look-forward offsets (`dateadd('h', 1, now())`), +further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` +predicates. A skipped policy stays correct at read time (the filter recomputes +on every read), but its disk is not reclaimed until the policy is changed to a +proven shape. :::warning A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that **un-expire** as `now()` advances. The read filter recomputes `now()` on every -read and stays correct, but the cleanup job assumes monotonicity and may -physically delete a row that a later read would otherwise show (recoverable only -by a full refresh). Write `WHEN` predicates that expire things in the **past** -(`ts < now()`) or against fixed thresholds — never rows that the passage of time -will later keep. +read and stays correct, and the cleanup job skips such a policy rather than +risk physically deleting a row a later read must show — at the cost of disk +never being reclaimed for it. Write `WHEN` predicates that expire things in the +**past** or against fixed thresholds — never rows that the passage of time will +later keep. ::: @@ -355,15 +382,15 @@ SHOW CREATE MATERIALIZED VIEW trades_latest; ``` The [`materialized_views()`](/docs/query/functions/meta/) function exposes the -policy in the `expire_predicate` and `expire_cleanup_every` columns (both +policy in the `expire_clause` and `expire_cleanup_every` columns (both `NULL` when no policy is set): ```questdb-sql title="List EXPIRE ROWS policies" -SELECT view_name, expire_predicate, expire_cleanup_every +SELECT view_name, expire_clause, expire_cleanup_every FROM materialized_views(); ``` -| view_name | expire_predicate | expire_cleanup_every | +| view_name | expire_clause | expire_cleanup_every | | ------------- | ------------------------------- | -------------------- | | trades_sized | amount < 1.5 | 1h | | trades_latest | KEEP LATEST PARTITION BY symbol | 1h | @@ -402,6 +429,16 @@ rather than breaking subsequent reads. rejected. - **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see [monotonicity](#monotonicity-and-cleanup-safety) above. +- **Reserved column name.** The window/keep modes compute the keep-set through a + synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view + that exposes a column with that name. +- **No line comments in the clause.** The clause text is stored verbatim and + embedded into generated SQL, so `--` comments are rejected inside an + `EXPIRE ROWS` clause; terminated block comments (`/* ... */`) are fine. +- **Compacting a Parquet partition rewrites it as native storage.** When cleanup + compacts a *partially*-expired partition held in Parquet, the partition + reverts to native QuestDB storage until the Parquet-conversion job re-converts + it. Reclamation correctness is unaffected. ## Related documentation diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 8c3d4216b8..17367847d4 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -7,8 +7,9 @@ description: --- Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) -row-retention policy on a **passthrough** materialized view. Expired rows are -hidden from queries immediately and reclaimed on disk in the background. +row-retention policy on a materialized view (designed for **passthrough** +views — see the concept page). Expired rows are hidden from queries immediately +and reclaimed on disk in the background. ## Syntax @@ -79,12 +80,13 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; ## Behavior -| Aspect | Description | -| ----------------- | ---------------------------------------------------------------------------- | -| Passthrough only | The view must be `SELECT * FROM base`; aggregating views are rejected | -| Validation | The policy is checked against the view's columns before it is applied | -| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | -| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | +| Aspect | Description | +| ----------------------- | ---------------------------------------------------------------------------- | +| Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | +| No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | +| Validation | The policy is checked against the view's columns before it is applied | +| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | +| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | ## Permissions (Enterprise) @@ -99,9 +101,11 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | -| `EXPIRE ROWS is only supported on passthrough (non-aggregating) materialized views` | View aggregates (e.g. `SAMPLE BY`) | +| `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | +| `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | ## See also diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 19593d34cf..5c80d36caf 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -31,7 +31,7 @@ Where: - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` - `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the passthrough views used by `EXPIRE ROWS` — see below) -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — passthrough views only +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) ## Parameters @@ -289,8 +289,11 @@ whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the late per key, the top-N per group, or rows matching a predicate — recomputed continuously as the view refreshes. -`EXPIRE ROWS` applies **only to passthrough (non-aggregating) views** — the query -is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`: +`EXPIRE ROWS` is designed for **passthrough (non-aggregating) views** — the +query is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`. An aggregating +view is accepted with a logged advisory (a later refresh can regenerate +reclaimed rows). The defining query must not read another policied view, as its +base or in a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( From 6abf656b3f7fe9a077a15600587fcf5b1d0e0847 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 18 Aug 2026 14:52:55 +0200 Subject: [PATCH 03/14] Update docs to match changes in code --- .../concepts/deep-dive/expire-rows.md | 98 +++++++++++++------ documentation/query/functions/meta.md | 47 +++++++-- .../query/sql/alter-mat-view-set-expire.md | 3 +- documentation/query/sql/create-mat-view.md | 7 +- 4 files changed, 116 insertions(+), 39 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 70cd194159..310fe73534 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -4,8 +4,8 @@ sidebar_label: Expiring rows description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a - predicate — recomputed continuously, with expired rows hidden immediately and - reclaimed in the background. + predicate — recomputed continuously, with expired rows hidden immediately, and + reclaimed in the background under the modes that allow it. --- `EXPIRE ROWS` is a row-level retention policy for @@ -15,8 +15,12 @@ ROWS` decides retention **row by row** — keep the latest row per key, the top- per group, rows matching a predicate, and so on — and recomputes the result continuously as the view refreshes. -Expired rows disappear from query results **immediately**; their on-disk storage -is reclaimed afterwards by a background job. +Expired rows disappear from query results **immediately** in every mode. Their +on-disk storage is reclaimed afterwards by a background job under a monotonic +`WHEN` predicate; the relative modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, +`KEEP N`) and window predicates hide rows without freeing disk. See +[The modes](#the-modes) and +[Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). :::note @@ -79,17 +83,27 @@ derive from — otherwise those views would copy expired rows on refresh. Every mode keeps a defined set of rows and expires the rest. A row is expired only when the rule selects it for removal. -| Mode | What it keeps | Syntax | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. +The **Frees disk** column is the difference between hiding a row and deleting +it. Every mode hides expired rows from every read, immediately. Only a monotonic +`WHEN` predicate also has those rows deleted from disk by the cleanup job: a +mode whose keep-set depends on the other rows in the view keeps its expired rows +on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore +holds a full copy of its base table unless the view's own +[TTL](/docs/concepts/ttl/) bounds it. +`materialized_views().expire_enforcement` reports which of the two a given view +gets; see [Inspecting a policy](#inspecting-a-policy). + The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the query, and after `PARTITION BY` if present), or set later with [`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): @@ -295,13 +309,21 @@ survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** — the read filter is authoritative, so deferred or skipped reclamation only affects disk usage, never query results. +The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, +`KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh +can remove the row those modes currently keep, which promotes an older row back +into the keep-set, and the job cannot reconstruct a row it has already deleted. +Those views accumulate their expired rows on disk. + On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither -runs the job nor needs to. Disable the job with `cairo.row.expiry.enabled=false` -in `server.conf` (reads stay filtered; only reclamation stops); the setting is -read at startup, so changing it requires a restart. A failing sweep retries -after one second, doubling the per-view retry gap up to a 10-minute cap. +runs the job nor needs to. Disable the job with +`cairo.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay filtered; +only reclamation stops — the setting does not disable `EXPIRE ROWS` itself); the +setting is read at startup, so changing it requires a restart. A failing sweep +retries after one second, doubling the per-view retry gap up to a 10-minute +cap. To observe reclamation, compare the physical row count per partition before and after a sweep: @@ -342,10 +364,17 @@ is not already unique). ### Monotonicity and cleanup safety Physical deletion is only safe when expiry is **monotonic**: a row that is -expired now must stay expired forever. All the relative modes (`KEEP LATEST`, -`KEEP HIGHEST/LOWEST`, `KEEP N`) are monotonic by construction. A scalar -`WHEN predicate` is arbitrary SQL, so the cleanup job reclaims disk only for -predicates it can **prove** monotonic: +expired now must stay expired forever. Two separate things can break that. + +The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, +window `WHEN`) decide each row's fate by comparing it against the other rows in +the view. A later refresh can remove or replace the row a key currently keeps, +which promotes an older row back into the keep-set — so the cleanup job never +deletes for these modes, whatever their predicate looks like. + +A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is +arbitrary SQL, so the cleanup job reclaims disk only for predicates it can +**prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare @@ -382,19 +411,27 @@ SHOW CREATE MATERIALIZED VIEW trades_latest; ``` The [`materialized_views()`](/docs/query/functions/meta/) function exposes the -policy in the `expire_clause` and `expire_cleanup_every` columns (both -`NULL` when no policy is set): +policy in the `expire_clause`, `expire_cleanup_every` and `expire_enforcement` +columns (all `NULL` when no policy is set): ```questdb-sql title="List EXPIRE ROWS policies" -SELECT view_name, expire_clause, expire_cleanup_every +SELECT view_name, expire_clause, expire_cleanup_every, expire_enforcement FROM materialized_views(); ``` -| view_name | expire_clause | expire_cleanup_every | -| ------------- | ------------------------------- | -------------------- | -| trades_sized | amount < 1.5 | 1h | -| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | -| trades_top2 | KEEP 2 HIGHEST price ... | 1h | +| view_name | expire_clause | expire_cleanup_every | expire_enforcement | +| ------------- | ------------------------------- | -------------------- | ------------------ | +| trades_sized | amount < 1.5 | 1h | FILTER_AND_RECLAIM | +| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | +| trades_top2 | KEEP 2 HIGHEST price ... | 1h | FILTER_ONLY | + +`expire_enforcement` is the verdict the cleanup job acts on: + +- `FILTER_AND_RECLAIM` — reads hide the expired rows and the job deletes them + from disk. +- `FILTER_ONLY` — reads hide the expired rows and they stay on disk. Every + relative and window policy reports this, as does a `WHEN` predicate that + cannot be proven monotonic. ## Changing or removing a policy @@ -427,8 +464,11 @@ rather than breaking subsequent reads. - **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but the view's designated timestamp is always used; naming a different column is rejected. -- **Non-monotonic `WHEN` predicates are unsupported for cleanup** — see - [monotonicity](#monotonicity-and-cleanup-safety) above. +- **Only a monotonic `WHEN` predicate frees disk.** The relative and window + modes hide expired rows on every read but never delete them, and neither does + a `WHEN` predicate that cannot be proven monotonic — see + [monotonicity](#monotonicity-and-cleanup-safety) above. Check a view's verdict + with `materialized_views().expire_enforcement`. - **Reserved column name.** The window/keep modes compute the keep-set through a synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view that exposes a column with that name. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 1c78bfce82..6d0254aa6f 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -262,25 +262,58 @@ Returns a `table` including the following information: - `view_table_dir_name` - view directory name - `invalidation_reason` - message explaining why the view was marked as invalid - `view_status` - view status: 'valid', 'refreshing', or 'invalid' +- `refresh_period_hi` - end of the last refreshed period, for a view defined + with a `REFRESH PERIOD` - `refresh_base_table_txn` - the last base table transaction used to refresh the materialized view - `base_table_txn` - the last committed transaction in the base table -- `refresh_limit_value` - how many units back in time the refresh limit goes +- `refresh_limit` - how many units back in time the refresh limit goes - `refresh_limit_unit` - how long each unit is +- `timer_time_zone` - time zone the refresh timer and period boundaries use - `timer_start` - start date for the scheduled refresh timer -- `timer_interval_value` - how many interval units between each refresh +- `timer_interval` - how many interval units between each refresh - `timer_interval_unit` - how long each unit is +- `period_length` - length of one refresh period (`PERIOD (LENGTH ...)`) +- `period_length_unit` - how long each period-length unit is +- `period_delay` - delay before a completed period is refreshed + (`PERIOD (DELAY ...)`) +- `period_delay_unit` - how long each period-delay unit is +- `refresh_avg_commit_nanos` - moving average of one refresh commit, in + nanoseconds. Held in memory only, so it resets on restart +- `refresh_avg_scan_sample_nanos` - moving average of one refresh iteration's + base-table scan, in nanoseconds. In memory only +- `refresh_avg_scan_range_ts_units` - moving average of the timestamp range one + refresh iteration covers, in the base table's timestamp unit. In memory only +- `refresh_gap_threshold_ts_units` - timestamp gap below which the refresh job + merges two adjacent intervals instead of paying for a second commit. `0` means + merging is disabled +- `expire_clause` - the view's + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy as written, or + `NULL` when the view has no policy +- `expire_cleanup_every` - how often the cleanup job runs for the policy, or + `NULL` +- `expire_enforcement` - `FILTER_AND_RECLAIM` when the cleanup job frees disk + for the policy, `FILTER_ONLY` when reads hide the expired rows but they stay + on disk, `NULL` when the view has no policy **Examples:** +`materialized_views()` on its own returns every column listed above. The example +below projects a readable subset over three views: an aggregating one with no +retention policy, and two passthrough views with an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. + ```questdb-sql title="List all materialized views" -materialized_views(); +SELECT view_name, view_status, base_table_name, refresh_base_table_txn, + base_table_txn, expire_clause, expire_cleanup_every, expire_enforcement +FROM materialized_views(); ``` -| view_name | refresh_type | base_table_name | last_refresh_start_timestamp | last_refresh_finish_timestamp | view_sql | view_table_dir_name | invalidation_reason | view_status | refresh_base_table_txn | base_table_txn | refresh_limit_value | refresh_limit_unit | timer_start | timer_interval_value | timer_interval_unit | -|------------------|--------------|-----------------|------------------------------|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|-------------|------------------------|----------------|---------------------|--------------------|-------------|----------------------|---------------------| -| trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | -| trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | +| view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | +| --------------- | ----------- | --------------- | ---------------------- | -------------- | ------------------------------------ | -------------------- | ------------------ | +| trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | +| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 17367847d4..c02337bdb9 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -9,7 +9,8 @@ description: Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) row-retention policy on a materialized view (designed for **passthrough** views — see the concept page). Expired rows are hidden from queries immediately -and reclaimed on disk in the background. +in every mode, and reclaimed on disk in the background under a monotonic `WHEN` +predicate — the relative and window modes hide rows without freeing disk. ## Syntax diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 5c80d36caf..ee597a5377 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -311,8 +311,11 @@ EXPIRE ROWS [ CLEANUP EVERY duration ] ``` -Expired rows are hidden from queries immediately and reclaimed on disk in the -background (`CLEANUP EVERY`, default `1h`). Change or remove a policy with +Expired rows are hidden from queries immediately in every mode. They are +reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a +monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and +window predicates hide rows without freeing disk. Change or remove a policy +with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for From e01ce4fc599c2a38fea720abe12b5a1ad53b02d2 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Wed, 19 Aug 2026 10:56:38 +0200 Subject: [PATCH 04/14] Clarified interaction between TTL and EXPIRE ROWS --- .../concepts/deep-dive/expire-rows.md | 32 ++++++++++++++++++- documentation/query/sql/create-mat-view.md | 5 +++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 310fe73534..63f5310d94 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -100,7 +100,8 @@ it. Every mode hides expired rows from every read, immediately. Only a monotonic mode whose keep-set depends on the other rows in the view keeps its expired rows on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore holds a full copy of its base table unless the view's own -[TTL](/docs/concepts/ttl/) bounds it. +[TTL](/docs/concepts/ttl/) bounds it — see +[Combining with TTL](#combining-with-ttl). `materialized_views().expire_enforcement` reports which of the two a given view gets; see [Inspecting a policy](#inspecting-a-policy). @@ -361,6 +362,35 @@ timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` is not already unique). +### Combining with TTL + +A view can carry a [TTL](/docs/concepts/ttl/) and an `EXPIRE ROWS` policy at the +same time, and the order is fixed: **TTL removes rows from the view first, then +the policy applies to the rows that stay.** TTL drops whole partitions from the +view's own storage as they age out, and the keep-set is computed over what +remains. + +```questdb-sql title="Highest price per symbol, over a 3-day window" +CREATE MATERIALIZED VIEW trades_peak_3d AS ( + SELECT * FROM trades +) PARTITION BY DAY TTL 3 DAYS + EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +`TTL` goes before `EXPIRE ROWS` in the statement, as it does after any +`PARTITION BY`. + +This view reports the highest price of the **last three days**, so its answer +can go **down** as the window moves: when the day holding a symbol's maximum +ages out, the next-highest price within the window takes over. That is what the +two clauses ask for together — the view is no longer "the highest price ever", +it is "the highest price still retained". The base table is unaffected; it keeps +whatever its own retention settings keep. + +TTL is also the only control that bounds the size of a `KEEP LATEST`, +`KEEP HIGHEST/LOWEST` or `KEEP N` view, since the cleanup job never reclaims +disk for those modes. + ### Monotonicity and cleanup safety Physical deletion is only safe when expiry is **monotonic**: a row that is diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index ee597a5377..c4073b4e24 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -318,6 +318,11 @@ window predicates hide rows without freeing disk. Change or remove a policy with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). +A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the +statement and first in effect: it removes rows from the view, and the +`EXPIRE ROWS` policy then applies to the rows that stay — see +[Combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). + See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for all modes, worked examples, and semantics (NULLs, ties, monotonicity). From 6e444360b064e07013f1c70f8b11a24f4fd2c918 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 24 Aug 2026 15:27:30 +0200 Subject: [PATCH 05/14] Document passthrough views as a view shape Passthrough materialized views only appeared in the docs as a prerequisite of EXPIRE ROWS. The single sentence describing which queries qualify lived in the EXPIRE ROWS concept page, and every example on every page was literally "SELECT * FROM base", so a reader who wanted a maintained subset of a table without a retention policy had no reason to find any of it. Three statements also still said a materialized view query has to aggregate, which is no longer true: - "Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query." in the concepts page - "Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column" in the same page's Technical reference - "Query must aggregate" in the CREATE MATERIALIZED VIEW rules table Give the shape its own section in the general materialized views concept page: what a passthrough view is, when to reach for one (a narrowed replica of a big table, or a target for EXPIRE ROWS), what it inherits from the base table, and which queries qualify. The inherited-properties part records something documented nowhere before: a passthrough view inherits the base table's symbol indexes, under whatever alias the projection gives the column, so an indexed lookup on the view costs what it costs on the base. An aggregating view never inherits an index, because its rows are not base rows. The query table lists shapes verified by running them, rather than read off isPassthrough(). A column subset, a column alias and a row-local expression such as "price * amount AS notional" are all passthrough. A window function is rejected, but not by either check inside isPassthrough() -- a separate guard reports window function on base table is not supported for materialized views Dropping the designated timestamp, including by an ORDER BY on another column, is rejected with materialized view query is required to have designated timestamp and REFRESH PERIOD is rejected with PERIOD is not supported for non-aggregating (passthrough) materialized views while REFRESH MANUAL and REFRESH EVERY are both accepted. Add a section to the EXPIRE ROWS page on choosing between a WHERE clause in the view's query and an EXPIRE ROWS WHEN predicate. For a deterministic predicate that judges each row on its own the two keep the same rows, and nothing said how to pick between them. A WHERE clause is cheaper on storage, on reads and on writes, because the row is never written. An EXPIRE ROWS policy can be retuned with ALTER instead of forcing a drop and re-create, which re-reads the base table and can lose rows the base no longer holds. A policied view also cannot serve as another view's base. Also correct the claim that JOIN enrichment fails because SAMPLE BY is mandatory. A passthrough view does keep raw rows; it just has to read a single table, so enrichment stays unavailable for a different reason. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 73 ++++++++++++- documentation/concepts/materialized-views.md | 103 +++++++++++++++++- documentation/query/sql/create-mat-view.md | 21 ++-- 3 files changed, 179 insertions(+), 18 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 63f5310d94..5a0cdbbee4 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -57,11 +57,13 @@ requirement. `EXPIRE ROWS` is designed for a **passthrough materialized view**: -- The view query is `SELECT * FROM base`. A column subset and a `WHERE` filter - keep the view passthrough; aggregation, `SAMPLE BY`, `GROUP BY`, `LATEST ON`, - `DISTINCT`, `UNION`, and `JOIN` make it non-passthrough. +- The view query keeps view rows 1:1 with base rows — a projection over a single + table, with or without a `WHERE` filter. See + [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) + for the full rules. - The view inherits the base table's - [designated timestamp](/docs/concepts/designated-timestamp/) and partitioning. + [designated timestamp](/docs/concepts/designated-timestamp/), partitioning and + symbol indexes. A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that @@ -78,6 +80,67 @@ defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views derive from — otherwise those views would copy expired rows on refresh. +## `WHERE` filter or `EXPIRE ROWS`? + +A passthrough view can exclude rows in two places: a `WHERE` clause in its +defining query, or an `EXPIRE ROWS WHEN` predicate. For a deterministic, +row-local predicate the two describe the same set of surviving rows, so the +choice is not about what the view contains — it is about whether the rule is +part of the view's *definition* or a *knob you expect to turn*. + +**Default to `WHERE`.** A row a `WHERE` clause excludes is never written, so it +costs nothing at any stage: + +| | `WHERE` in the query | `EXPIRE ROWS WHEN` | +| --- | --- | --- | +| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Read cost | None | The keep-set filter is applied on every read of the view | +| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Can be another view's base | Yes | No — a policied view is rejected as a base | + +That last row is a hard constraint rather than a preference. If any other view +will read this one, the policy is not available and the predicate has to go in +the `WHERE` clause. + +**Reach for `EXPIRE ROWS` when the cutoff is a knob.** There is no +`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means +dropping the view and re-creating it, which re-materializes it from the base. +Changing a policy is a metadata operation: + +```questdb-sql title="Retuning a retention horizon without a rebuild" +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; +``` + +The rebuild a `WHERE` change forces is not only slow, it can lose data: if the +base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a +base that no longer holds everything the view held. A view whose retention +horizon is longer than its base table's cannot afford to be rebuilt, so its +cutoff belongs in a policy. + +Two things only `EXPIRE ROWS` can express at all, whichever way you lean: + +- **Rules that move with the clock.** A view's defining query rejects + non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not + accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported way + to write a rolling window. +- **Rules that compare rows against each other.** `KEEP LATEST`, + `KEEP N HIGHEST/LOWEST` and window predicates have no `WHERE` equivalent — a + `LATEST ON` or a window function in the defining query makes the view + non-passthrough. + +In practice the two compose, and on a passthrough view that is usually the right +shape: the `WHERE` clause fixes what the view is about, and the policy fixes how +long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + ## The modes Every mode keeps a defined set of rows and expires the rest. A row is expired @@ -514,6 +577,8 @@ rather than breaking subsequent reads. - [Materialized views](/docs/concepts/materialized-views/) — the view type `EXPIRE ROWS` runs on +- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — + the non-aggregating view shape `EXPIRE ROWS` is designed for - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create syntax, including the `EXPIRE ROWS` clause - [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index cf7ed4ac26..f8f3e7eea8 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -12,6 +12,11 @@ their results at query time, materialized views persist their data to disk, making them particularly efficient for expensive aggregate queries that are run frequently. +Most materialized views aggregate, bucketing base rows with `SAMPLE BY` or a +time-based `GROUP BY`. A view can also be a +[passthrough view](#passthrough-views), which projects base rows one-for-one +instead of summarising them. + ## What are materialized views for? Let's say your application ingests trade data into a table like this: @@ -116,9 +121,10 @@ reads on a smaller, pre-aggregated dataset. ### Not suited for: data enrichment -Materialized views support JOINs, but `SAMPLE BY` (aggregation) is mandatory. -This means you can enrich aggregated results with data from other tables, but -you cannot keep raw (non-aggregated) rows while adding enrichment columns. +Materialized views support JOINs only in an aggregating query. A +[passthrough view](#passthrough-views) keeps raw rows, but it must read a single +table. So neither shape lets you keep raw rows while adding columns from another +table. For example, joining aggregated trades with instrument metadata works: @@ -157,6 +163,89 @@ refresh. Changes to joined tables do not trigger updates. **Coming soon**: We are actively developing a new type of materialized view that will support data enrichment use cases. Stay tuned for updates. +## Passthrough views + +Not every materialized view aggregates. A **passthrough view** projects base +rows one-for-one instead of bucketing them, so the view is a +continuously-maintained copy of its base table, optionally narrowed to a subset +of columns, a subset of rows, or both. Its query has no `SAMPLE BY` and no +time-based `GROUP BY`: + +```questdb-sql title="Passthrough view: a maintained, filtered copy of trades" +CREATE MATERIALIZED VIEW trades_btc AS ( + SELECT timestamp, symbol, price, amount + FROM trades + WHERE symbol = 'BTC' +); +``` + +Reach for one when you want a maintained *subset* of a large table rather than a +summary of it: + +- **A narrowed replica** — one symbol, one tenant, one region, or a handful of + columns out of a wide table, kept current automatically and queried without + the base table's scan cost. +- **A row-level retention target** — a passthrough view is the shape + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) is built for. Its rows + *are* base rows, so reclaiming one is permanent. + +Passthrough views refresh incrementally like any other materialized view, and +accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and +`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets +for a period to align to. + +### What a passthrough view inherits + +A passthrough view takes its shape from the base table instead of from a +`SAMPLE BY` clause: + +- **Designated timestamp and partitioning** come from the base table. The + projection has to keep the designated timestamp; a query that drops it is + rejected with `materialized view query is required to have designated + timestamp`. `PARTITION BY` and `TTL` can still be stated explicitly. +- **Symbol indexes are inherited.** A base column declared `SYMBOL INDEX` stays + indexed in the view, under whatever alias the projection gives it, so an + indexed lookup on the view costs what it costs on the base. An aggregating + view never inherits an index, because its rows are not base rows. + +### Which queries are passthrough + +The rule is that view rows stay 1:1 with base rows. A projection over a single +table qualifies, with or without a filter: + +| Query | | +| ----- | --- | +| `SELECT * FROM trades` | Passthrough | +| `SELECT timestamp, symbol, price FROM trades` | Passthrough — column subset | +| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough — aliases are fine | +| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough — a row-local expression | +| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough — a filter only removes rows | +| `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view | +| `SELECT DISTINCT ...` | Rejected | +| `... LATEST ON timestamp PARTITION BY symbol` | Rejected | +| `JOIN`, `UNION` | Rejected | +| `row_number() OVER (...)` and other window functions | Rejected | +| `LIMIT` | Rejected | +| `ORDER BY` a non-timestamp column | Rejected — the view loses its designated timestamp | + +Everything in the rejected group produces output that depends on rows *other +than* the one being emitted. An incremental refresh sees only the newly-arrived +slice of the base table, so it cannot compute those correctly — a `LIMIT 100` +would admit 100 rows per refresh rather than 100 in total, and a `row_number()` +would restart within each slice. + +A query that is neither passthrough nor aggregating is rejected at creation +time. Most report `materialized view query requires a sampling interval, use +SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to +aggregate but named no bucket. `LIMIT` and window functions have their own +messages. + +To keep only some of a passthrough view's rows over time — the latest per key, +the top-N per group, or rows matching a predicate — attach an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. That page also +covers +[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + ## Creating a materialized view ### Basic syntax @@ -214,7 +303,9 @@ interval: ### The query -Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query. +An aggregating materialized view uses a `SAMPLE BY` or time-based `GROUP BY` +query. (The other shape is a [passthrough view](#passthrough-views), which does +not aggregate; the rules below apply to aggregating views.) **Supported:** @@ -526,7 +617,9 @@ rows. Materialized view queries: -- Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column +- Must either aggregate with `SAMPLE BY` / `GROUP BY` on a designated timestamp + column, or be a [passthrough](#passthrough-views) projection over a single + table - Must not use `FROM-TO`, `FILL`, or `ALIGN TO FIRST OBSERVATION` - Must not use non-deterministic functions (`now()`, `rnd_uuid4()`) - Must use join conditions compatible with incremental refresh diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index c4073b4e24..135e25e1fd 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -29,8 +29,10 @@ AS [ ( ] query [ ) ] Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` (except for the - passthrough views used by `EXPIRE ROWS` — see below) +- `query`: Either an aggregating query with `SAMPLE BY` or a time-based + `GROUP BY`, or a + [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection + over a single table - `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) ## Parameters @@ -42,7 +44,7 @@ Where: | `WITH BASE` | Specify base table (required for JOINs) | | `REFRESH` | Refresh strategy (default: `IMMEDIATE`) | | `DEFERRED` | Skip initial refresh on creation | -| `query` | A `SAMPLE BY` or time-based `GROUP BY` query | +| `query` | An aggregating (`SAMPLE BY` / time-based `GROUP BY`) or [passthrough](/docs/concepts/materialized-views/#passthrough-views) query | | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | @@ -53,7 +55,7 @@ Where: | Rule | Description | | ---- | ----------- | -| Query must aggregate | Requires `SAMPLE BY` or `GROUP BY` with designated timestamp | +| Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a 1:1 [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table | | Default refresh | `IMMEDIATE` (refreshes after each base table transaction) | | WITH BASE required | Must specify when query contains JOINs | | PARTITION BY sizing | Should be larger than or equal to `SAMPLE BY` interval | @@ -289,11 +291,12 @@ whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the late per key, the top-N per group, or rows matching a predicate — recomputed continuously as the view refreshes. -`EXPIRE ROWS` is designed for **passthrough (non-aggregating) views** — the -query is `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY`. An aggregating -view is accepted with a logged advisory (a later refresh can regenerate -reclaimed rows). The defining query must not read another policied view, as its -base or in a join: +`EXPIRE ROWS` is designed for +[**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) +— a projection over a single table, with no `SAMPLE BY` / `GROUP BY`, whose rows +stay 1:1 with the base. An aggregating view is accepted with a logged advisory +(a later refresh can regenerate reclaimed rows). The defining query must not +read another policied view, as its base or in a join: ```questdb-sql title="Passthrough view that keeps the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( From f3ef9cc142571a3526e53680e5866a956c086341 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 24 Aug 2026 18:07:05 +0200 Subject: [PATCH 06/14] Steer EXPIRE ROWS WHEN toward clock-based rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A predicate that looks only at the row's own values, such as EXPIRE ROWS WHEN amount < 1.5 selects the same surviving rows as writing the opposite test in the view's own query: SELECT * FROM trades WHERE amount >= 1.5 The `WHERE` version is cheaper everywhere. The rows it excludes are never copied into the view, so they cost no disk, no read-time filtering, and no rewriting of partitions by the cleanup job. The `EXPIRE ROWS` version writes those rows, hides them on every read, and reclaims them later. The docs presented the two as a roughly even choice and left the one thing only `EXPIRE ROWS WHEN` can do — a cutoff that moves with the clock — as a footnote at the end of the comparison. Someone reading quickly could reasonably conclude that any filter is fine to write as a policy. The comparison section now opens with the rule instead: use `EXPIRE ROWS WHEN` for predicates that involve wall-clock time, and put a predicate that depends only on the row's values in the `WHERE` clause. A rolling window genuinely has no `WHERE` equivalent, because a view's defining query rejects non-deterministic functions and will not accept `now()`. The argument for keeping a fixed, hand-advanced cutoff in a policy is still true and still here, moved into its own subsection so it reads as the exception rather than an equal option. The section keeps its `#where-filter-or-expire-rows` anchor, which `materialized-views.md` links to. Two other notes were added while rewriting. The two forms are not exact opposites where `NULL`s are concerned: `WHERE` keeps a row only when the test is `TRUE`, while `EXPIRE ROWS WHEN` expires a row only when the test is `TRUE`, so a `NULL` amount is dropped by the first and kept by the second. And the worked example that expires small trades now says up front that it is written that way to make the kept rows easy to read off the sample data, not because that is where such a filter belongs. The same guidance was added to the two SQL reference pages, so a reader who lands on one of them without reading the concept page still sees it. The `ALTER MATERIALIZED VIEW SET EXPIRE` page previously opened its examples with `WHEN amount < 1.5`, which demonstrated exactly the shape being discouraged; that example is now a rolling seven-day window. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 93 ++++++++++++------- documentation/concepts/materialized-views.md | 7 +- .../query/sql/alter-mat-view-set-expire.md | 10 +- documentation/query/sql/create-mat-view.md | 7 ++ 4 files changed, 77 insertions(+), 40 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index 5a0cdbbee4..f2c3a1ab0d 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -45,10 +45,16 @@ continuously-maintained, pruned copy of a base table: symbol, or session (`KEEP LATEST`). - **Per-group extremes or leaderboards** — keep the highest/lowest value per group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). -- **Rolling row-level windows** — keep rows newer than a cutoff, or matching any - predicate, at finer granularity than TTL's whole-partition drops +- **Rolling row-level windows** — keep rows newer than a moving cutoff such as + `now() - 7d`, at finer granularity than TTL's whole-partition drops (`WHEN predicate`). +The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A +deterministic predicate — one that depends only on the row's own values — +selects the same rows more cheaply as a `WHERE` clause in the view's defining +query, which never copies the excluded rows into the view at all. See +[`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). + Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based retention on a base table is enough — it is cheaper and has no passthrough-view requirement. @@ -83,13 +89,22 @@ derive from — otherwise those views would copy expired rows on refresh. ## `WHERE` filter or `EXPIRE ROWS`? A passthrough view can exclude rows in two places: a `WHERE` clause in its -defining query, or an `EXPIRE ROWS WHEN` predicate. For a deterministic, -row-local predicate the two describe the same set of surviving rows, so the -choice is not about what the view contains — it is about whether the rule is -part of the view's *definition* or a *knob you expect to turn*. - -**Default to `WHERE`.** A row a `WHERE` clause excludes is never written, so it -costs nothing at any stage: +defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the +clock. + +**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling +window cannot be written as a `WHERE` clause at all: a view's defining query +rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is +not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported +way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +every read, so the window rolls forward on its own and the cleanup job reclaims +the disk behind it. This is what the `WHEN` form is for. + +**Put a deterministic predicate in the `WHERE` clause.** A predicate that +depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — +describes the same surviving rows either way, so the two are near-equivalent in +what the view contains, and `WHERE` is the cheaper of the two at every stage. A +row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | @@ -103,7 +118,25 @@ That last row is a hard constraint rather than a preference. If any other view will read this one, the policy is not available and the predicate has to go in the `WHERE` clause. -**Reach for `EXPIRE ROWS` when the cutoff is a knob.** There is no +The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a +row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and +kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). + +The two compose, and on a passthrough view that combination is usually the right +shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy +fixes how long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + +### When a deterministic cutoff still belongs in a policy + +One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed +threshold you expect to advance by hand. There is no `ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means dropping the view and re-creating it, which re-materializes it from the base. Changing a policy is a metadata operation: @@ -120,26 +153,10 @@ base that no longer holds everything the view held. A view whose retention horizon is longer than its base table's cannot afford to be rebuilt, so its cutoff belongs in a policy. -Two things only `EXPIRE ROWS` can express at all, whichever way you lean: - -- **Rules that move with the clock.** A view's defining query rejects - non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not - accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported way - to write a rolling window. -- **Rules that compare rows against each other.** `KEEP LATEST`, - `KEEP N HIGHEST/LOWEST` and window predicates have no `WHERE` equivalent — a - `LATEST ON` or a window function in the defining query makes the view - non-passthrough. - -In practice the two compose, and on a passthrough view that is usually the right -shape: the `WHERE` clause fixes what the view is about, and the policy fixes how -long it keeps what it has. - -```questdb-sql title="A filter for the subject, a policy for the horizon" -CREATE MATERIALIZED VIEW trades_btc_recent AS ( - SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; -``` +A rule that compares rows against each other has no `WHERE` equivalent either: +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed +in the defining query, because a `LATEST ON` or a window function there makes +the view non-passthrough. ## The modes @@ -223,8 +240,12 @@ this small dataset that is effectively instant. ### Per-row predicate: `WHEN` -A per-row predicate expires a row when it evaluates `TRUE`. Here, expire small -trades (`amount < 1.5`): +A per-row predicate expires a row when it evaluates `TRUE`. The example below +uses a deterministic predicate — expire small trades (`amount < 1.5`) — +because it makes the keep-set easy to read off the sample data. In production +that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which +keeps those rows out of the view entirely; the rolling window further down is +the case `WHEN` exists for. ```questdb-sql title="Expire rows where amount < 1.5" CREATE MATERIALIZED VIEW trades_sized AS ( @@ -244,8 +265,10 @@ The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, not `TRUE` — see [NULLs](#nulls)). -A predicate on the designated timestamp gives a **rolling retention window**, -re-evaluated on every read so the visible set rolls forward with the clock: +A predicate on the designated timestamp gives a **rolling retention window** — +the main use for `WHEN`. It is re-evaluated on every read, so the visible set +rolls forward with the clock, and no `WHERE` clause can express it because the +defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -578,7 +601,7 @@ rather than breaking subsequent reads. - [Materialized views](/docs/concepts/materialized-views/) — the view type `EXPIRE ROWS` runs on - [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — - the non-aggregating view shape `EXPIRE ROWS` is designed for + the non-aggregating views `EXPIRE ROWS` applies to - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create syntax, including the `EXPIRE ROWS` clause - [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index f8f3e7eea8..973e2401a9 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -185,9 +185,10 @@ summary of it: - **A narrowed replica** — one symbol, one tenant, one region, or a handful of columns out of a wide table, kept current automatically and queried without the base table's scan cost. -- **A row-level retention target** — a passthrough view is the shape - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) is built for. Its rows - *are* base rows, so reclaiming one is permanent. +- **A row-level retention target** — attach an + [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only + some of the view's rows: the latest per key, the top-N per group, or rows + inside a rolling time window. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index c02337bdb9..66eec973a1 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -55,11 +55,17 @@ reclamation follows in the background. See ## Examples -```questdb-sql title="Per-row predicate, with a tighter cleanup cadence" +```questdb-sql title="Rolling 7-day window, with a tighter cleanup cadence" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS WHEN amount < 1.5 CLEANUP EVERY 30m; + SET EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 30m; ``` +A `WHEN` predicate is the right tool for a cutoff that moves with the clock like +this one. A deterministic predicate such as `amount < 1.5` is accepted too, but +it selects the same rows more cheaply as a `WHERE` clause in the view's defining +query — see +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + ```questdb-sql title="Keep the latest row per symbol" ALTER MATERIALIZED VIEW trades_mirror SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 135e25e1fd..68f9f2e624 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -304,6 +304,13 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; ``` +A `WHEN` predicate is for rules that move with **wall-clock time**, such as a +rolling `ts < dateadd('d', -7, now())` window — the defining query cannot +express those, because it rejects non-deterministic functions. A predicate that +depends only on the row's own values belongs in the query's `WHERE` clause +instead, which keeps those rows out of the view entirely; see +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). + The clause goes after the query (and after `PARTITION BY` if present): ``` From c83a79ccbc7f58d7558f582c5df298d76a131d33 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 25 Aug 2026 10:39:03 +0200 Subject: [PATCH 07/14] Add use-case examples for passthrough views The passthrough views section gave two reasons to reach for one, a narrowed replica and a target for EXPIRE ROWS, as bullet points with no SQL under them. A reader who wanted either had nothing to copy. Add a "Use cases" section with three worked examples, each opening with a one-sentence scenario and naming the domain it comes from, so the section covers sensor telemetry as well as capital markets. The examples are a current-reading-per-sensor view using KEEP LATEST, a live options chain using a WHEN predicate, and the largest trades per symbol using KEEP 10 HIGHEST. Every statement was run against a live server before being written down. Two behaviors are recorded here that were not documented anywhere else. First, the cleanup job reclaims disk only when the cutoff names the view's designated timestamp. The options example compares a contract's expiry column against now(), and materialized_views() reports options_live expiry < now() FILTER_ONLY options_ctl ts < dateadd('d', -7, now()) FILTER_AND_RECLAIM for two views over the same base table, so the column is what decides it, not the shape of the predicate. Expired contracts stop showing up in queries straight away either way; only the disk behavior differs. Second, when two rows tie at the Nth place under KEEP N, the newer row survives. An order book example was written first and then dropped. KEEP N HIGHEST ranks over everything the view holds rather than over a recent window, so a view of the ten highest bids per symbol fills up with the highest prices ever seen. With one bid from the previous day at 200.00 and a live book around 100, the stale bid outranked every current one and the live bids were expired out of the view. An order book is state that gets superseded, and a materialized view can only add rows, so no policy can remove an order that was later cancelled. The trades example replaces it because a trade is never retracted, which makes ranking over all history the intended meaning rather than a defect. The closing paragraph of that example now points at KEEP LATEST for values that do get superseded, to save the next reader the same detour. Also drop the list of policy shapes from the second bullet, since the examples below it now demonstrate each one. Co-Authored-By: Claude Opus 5 --- documentation/concepts/materialized-views.md | 97 +++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 973e2401a9..637f3cbe5b 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -187,14 +187,107 @@ summary of it: the base table's scan cost. - **A row-level retention target** — attach an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only - some of the view's rows: the latest per key, the top-N per group, or rows - inside a rolling time window. The base table is left alone. + some of the view's rows. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and `REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets for a period to align to. +### Use cases + +On its own, a passthrough view is a maintained copy of the base table. Add an +[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy and it becomes a +maintained *subset*: you describe which rows are worth keeping, and QuestDB +keeps that set current as new data arrives. The three examples below are drawn +from sensor telemetry and from capital markets, and each keeps a different kind +of subset. + +#### IoT: The current reading from every sensor + +A building management platform records temperature and humidity from tens of +thousands of sensors, and its operations screen shows the newest reading from +each one. + +Sensors report at their own pace. Some send a reading every second, others go +quiet for days. Against the base table that screen runs +`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it has found +a row for even the quietest sensor, and over a long history that is most of the +table. + +A view that keeps only the newest row per sensor answers the same question from +a handful of rows: + +```questdb-sql title="Latest reading per sensor" +CREATE MATERIALIZED VIEW sensor_current AS ( + SELECT * FROM sensor_readings +) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; +``` + +`sensor_current` holds one row per `sensor_id` and moves forward on its own as +readings arrive. Superseded rows stop appearing in queries but stay on disk, so +the view keeps growing at the same rate as the base table. Give it a +[TTL](/docs/concepts/ttl/) to cap its size. + +#### Finance: Options that have not expired yet + +A market maker quotes an options chain where contracts expire every Friday, and +the pricing screen must never show a contract that has already expired. + +That rule cannot live in the view's query. The query runs when rows are written +into the view, and it may not call `now()`, so there is no way to say "expiry is +still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` predicate is +evaluated on every read, which is what this case needs: + +```questdb-sql title="Options that have not expired yet" +CREATE MATERIALIZED VIEW options_live AS ( + SELECT * FROM options_quotes +) EXPIRE ROWS WHEN expiry < now(); +``` + +Contracts leave `options_live` as their expiry passes. There is no job to +schedule and nothing to re-create. + +One caveat, about disk rather than about results. QuestDB deletes expired rows +only when it can tell that a row, once expired, can never qualify again. It can +tell that for a cutoff on the view's designated timestamp, which only moves +forward. `expiry` is a different column, so QuestDB takes the safe route: +expired contracts stop appearing in queries straight away, but their rows stay +on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see +[when expired rows are deleted from disk](/docs/concepts/deep-dive/expire-rows/#monotonicity-and-cleanup-safety). + +#### Finance: The largest trades per symbol + +A surveillance desk watches for block trades and wants the ten biggest prints +for every instrument on hand at all times. + +A trade is a fact that does not change once it has happened, so "the ten biggest +so far" is a set that only gets refined as larger trades arrive. That is what a +top-N policy maintains: + +```questdb-sql title="Ten largest trades per symbol" +CREATE MATERIALIZED VIEW trades_largest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; +``` + +`trades_largest` holds ten rows per symbol however far the base table grows, and +the desk reads it directly instead of ranking the base table on every query. +When an eleventh large trade arrives, the smallest of the ten drops out. Trades +tied on `amount` at the tenth place are separated by the designated timestamp, +with the newer one staying. + +Ten rows per symbol are visible, but the view still stores every base row it has +taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) +is what bounds that, at the price of changing the question the view answers from +"the biggest so far" to "the biggest still retained". See +[combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). + +The ranking covers everything the view holds rather than a recent window, so +`KEEP N` fits records that stay true once written. For a value that gets +superseded later, such as a resting order that is then cancelled, `KEEP LATEST` +is the mode that tracks the current version. + ### What a passthrough view inherits A passthrough view takes its shape from the base table instead of from a From c578889f7c89bf6f7991cbd555da78b5b407fd7c Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Tue, 25 Aug 2026 11:01:33 +0200 Subject: [PATCH 08/14] Document the NULL EXPIRE ROWS threshold rejection CREATE and ALTER now refuse a scalar WHEN threshold that is constant at definition time and evaluates to NULL, since such a policy expires nothing. Add a Semantics section covering the explicit CAST(NULL AS TIMESTAMP) and the arithmetic spellings that overflow onto the reserved NULL value, and note that a clock-based threshold is evaluated per read and so cannot be checked in advance. Add the error to the SET EXPIRE error table, and a pointer to the new section from the CREATE MATERIALIZED VIEW reference. Also steer the rolling-window example toward dateadd. It and now() - 86400000000 retain the same rows, but only the dateadd form reduces to a timestamp range the scan can use to skip partitions; the arithmetic form is evaluated row by row across the whole view on every read. Both reclaim disk, so the difference is read cost alone. Co-Authored-By: Claude Opus 5 --- .../concepts/deep-dive/expire-rows.md | 42 +++++++++++++++++++ .../query/sql/alter-mat-view-set-expire.md | 1 + documentation/query/sql/create-mat-view.md | 6 +++ 3 files changed, 49 insertions(+) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index f2c3a1ab0d..cffc08b87e 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -279,6 +279,13 @@ CREATE MATERIALIZED VIEW trades_recent AS ( (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) +Write the offset with `dateadd`, as above, rather than as clock arithmetic. +`WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain +the same rows, but only the `dateadd` form reduces to a timestamp range that the +scan can use to skip partitions. The arithmetic form is evaluated row by row +across the whole view on every read. Both forms reclaim disk — the cleanup job +proves either monotonic — so the difference is read cost alone. + ### Keep latest per key: `KEEP LATEST` Keep only the most recent row per key — turning the passthrough view into a @@ -440,6 +447,41 @@ Therefore: `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of type. +### A `NULL` threshold is rejected + +A `WHEN` threshold that evaluates to a constant `NULL` expires nothing — +`ts < NULL` is never `TRUE` — so the policy would be inert. QuestDB refuses it at +`CREATE` and `ALTER` time rather than storing a view that silently never +reclaims: + +```questdb-sql title="Rejected: the threshold is NULL" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < CAST(NULL AS TIMESTAMP); +-- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire +``` + +The check matters most where the `NULL` is not written down. QuestDB stores a +`NULL` `TIMESTAMP`, `LONG` or `INT` as a reserved value at the bottom of the +type's range, and integer arithmetic wraps silently when it overflows, so an +arithmetic threshold can land on that value: + +```questdb-sql title="Also rejected: arithmetic that overflows onto NULL" +-- LONG overflow +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < 4611686018427387904 * 2; + +-- INT overflow, reached three orders of magnitude sooner +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN ts < 2147483647 + 1; +``` + +Only thresholds that are constant at definition time are checked this way. One +built from a clock, such as `ts < now() - 3600000000`, is evaluated per read and +cannot be checked in advance. + ### Ties and determinism `KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 66eec973a1..503f7754e7 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -113,6 +113,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | +| `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | | `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | ## See also diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 68f9f2e624..a5bc445b7a 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -321,6 +321,12 @@ EXPIRE ROWS [ CLEANUP EVERY duration ] ``` +A `WHEN` threshold that is constant at definition time and evaluates to `NULL` +is rejected, since it would expire nothing. That covers the explicit +`ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved +`NULL` value, such as `ts < 2147483647 + 1` — see +[A `NULL` threshold is rejected](/docs/concepts/deep-dive/expire-rows/#a-null-threshold-is-rejected). + Expired rows are hidden from queries immediately in every mode. They are reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and From 8795cbee2e562dcdb6bc6e6b3a459cc5fc9c1509 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 10:27:28 +0200 Subject: [PATCH 09/14] docs: correct expire rows behavior --- .../concepts/deep-dive/expire-rows.md | 48 +++++++++++-------- .../query/sql/alter-mat-view-set-expire.md | 5 ++ 2 files changed, 34 insertions(+), 19 deletions(-) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/deep-dive/expire-rows.md index cffc08b87e..b95c8539af 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/deep-dive/expire-rows.md @@ -174,6 +174,12 @@ only when the rule selects it for removal. `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. +The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, +`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has +a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it +accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. + The **Frees disk** column is the difference between hiding a row and deleting it. Every mode hides expired rows from every read, immediately. Only a monotonic `WHEN` predicate also has those rows deleted from disk by the cleanup job: a @@ -252,14 +258,14 @@ CREATE MATERIALIZED VIEW trades_sized AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN amount < 1.5; -SELECT * FROM trades_sized; +SELECT * FROM trades_sized ORDER BY ts; ``` | symbol | price | amount | ts | | ------ | ----- | ------ | --------------------------- | +| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | | BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, @@ -279,12 +285,8 @@ CREATE MATERIALIZED VIEW trades_recent AS ( (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) -Write the offset with `dateadd`, as above, rather than as clock arithmetic. `WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows, but only the `dateadd` form reduces to a timestamp range that the -scan can use to skip partitions. The arithmetic form is evaluated row by row -across the whole view on every read. Both forms reclaim disk — the cleanup job -proves either monotonic — so the difference is read cost alone. +the same rows and both reclaim disk — the cleanup job proves either monotonic. ### Keep latest per key: `KEEP LATEST` @@ -296,13 +298,13 @@ CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; -SELECT * FROM trades_latest; +SELECT * FROM trades_latest ORDER BY ts; ``` | symbol | price | amount | ts | | ------ | ----- | ------ | --------------------------- | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | | ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | One row per symbol — the latest by the designated timestamp `ts`. As new trades arrive, the kept row advances automatically. `PARTITION BY` may list multiple key @@ -397,11 +399,19 @@ time-based predicate) reappears on the next read. ### Physical cleanup (best-effort) -A background job reclaims disk for non-active partitions: a fully-expired -partition is removed, and a partially-expired one is compacted down to its -survivors. It runs at the `CLEANUP EVERY` cadence (default `1h`) and is -**best-effort** — the read filter is authoritative, so deferred or skipped -reclamation only affects disk usage, never query results. +A background job reclaims disk for non-active partitions. A fully-expired +partition is removed. Under a rolling clock-based predicate, a partially-expired +partition is compacted down to its survivors only when the expired-row fraction +reaches `cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults +to `0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff +moves through it. Set the property to `0` to compact on the first expired row, +or to `1` to disable partial-partition compaction; fully-expired partitions are +still removed. The threshold does not delay a fixed, deterministic predicate, +whose expired-row verdicts cannot change with time. + +The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** +— the read filter is authoritative, so deferred or skipped reclamation only +affects disk usage, never query results. The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh @@ -413,11 +423,11 @@ On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with -`cairo.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay filtered; -only reclamation stops — the setting does not disable `EXPIRE ROWS` itself); the -setting is read at startup, so changing it requires a restart. A failing sweep -retries after one second, doubling the per-view retry gap up to a 10-minute -cap. +`cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay +filtered; only reclamation stops — the setting does not disable `EXPIRE ROWS` +itself). Cleanup settings are read at startup, so changing this property or the +minimum expired fraction requires a restart. A failing sweep retries after one +second, doubling the per-view retry gap up to a 10-minute cap. To observe reclamation, compare the physical row count per partition before and after a sweep: diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 503f7754e7..2a775ce2a8 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -34,6 +34,10 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE | `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | | `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | +Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, +`DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with +`ORDER BY` and accepts any orderable column type. + For the full description of each mode and its semantics, see the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. @@ -111,6 +115,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | | `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | From c66ce5e088626b5d785e80999ef7be357baf677f Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 10:56:35 +0200 Subject: [PATCH 10/14] docs: address EXPIRE ROWS review feedback --- .../concepts/{deep-dive => }/expire-rows.md | 265 +++++++++--------- documentation/concepts/materialized-views.md | 12 +- documentation/concepts/ttl.md | 2 + documentation/query/functions/meta.md | 43 +-- .../query/sql/alter-mat-view-set-expire.md | 19 +- documentation/query/sql/create-mat-view.md | 16 +- documentation/sidebars.js | 2 +- 7 files changed, 179 insertions(+), 180 deletions(-) rename documentation/concepts/{deep-dive => }/expire-rows.md (94%) diff --git a/documentation/concepts/deep-dive/expire-rows.md b/documentation/concepts/expire-rows.md similarity index 94% rename from documentation/concepts/deep-dive/expire-rows.md rename to documentation/concepts/expire-rows.md index b95c8539af..1b13e0b159 100644 --- a/documentation/concepts/deep-dive/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -1,6 +1,6 @@ --- title: Expiring rows (EXPIRE ROWS) -sidebar_label: Expiring rows +sidebar_label: EXPIRE ROWS description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a @@ -22,14 +22,61 @@ on-disk storage is reclaimed afterwards by a background job under a monotonic [The modes](#the-modes) and [Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). +## The modes + +Every mode keeps a defined set of rows and expires the rest. A row is expired +only when the rule selects it for removal. + +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a +window predicate, so the window `WHEN` is the general escape hatch. + +The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, +`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has +a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it +accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. + +For how read filtering and physical reclamation differ between modes, see +[How it works](#how-it-works). You can inspect the behavior selected for a view +through `materialized_views().expire_enforcement`; see +[Inspecting a policy](#inspecting-a-policy). + +The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if present), or set later with +[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +| Element | Meaning | +| ------------------ | ---------------------------------------------------------------------------------------- | +| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | + :::note `EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT * FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a -logged advisory**: reads stay filtered, but a later refresh can regenerate -reclaimed rows from base rows that still exist (see +logged advisory** because a later refresh can regenerate reclaimed rows from +base rows that still exist (see [Requirements](#requirements)). For base-table retention use [TTL](/docs/concepts/ttl/) or, on Enterprise, [storage policies](/docs/concepts/storage-policy/). @@ -76,141 +123,15 @@ is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that replica down to the rows you want to keep — without touching the base table. A **non-passthrough (aggregating) view is accepted with a logged advisory** -rather than rejected: the read filter still hides expired rows, but physical -reclamation only sticks when base-table retention is aligned with the expiry -horizon — a later incremental or full refresh can regenerate a reclaimed row -from base rows that still exist. +rather than rejected: physical reclamation only sticks when base-table retention +is aligned with the expiry horizon — a later incremental or full refresh can +regenerate a reclaimed row from base rows that still exist. A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views derive from — otherwise those views would copy expired rows on refresh. -## `WHERE` filter or `EXPIRE ROWS`? - -A passthrough view can exclude rows in two places: a `WHERE` clause in its -defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the -clock. - -**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling -window cannot be written as a `WHERE` clause at all: a view's defining query -rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is -not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days" — the read filter re-evaluates `now()` on -every read, so the window rolls forward on its own and the cleanup job reclaims -the disk behind it. This is what the `WHEN` form is for. - -**Put a deterministic predicate in the `WHERE` clause.** A predicate that -depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — -describes the same surviving rows either way, so the two are near-equivalent in -what the view contains, and `WHERE` is the cheaper of the two at every stage. A -row the `WHERE` clause excludes is never copied into the view: - -| | `WHERE` in the query | `EXPIRE ROWS WHEN` | -| --- | --- | --- | -| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | -| Read cost | None | The keep-set filter is applied on every read of the view | -| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | -| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | -| Can be another view's base | Yes | No — a policied view is rejected as a base | - -That last row is a hard constraint rather than a preference. If any other view -will read this one, the policy is not available and the predicate has to go in -the `WHERE` clause. - -The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a -row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row -only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and -kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). - -The two compose, and on a passthrough view that combination is usually the right -shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy -fixes how long it keeps what it has. - -```questdb-sql title="A filter for the subject, a policy for the horizon" -CREATE MATERIALIZED VIEW trades_btc_recent AS ( - SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; -``` - -### When a deterministic cutoff still belongs in a policy - -One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed -threshold you expect to advance by hand. There is no -`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means -dropping the view and re-creating it, which re-materializes it from the base. -Changing a policy is a metadata operation: - -```questdb-sql title="Retuning a retention horizon without a rebuild" -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; -``` - -The rebuild a `WHERE` change forces is not only slow, it can lose data: if the -base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a -base that no longer holds everything the view held. A view whose retention -horizon is longer than its base table's cannot afford to be rebuilt, so its -cutoff belongs in a policy. - -A rule that compares rows against each other has no `WHERE` equivalent either: -`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed -in the defining query, because a `LATEST ON` or a window function there makes -the view non-passthrough. - -## The modes - -Every mode keeps a defined set of rows and expires the rest. A row is expired -only when the rule selects it for removal. - -| Mode | What it keeps | Syntax | Frees disk | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | - -`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a -window predicate, so the window `WHEN` is the general escape hatch. - -The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, -`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has -a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it -accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not -`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. - -The **Frees disk** column is the difference between hiding a row and deleting -it. Every mode hides expired rows from every read, immediately. Only a monotonic -`WHEN` predicate also has those rows deleted from disk by the cleanup job: a -mode whose keep-set depends on the other rows in the view keeps its expired rows -on disk until a full refresh rebuilds the view. A `KEEP LATEST` view therefore -holds a full copy of its base table unless the view's own -[TTL](/docs/concepts/ttl/) bounds it — see -[Combining with TTL](#combining-with-ttl). -`materialized_views().expire_enforcement` reports which of the two a given view -gets; see [Inspecting a policy](#inspecting-a-policy). - -The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the -query, and after `PARTITION BY` if present), or set later with -[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): - -``` -EXPIRE ROWS - { WHEN predicate - | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] - | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } - [ CLEANUP EVERY duration ] -``` - -| Element | Meaning | -| ------------------ | ---------------------------------------------------------------------------------------- | -| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | -| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | -| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | -| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | -| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | - ## Worked examples The following walks through every mode on a small fixed dataset so you can see @@ -374,6 +295,78 @@ express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window (`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). +## `WHERE` filter or `EXPIRE ROWS`? + +A passthrough view can exclude rows in two places: a `WHERE` clause in its +defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the +clock. + +**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling +window cannot be written as a `WHERE` clause at all: a view's defining query +rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is +not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported +way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +every read, so the window rolls forward on its own and the cleanup job reclaims +the disk behind it. This is what the `WHEN` form is for. + +**Put a deterministic predicate in the `WHERE` clause.** A predicate that +depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — +describes the same surviving rows either way, so the two are near-equivalent in +what the view contains, and `WHERE` is the cheaper of the two at every stage. A +row the `WHERE` clause excludes is never copied into the view: + +| | `WHERE` in the query | `EXPIRE ROWS WHEN` | +| --- | --- | --- | +| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Read cost | None | The keep-set filter is applied on every read of the view | +| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Can be another view's base | Yes | No — a policied view is rejected as a base | + +That last row is a hard constraint rather than a preference. If any other view +will read this one, the policy is not available and the predicate has to go in +the `WHERE` clause. + +The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a +row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and +kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). + +The two compose, and on a passthrough view that combination is usually the right +shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy +fixes how long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + +### When a deterministic cutoff still belongs in a policy + +One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed +threshold you expect to advance by hand. There is no +`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means +dropping the view and re-creating it, which re-materializes it from the base. +Changing a policy is a metadata operation: + +```questdb-sql title="Retuning a retention horizon without a rebuild" +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; +``` + +The rebuild a `WHERE` change forces is not only slow, it can lose data: if the +base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a +base that no longer holds everything the view held. A view whose retention +horizon is longer than its base table's cannot afford to be rebuilt, so its +cutoff belongs in a policy. + +A rule that compares rows against each other has no `WHERE` equivalent either: +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed +in the defining query, because a `LATEST ON` or a window function there makes +the view non-passthrough. + ## How it works `EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a @@ -632,11 +625,9 @@ rather than breaking subsequent reads. - **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but the view's designated timestamp is always used; naming a different column is rejected. -- **Only a monotonic `WHEN` predicate frees disk.** The relative and window - modes hide expired rows on every read but never delete them, and neither does - a `WHEN` predicate that cannot be proven monotonic — see - [monotonicity](#monotonicity-and-cleanup-safety) above. Check a view's verdict - with `materialized_views().expire_enforcement`. +- **Cleanup eligibility.** See + [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check + a view's verdict with `materialized_views().expire_enforcement`. - **Reserved column name.** The window/keep modes compute the keep-set through a synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view that exposes a column with that name. diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 637f3cbe5b..aaa8fcc523 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -186,7 +186,7 @@ summary of it: columns out of a wide table, kept current automatically and queried without the base table's scan cost. - **A row-level retention target** — attach an - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy to keep only + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only some of the view's rows. The base table is left alone. Passthrough views refresh incrementally like any other materialized view, and @@ -197,7 +197,7 @@ for a period to align to. ### Use cases On its own, a passthrough view is a maintained copy of the base table. Add an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy and it becomes a +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a maintained *subset*: you describe which rows are worth keeping, and QuestDB keeps that set current as new data arrives. The three examples below are drawn from sensor telemetry and from capital markets, and each keeps a different kind @@ -254,7 +254,7 @@ tell that for a cutoff on the view's designated timestamp, which only moves forward. `expiry` is a different column, so QuestDB takes the safe route: expired contracts stop appearing in queries straight away, but their rows stay on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see -[when expired rows are deleted from disk](/docs/concepts/deep-dive/expire-rows/#monotonicity-and-cleanup-safety). +[when expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety). #### Finance: The largest trades per symbol @@ -281,7 +281,7 @@ Ten rows per symbol are visible, but the view still stores every base row it has taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) is what bounds that, at the price of changing the question the view answers from "the biggest so far" to "the biggest still retained". See -[combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). +[combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). The ranking covers everything the view holds rather than a recent window, so `KEEP N` fits records that stay true once written. For a value that gets @@ -336,9 +336,9 @@ messages. To keep only some of a passthrough view's rows over time — the latest per key, the top-N per group, or rows matching a predicate — attach an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. That page also +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also covers -[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ## Creating a materialized view diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index 9c1ee9f40f..4a9040b7f2 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -89,6 +89,8 @@ A view's TTL is independent of its base table's TTL. For full syntax, see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#ttl-time-to-live) and [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/). +For row-level retention on passthrough materialized views, such as keeping the +latest row per key, see [`EXPIRE ROWS`](/docs/concepts/expire-rows/). ## How TTL works diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 945fbfce75..b489cabf0d 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -288,7 +288,7 @@ Returns a `table` including the following information: merges two adjacent intervals instead of paying for a second commit. `0` means merging is disabled - `expire_clause` - the view's - [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy as written, or + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or `NULL` when the view has no policy - `expire_cleanup_every` - how often the cleanup job runs for the policy, or `NULL` @@ -301,7 +301,7 @@ Returns a `table` including the following information: `materialized_views()` on its own returns every column listed above. The example below projects a readable subset over three views: an aggregating one with no retention policy, and two passthrough views with an -[`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) policy. +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. ```questdb-sql title="List all materialized views" SELECT view_name, view_status, base_table_name, refresh_base_table_txn, @@ -790,20 +790,20 @@ Returns a `table` with the following columns: ### Basic table information -| Column | Type | Description | -| --------------------- | ------- | ------------------------------------------------------------------ | -| `id` | INT | Internal table ID | -| `table_name` | STRING | Table name | -| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | -| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | -| `dedup` | BOOLEAN | Whether deduplication is enabled | -| `ttlValue` | INT | TTL (Time-To-Live) value | -| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `matView` | BOOLEAN | Whether this is a materialized view | -| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | -| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | -| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | +| Column | Type | Description | +| ---------------------- | ------- | ------------------------------------------------------------------ | +| `id` | INT | Internal table ID | +| `table_name` | STRING | Table name | +| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | +| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | +| `dedup` | BOOLEAN | Whether deduplication is enabled | +| `ttlValue` | INT | TTL (Time-To-Live) value | +| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `matView` | BOOLEAN | Whether this is a materialized view | +| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | +| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | +| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | :::note @@ -873,6 +873,17 @@ These columns are populated on **replicas only** via replication download tracki On primary instances, these columns will be `0` or `false`. +### Row-expiry policy + +| Column | Type | Description | +| ---------------------- | ------ | -------------------------------------------------------------- | +| `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy as written, or `null` | +| `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | + +Unlike `materialized_views()`, `tables()` does not expose +`expire_enforcement`; use `materialized_views()` to inspect whether a policy is +`FILTER_AND_RECLAIM` or `FILTER_ONLY`. + ### Data precision and limitations These values are approximations, not precise real-time metrics: diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 2a775ce2a8..3303c43bd5 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -6,11 +6,10 @@ description: documentation. --- -Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/deep-dive/expire-rows/) +Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) row-retention policy on a materialized view (designed for **passthrough** -views — see the concept page). Expired rows are hidden from queries immediately -in every mode, and reclaimed on disk in the background under a monotonic `WHEN` -predicate — the relative and window modes hide rows without freeing disk. +views — see the concept page). For filtering and disk-reclamation behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ## Syntax @@ -39,7 +38,7 @@ Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `ORDER BY` and accepts any orderable column type. For the full description of each mode and its semantics, see the -[Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page. +[Expiring rows](/docs/concepts/expire-rows/) concept page. ## When to use @@ -53,9 +52,8 @@ For the full description of each mode and its semantics, see the `SET EXPIRE ROWS` validates the new policy against the view's columns first (compiling the predicate / checking the key columns), so an invalid predicate or an unknown column is rejected immediately rather than breaking later reads. Once -set, queries against the view are filtered to the kept rows immediately; physical -reclamation follows in the background. See -[How it works](/docs/concepts/deep-dive/expire-rows/#how-it-works). +set, the policy takes effect without rebuilding the view. See +[How it works](/docs/concepts/expire-rows/#how-it-works). ## Examples @@ -68,7 +66,7 @@ A `WHEN` predicate is the right tool for a cutoff that moves with the clock like this one. A deterministic predicate such as `amount < 1.5` is accepted too, but it selects the same rows more cheaply as a `WHERE` clause in the view's defining query — see -[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ```questdb-sql title="Keep the latest row per symbol" ALTER MATERIALIZED VIEW trades_mirror @@ -96,7 +94,6 @@ ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; | Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | | No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | | Validation | The policy is checked against the view's columns before it is applied | -| Immediate effect | Reads are filtered to the kept rows as soon as the policy is set | | Replication | The policy and the reclamation it drives replicate as normal WAL traffic | ## Permissions (Enterprise) @@ -123,7 +120,7 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; ## See also -- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/deep-dive/expire-rows/) +- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/expire-rows/) - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) - [Materialized views concept](/docs/concepts/materialized-views/) - [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index a5bc445b7a..75bdb8d5f1 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -309,7 +309,7 @@ rolling `ts < dateadd('d', -7, now())` window — the defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see -[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/deep-dive/expire-rows/#where-filter-or-expire-rows). +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). The clause goes after the query (and after `PARTITION BY` if present): @@ -325,21 +325,19 @@ A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit `ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved `NULL` value, such as `ts < 2147483647 + 1` — see -[A `NULL` threshold is rejected](/docs/concepts/deep-dive/expire-rows/#a-null-threshold-is-rejected). +[A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). -Expired rows are hidden from queries immediately in every mode. They are -reclaimed on disk in the background (`CLEANUP EVERY`, default `1h`) under a -monotonic `WHEN` predicate; `KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N` and -window predicates hide rows without freeing disk. Change or remove a policy -with +For filtering and disk-reclamation behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). Change or +remove a policy with [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the statement and first in effect: it removes rows from the view, and the `EXPIRE ROWS` policy then applies to the rows that stay — see -[Combining with TTL](/docs/concepts/deep-dive/expire-rows/#combining-with-ttl). +[Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). -See the [Expiring rows](/docs/concepts/deep-dive/expire-rows/) concept page for +See the [Expiring rows](/docs/concepts/expire-rows/) concept page for all modes, worked examples, and semantics (NULLs, ties, monotonicity). ## Complete example diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 6cc84b3cc3..23ae94729c 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -627,6 +627,7 @@ module.exports = { "concepts/deduplication", "concepts/delivery-semantics", "concepts/ttl", + "concepts/expire-rows", "concepts/storage-policy", { id: "concepts/cold-storage", @@ -643,7 +644,6 @@ module.exports = { items: [ "concepts/deep-dive/indexes", "concepts/deep-dive/posting-index", - "concepts/deep-dive/expire-rows", "concepts/deep-dive/interval-scan", "concepts/deep-dive/jit-compiler", "concepts/deep-dive/query-tracing", From 6b86c2ab2b29928cbfe19812a9c2023620937bf8 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 12:11:06 +0200 Subject: [PATCH 11/14] docs: address expire rows review comments --- documentation/changelog.mdx | 1 + documentation/concepts/expire-rows.md | 162 +++++++++--------- documentation/concepts/materialized-views.md | 20 +-- .../query/sql/alter-mat-view-set-expire.md | 7 +- documentation/query/sql/create-mat-view.md | 39 +++-- 5 files changed, 121 insertions(+), 108 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 65d12a89d5..1977726f06 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - New row-level retention for materialized views, with creation syntax in [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/), immediate read filtering, and background reclamation for eligible policies - [Cold storage](/docs/concepts/cold-storage/) - Historical partitions move to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store as Parquet and stay queryable with normal SQL, covering the partition lifecycle, the range-read path, the immutability boundary at `DROP LOCAL`, and the manager and refresher roles - [Operating cold storage](/docs/operations/cold-storage/) - Setup guide and runbook: preparing the object store prefix, configuring the cluster, the two-step manager handoff, remote garbage collection, cold-read metrics, backup interaction, and a troubleshooting table - [Cold storage configuration](/docs/configuration/cold-storage/) - Every `cold.storage.*` setting, covering the store connection, roles, upload, read path, garbage collection, and request timeouts diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 1b13e0b159..f489e402f2 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -4,15 +4,15 @@ sidebar_label: EXPIRE ROWS description: EXPIRE ROWS is a row-level retention policy for passthrough materialized views. Keep the latest row per key, the top-N per group, or rows matching a - predicate — recomputed continuously, with expired rows hidden immediately, and - reclaimed in the background under the modes that allow it. + predicate. Policies are recomputed continuously, with expired rows hidden + immediately and reclaimed in the background under the modes that allow it. --- `EXPIRE ROWS` is a row-level retention policy for [materialized views](/docs/concepts/materialized-views/). Where [TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE -ROWS` decides retention **row by row** — keep the latest row per key, the top-N -per group, rows matching a predicate, and so on — and recomputes the result +ROWS` decides retention **row by row**. It can keep the latest row per key, the +top-N per group, rows matching a predicate, and so on. It recomputes the result continuously as the view refreshes. Expired rows disappear from query results **immediately** in every mode. Their @@ -30,10 +30,10 @@ only when the rule selects it for removal. | Mode | What it keeps | Syntax | Frees disk | | --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | | Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No — read filter only | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No — read filter only | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No — read filter only | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No (read filter only) | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. @@ -72,9 +72,9 @@ EXPIRE ROWS :::note `EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is -rejected. It is designed for a **passthrough** (non-aggregating) view — `SELECT -* FROM base` with no `SAMPLE BY` / `GROUP BY` — where the view mirrors base -rows 1:1 and reclamation is permanent. An aggregating view is **accepted with a +rejected. It is designed for a **passthrough** (non-aggregating) view, where +`SELECT * FROM base` has no `SAMPLE BY` / `GROUP BY`, the view mirrors base rows +1:1, and reclamation is permanent. An aggregating view is **accepted with a logged advisory** because a later refresh can regenerate reclaimed rows from base rows that still exist (see [Requirements](#requirements)). For base-table retention use @@ -88,29 +88,29 @@ base rows that still exist (see Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a continuously-maintained, pruned copy of a base table: -- **Current-state-per-key tables** — keep only the latest row per device, +- **Current-state-per-key tables**: keep only the latest row per device, symbol, or session (`KEEP LATEST`). -- **Per-group extremes or leaderboards** — keep the highest/lowest value per +- **Per-group extremes or leaderboards**: keep the highest/lowest value per group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). -- **Rolling row-level windows** — keep rows newer than a moving cutoff such as +- **Rolling row-level windows**: keep rows newer than a moving cutoff such as `now() - 7d`, at finer granularity than TTL's whole-partition drops (`WHEN predicate`). The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A -deterministic predicate — one that depends only on the row's own values — -selects the same rows more cheaply as a `WHERE` clause in the view's defining -query, which never copies the excluded rows into the view at all. See +deterministic predicate depends only on the row's own values and selects the +same rows more cheaply as a `WHERE` clause in the view's defining query, which +never copies the excluded rows into the view at all. See [`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based -retention on a base table is enough — it is cheaper and has no passthrough-view +retention on a base table is enough. It is cheaper and has no passthrough-view requirement. ## Requirements `EXPIRE ROWS` is designed for a **passthrough materialized view**: -- The view query keeps view rows 1:1 with base rows — a projection over a single +- The view query keeps view rows 1:1 with base rows as a projection over a single table, with or without a `WHERE` filter. See [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) for the full rules. @@ -120,17 +120,17 @@ requirement. A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that -replica down to the rows you want to keep — without touching the base table. +replica down to the rows you want to keep without touching the base table. A **non-passthrough (aggregating) view is accepted with a logged advisory** rather than rejected: physical reclamation only sticks when base-table retention -is aligned with the expiry horizon — a later incremental or full refresh can -regenerate a reclaimed row from base rows that still exist. +is aligned with the expiry horizon because a later incremental or full refresh +can regenerate a reclaimed row from base rows that still exist. A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a defining query that reads a policied view (as its base or in a join), and `ALTER ... SET EXPIRE` is rejected on a view that other materialized views -derive from — otherwise those views would copy expired rows on refresh. +derive from because those views would copy expired rows on refresh. ## Worked examples @@ -160,7 +160,7 @@ INSERT INTO trades VALUES A materialized view starts an **asynchronous initial refresh** on creation, so it may briefly return no rows. Check progress with `SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` -— the view is up to date when `refresh_base_table_txn = base_table_txn`. With +The view is up to date when `refresh_base_table_txn = base_table_txn`. With this small dataset that is effectively instant. ::: @@ -168,8 +168,8 @@ this small dataset that is effectively instant. ### Per-row predicate: `WHEN` A per-row predicate expires a row when it evaluates `TRUE`. The example below -uses a deterministic predicate — expire small trades (`amount < 1.5`) — -because it makes the keep-set easy to read off the sample data. In production +uses a deterministic predicate to expire small trades (`amount < 1.5`) because +it makes the keep-set easy to read off the sample data. In production that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which keeps those rows out of the view entirely; the rolling window further down is the case `WHEN` exists for. @@ -190,12 +190,12 @@ SELECT * FROM trades_sized ORDER BY ts; The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, -not `TRUE` — see [NULLs](#nulls)). +not `TRUE`). See [NULLs](#nulls). -A predicate on the designated timestamp gives a **rolling retention window** — -the main use for `WHEN`. It is re-evaluated on every read, so the visible set -rolls forward with the clock, and no `WHERE` clause can express it because the -defining query rejects `now()`: +A predicate on the designated timestamp gives a **rolling retention window**, +which is the main use for `WHEN`. It is re-evaluated on every read, so the +visible set rolls forward with the clock, and no `WHERE` clause can express it +because the defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( @@ -207,12 +207,13 @@ CREATE MATERIALIZED VIEW trades_recent AS ( and would be hidden; use recent data to see rows retained.) `WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows and both reclaim disk — the cleanup job proves either monotonic. +the same rows and both reclaim disk because the cleanup job proves either form +monotonic. ### Keep latest per key: `KEEP LATEST` -Keep only the most recent row per key — turning the passthrough view into a -live, current-state-per-symbol table: +Keep only the most recent row per key to turn the passthrough view into a live, +current-state-per-symbol table: ```questdb-sql title="Keep the latest row per symbol" CREATE MATERIALIZED VIEW trades_latest AS ( @@ -227,10 +228,10 @@ SELECT * FROM trades_latest ORDER BY ts; | ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -One row per symbol — the latest by the designated timestamp `ts`. As new trades -arrive, the kept row advances automatically. `PARTITION BY` may list multiple key -columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but the `ON` -column must be the view's designated timestamp. +The designated timestamp `ts` determines the latest row for each symbol. As new +trades arrive, the kept row advances automatically. `PARTITION BY` may list +multiple key columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but +the `ON` column must be the view's designated timestamp. ### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` @@ -255,7 +256,7 @@ are kept (a `NULL` is never less than the max). ### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` -Keep a per-group leaderboard — the `N` highest (or lowest) rows: +Keep a per-group leaderboard with the `N` highest (or lowest) rows: ```questdb-sql title="Keep the 2 highest-priced trades per symbol" CREATE MATERIALIZED VIEW trades_top2 AS ( @@ -290,8 +291,8 @@ CREATE MATERIALIZED VIEW trades_peak_win AS ( ``` A row expires when its price is below its symbol's maximum, so only the peak per -symbol survives — the same result as `trades_peak` above. From here you can -express richer rules, for example keeping rows within 5% of the peak +symbol survives. This produces the same result as `trades_peak` above. From here +you can express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window (`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). @@ -305,15 +306,15 @@ clock. window cannot be written as a `WHERE` clause at all: a view's defining query rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days" — the read filter re-evaluates `now()` on +way to say "keep the last 7 days". The read filter re-evaluates `now()` on every read, so the window rolls forward on its own and the cleanup job reclaims the disk behind it. This is what the `WHEN` form is for. **Put a deterministic predicate in the `WHERE` clause.** A predicate that -depends only on the row's own values — `symbol = 'BTC'`, `amount >= 1.5` — -describes the same surviving rows either way, so the two are near-equivalent in -what the view contains, and `WHERE` is the cheaper of the two at every stage. A -row the `WHERE` clause excludes is never copied into the view: +depends only on the row's own values, such as `symbol = 'BTC'` or +`amount >= 1.5`, describes the same surviving rows either way, so the two are +near-equivalent in what the view contains, and `WHERE` is the cheaper of the two +at every stage. A row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | @@ -321,7 +322,7 @@ row the `WHERE` clause excludes is never copied into the view: | Read cost | None | The keep-set filter is applied on every read of the view | | Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | | After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | -| Can be another view's base | Yes | No — a policied view is rejected as a base | +| Can be another view's base | Yes | No (a policied view is rejected as a base) | That last row is a hard constraint rather than a preference. If any other view will read this one, the policy is not available and the predicate has to go in @@ -330,7 +331,7 @@ the `WHERE` clause. The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and -kept by `EXPIRE ROWS WHEN amount < 1.5` — see [NULLs](#nulls). +kept by `EXPIRE ROWS WHEN amount < 1.5`. See [NULLs](#nulls). The two compose, and on a passthrough view that combination is usually the right shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy @@ -375,8 +376,8 @@ best-effort background cleanup. ### Read-time filter (authoritative) Every query against a policied view is transparently rewritten so that only the -kept rows are visible — **immediately, and regardless of whether cleanup has -run**. This is what makes results correct at all times: +kept rows are visible **immediately, regardless of whether cleanup has run**. +This is what makes results correct at all times: - **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB filtering is three-valued, so `FALSE` **and** `NULL` are kept (see @@ -402,8 +403,8 @@ or to `1` to disable partial-partition compaction; fully-expired partitions are still removed. The threshold does not delay a fixed, deterministic predicate, whose expired-row verdicts cannot change with time. -The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort** -— the read filter is authoritative, so deferred or skipped reclamation only +The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. +The read filter is authoritative, so deferred or skipped reclamation only affects disk usage, never query results. The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, @@ -417,7 +418,7 @@ still replicates: the compaction commits are ordinary WAL transactions, so replicas reclaim the identical rows by applying them. A read-only replica neither runs the job nor needs to. Disable the job with `cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay -filtered; only reclamation stops — the setting does not disable `EXPIRE ROWS` +filtered, but only reclamation stops; the setting does not disable `EXPIRE ROWS` itself). Cleanup settings are read at startup, so changing this property or the minimum expired fraction requires a restart. A failing sweep retries after one second, doubling the per-view retry gap up to a 10-minute cap. @@ -452,8 +453,8 @@ Therefore: ### A `NULL` threshold is rejected -A `WHEN` threshold that evaluates to a constant `NULL` expires nothing — -`ts < NULL` is never `TRUE` — so the policy would be inert. QuestDB refuses it at +A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because +`ts < NULL` is never `TRUE`, so the policy would be inert. QuestDB refuses it at `CREATE` and `ALTER` time rather than storing a view that silently never reclaims: @@ -487,11 +488,11 @@ cannot be checked in advance. ### Ties and determinism -`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min — deterministic by -construction. `KEEP N` makes the order total by appending the designated -timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base -table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if `(col, ts)` -is not already unique). +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min, making the result +deterministic by construction. `KEEP N` makes the order total by appending the +designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair +the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if +`(col, ts)` is not already unique). ### Combining with TTL @@ -514,7 +515,7 @@ CREATE MATERIALIZED VIEW trades_peak_3d AS ( This view reports the highest price of the **last three days**, so its answer can go **down** as the window moves: when the day holding a symbol's maximum ages out, the next-highest price within the window takes over. That is what the -two clauses ask for together — the view is no longer "the highest price ever", +two clauses ask for together. The view is no longer "the highest price ever"; it is "the highest price still retained". The base table is unaffected; it keeps whatever its own retention settings keep. @@ -530,8 +531,8 @@ expired now must stay expired forever. Two separate things can break that. The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, window `WHEN`) decide each row's fate by comparing it against the other rows in the view. A later refresh can remove or replace the row a key currently keeps, -which promotes an older row back into the keep-set — so the cleanup job never -deletes for these modes, whatever their predicate looks like. +which promotes an older row back into the keep-set. For this reason, the cleanup +job never deletes for these modes, whatever their predicate looks like. A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is arbitrary SQL, so the cleanup job reclaims disk only for predicates it can @@ -543,8 +544,9 @@ arbitrary SQL, so the cleanup job reclaims disk only for predicates it can (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). -Anything else **skips cleanup**: calendar units (`dateadd('M', -1, now())` — a -month is a variable amount), look-forward offsets (`dateadd('h', 1, now())`), +Anything else **skips cleanup**: calendar units such as +`dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets +(`dateadd('h', 1, now())`), further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` predicates. A skipped policy stays correct at read time (the filter recomputes on every read), but its disk is not reclaimed until the policy is changed to a @@ -555,9 +557,9 @@ proven shape. A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that **un-expire** as `now()` advances. The read filter recomputes `now()` on every read and stays correct, and the cleanup job skips such a policy rather than -risk physically deleting a row a later read must show — at the cost of disk -never being reclaimed for it. Write `WHEN` predicates that expire things in the -**past** or against fixed thresholds — never rows that the passage of time will +risk physically deleting a row a later read must show. The tradeoff is that its +disk is never reclaimed. Write `WHEN` predicates that expire things in the +**past** or against fixed thresholds, never rows that the passage of time will later keep. ::: @@ -588,15 +590,15 @@ FROM materialized_views(); `expire_enforcement` is the verdict the cleanup job acts on: -- `FILTER_AND_RECLAIM` — reads hide the expired rows and the job deletes them +- `FILTER_AND_RECLAIM`: reads hide the expired rows and the job deletes them from disk. -- `FILTER_ONLY` — reads hide the expired rows and they stay on disk. Every +- `FILTER_ONLY`: reads hide the expired rows and they stay on disk. Every relative and window policy reports this, as does a `WHEN` predicate that cannot be proven monotonic. ## Changing or removing a policy -Set, change, or drop a policy on an existing passthrough view — see +Set, change, or drop a policy on an existing passthrough view. See [`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): ```questdb-sql @@ -617,7 +619,7 @@ rather than breaking subsequent reads. keep-set over the whole physical view on every read. `KEEP LATEST` on an [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` - keeps the physical residue — and therefore the read cost — small. + keeps the physical residue, and therefore the read cost, small. - **Cleanup defers under continuous refresh.** Reclamation only proceeds when the view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative @@ -641,14 +643,14 @@ rather than breaking subsequent reads. ## Related documentation -- [Materialized views](/docs/concepts/materialized-views/) — the view type +- [Materialized views](/docs/concepts/materialized-views/): the view type `EXPIRE ROWS` runs on -- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views) — - the non-aggregating views `EXPIRE ROWS` applies to -- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) — full create +- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views): the + non-aggregating views `EXPIRE ROWS` applies to +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/): full create syntax, including the `EXPIRE ROWS` clause -- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - — set, change, or drop a policy -- [Time To Live (TTL)](/docs/concepts/ttl/) — partition-level retention by age -- [Storage policy](/docs/concepts/storage-policy/) — graduated partition +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/): + set, change, or drop a policy +- [Time To Live (TTL)](/docs/concepts/ttl/): partition-level retention by age +- [Storage policy](/docs/concepts/storage-policy/): graduated partition lifecycle (Enterprise) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index aaa8fcc523..62c5397c5f 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -182,10 +182,10 @@ CREATE MATERIALIZED VIEW trades_btc AS ( Reach for one when you want a maintained *subset* of a large table rather than a summary of it: -- **A narrowed replica** — one symbol, one tenant, one region, or a handful of +- **A narrowed replica**: one symbol, one tenant, one region, or a handful of columns out of a wide table, kept current automatically and queried without the base table's scan cost. -- **A row-level retention target** — attach an +- **A row-level retention target**: attach an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only some of the view's rows. The base table is left alone. @@ -310,21 +310,21 @@ table qualifies, with or without a filter: | Query | | | ----- | --- | | `SELECT * FROM trades` | Passthrough | -| `SELECT timestamp, symbol, price FROM trades` | Passthrough — column subset | -| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough — aliases are fine | -| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough — a row-local expression | -| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough — a filter only removes rows | +| `SELECT timestamp, symbol, price FROM trades` | Passthrough: column subset | +| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough: aliases are fine | +| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough: a row-local expression | +| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough: a filter only removes rows | | `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view | | `SELECT DISTINCT ...` | Rejected | | `... LATEST ON timestamp PARTITION BY symbol` | Rejected | | `JOIN`, `UNION` | Rejected | | `row_number() OVER (...)` and other window functions | Rejected | | `LIMIT` | Rejected | -| `ORDER BY` a non-timestamp column | Rejected — the view loses its designated timestamp | +| `ORDER BY` a non-timestamp column | Rejected: the view loses its designated timestamp | Everything in the rejected group produces output that depends on rows *other than* the one being emitted. An incremental refresh sees only the newly-arrived -slice of the base table, so it cannot compute those correctly — a `LIMIT 100` +slice of the base table, so it cannot compute those correctly. A `LIMIT 100` would admit 100 rows per refresh rather than 100 in total, and a `row_number()` would restart within each slice. @@ -334,8 +334,8 @@ SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to aggregate but named no bucket. `LIMIT` and window functions have their own messages. -To keep only some of a passthrough view's rows over time — the latest per key, -the top-N per group, or rows matching a predicate — attach an +To keep only some of a passthrough view's rows over time, such as the latest per +key, the top-N per group, or rows matching a predicate, attach an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also covers [when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 3303c43bd5..2fed2afa47 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -7,8 +7,9 @@ description: --- Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) -row-retention policy on a materialized view (designed for **passthrough** -views — see the concept page). For filtering and disk-reclamation behavior, see +row-retention policy on a materialized view. It is designed for +**passthrough views**; see the concept page. For filtering and disk-reclamation +behavior, see [How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). ## Syntax @@ -65,7 +66,7 @@ ALTER MATERIALIZED VIEW trades_mirror A `WHEN` predicate is the right tool for a cutoff that moves with the clock like this one. A deterministic predicate such as `amount < 1.5` is accepted too, but it selects the same rows more cheaply as a `WHERE` clause in the view's defining -query — see +query. See [`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). ```questdb-sql title="Keep the latest row per symbol" diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 75bdb8d5f1..57988083b9 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -18,22 +18,31 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] viewName [ START timestamp ] [ TIME ZONE timezone ] [ PERIOD ( LENGTH length [ TIME ZONE tz ] [ DELAY delay ] ) ] [ PERIOD ( SAMPLE BY INTERVAL ) ] ] -AS [ ( ] query [ ) ] -[ TIMESTAMP ( columnRef ) ] -[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) - [ TTL n timeUnit ] ] -[ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] -[ OWNED BY ownerName ] +AS +{ query +| ( query ) + [ TIMESTAMP ( columnRef ) ] + [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) + [ TTL n timeUnit ] ] + [ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] + [ OWNED BY ownerName ] +} ``` +Parentheses around `query` are optional only when the query ends the statement. +They are required when any trailing clause follows the query, including +`TIMESTAMP`, `PARTITION BY`, `TTL`, `EXPIRE ROWS`, or `OWNED BY`. + Where: + - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` - `query`: Either an aggregating query with `SAMPLE BY` or a time-based `GROUP BY`, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]` — designed for passthrough views (see below) +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. + This policy is designed for passthrough views (see below). ## Parameters @@ -287,14 +296,14 @@ The view's TTL is independent of the base table's TTL. See ## EXPIRE ROWS Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops -whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows — the latest -per key, the top-N per group, or rows matching a predicate — recomputed -continuously as the view refreshes. +whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows: the latest +per key, the top-N per group, or rows matching a predicate. It recomputes that +set continuously as the view refreshes. `EXPIRE ROWS` is designed for [**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) -— a projection over a single table, with no `SAMPLE BY` / `GROUP BY`, whose rows -stay 1:1 with the base. An aggregating view is accepted with a logged advisory +(a projection over a single table with no `SAMPLE BY` / `GROUP BY`, whose rows +stay 1:1 with the base). An aggregating view is accepted with a logged advisory (a later refresh can regenerate reclaimed rows). The defining query must not read another policied view, as its base or in a join: @@ -305,7 +314,7 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ``` A `WHEN` predicate is for rules that move with **wall-clock time**, such as a -rolling `ts < dateadd('d', -7, now())` window — the defining query cannot +rolling `ts < dateadd('d', -7, now())` window. The defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see @@ -324,7 +333,7 @@ EXPIRE ROWS A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit `ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved -`NULL` value, such as `ts < 2147483647 + 1` — see +`NULL` value, such as `ts < 2147483647 + 1`. See [A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). For filtering and disk-reclamation behavior, see @@ -334,7 +343,7 @@ remove a policy with A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the statement and first in effect: it removes rows from the view, and the -`EXPIRE ROWS` policy then applies to the rows that stay — see +`EXPIRE ROWS` policy then applies to the rows that stay. See [Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). See the [Expiring rows](/docs/concepts/expire-rows/) concept page for From 92dae9b567027369a73a64218d75d34001ed59b1 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 12:45:19 +0200 Subject: [PATCH 12/14] docs: refine expire rows reference --- documentation/concepts/expire-rows.md | 19 +++++++++----- .../configuration/materialized-views.md | 25 ++++++++++++++++--- documentation/query/functions/meta.md | 2 +- .../query/sql/alter-mat-view-set-expire.md | 7 +++--- documentation/query/sql/show.md | 6 +++-- 5 files changed, 44 insertions(+), 15 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index f489e402f2..5d251b0f3b 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -42,7 +42,11 @@ The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not -`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. This changes tie and `NULL` +behavior: the top-N form keeps exactly one row per group using the designated +timestamp as a descending tiebreaker. An integer or timestamp `NULL` sorts last +and is expired, while a floating-point `NULL` sorts first. The bare form keeps +every row tied at the extreme and every `NULL`. For how read filtering and physical reclamation differ between modes, see [How it works](#how-it-works). You can inspect the behavior selected for a view @@ -535,14 +539,17 @@ which promotes an older row back into the keep-set. For this reason, the cleanup job never deletes for these modes, whatever their predicate looks like. A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is -arbitrary SQL, so the cleanup job reclaims disk only for predicates it can -**prove** monotonic: +arbitrary SQL. QuestDB recognizes `now()`, `now_ns()`, `sysdate()`, +`systimestamp()` and `systimestamp_ns()` as wall-clock functions, and gives each +the same monotonicity proof. The cleanup job reclaims disk only for predicates +it can **prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare - clock (`ts < now()`), a bare clock minus a non-negative constant - (`ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` on a bare - clock (`ts < dateadd('d', -1, now())`, units `s`/`m`/`h`/`d`/`w` and finer). + clock (for example, `ts < now()`), a bare clock minus a non-negative constant + (for example, `ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` + on a bare clock (for example, `ts < dateadd('d', -1, now())`, with units + `s`/`m`/`h`/`d`/`w` and finer). Anything else **skips cleanup**: calendar units such as `dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index 49f73b12ca..4d22ddc6b2 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -3,9 +3,9 @@ title: Materialized views description: Configuration settings for materialized views in QuestDB. --- -These settings control materialized view SQL support and the background refresh -job. Materialized views can use dedicated worker threads or share the server's -common pool. +These settings control materialized view SQL support, background refresh, and +row-expiry cleanup. Materialized views can use dedicated worker threads or share +the server's common pool. ## cairo.mat.view.enabled @@ -22,6 +22,25 @@ Enables or disables SQL support and the refresh job for materialized views. When disabled, SQL executed by the materialized view refresh job always runs single-threaded. +## cairo.mat.view.row.expiry.cleanup.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables the background job that reclaims rows removed by an eligible +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Disabling the job does not +disable read filtering, so expired rows remain hidden from query results. + +## cairo.mat.view.row.expiry.cleanup.min.expired.fraction + +- **Default**: `0.5` +- **Reloadable**: no + +Minimum fraction of expired rows required before the background cleanup job +compacts a partially expired partition. Set this property to `0` to compact on +the first expired row, or to `1` to disable partial-partition compaction. Fully +expired partitions are still removed. + ## mat.view.refresh.worker.affinity - **Default**: equal to the CPU core count diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index b489cabf0d..a6d726a146 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -312,7 +312,7 @@ FROM materialized_views(); | view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | | --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | | trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | -| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_recent | valid | trades | 1 | 1 | ts < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | | trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index 2fed2afa47..a796e2464d 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -32,7 +32,7 @@ ALTER MATERIALIZED VIEW viewName DROP EXPIRE | `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | | `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | | `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | -| `CLEANUP EVERY` | Background reclamation cadence (e.g. `30m`, `1h`). Defaults to `1h` if omitted | +| `CLEANUP EVERY` | Background reclamation cadence as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` if omitted | Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with @@ -110,10 +110,11 @@ GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; | Error | Cause | | ----- | ----- | | `materialized view does not exist` | View with the specified name doesn't exist | -| `cannot set an EXPIRE ROWS policy on '...': it is the base of N materialized view(s), which would copy expired rows on refresh` | Other materialized views derive from this view | +| `cannot set an EXPIRE ROWS policy on '...': it is the base of N view(s), including '', which would copy expired rows on refresh` | Other views derive from this view; the message names the first dependent | | `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | -| `invalid EXPIRE ROWS KEEP LATEST column: ...` | A `PARTITION BY` key column does not exist | +| `invalid EXPIRE ROWS KEEP LATEST PARTITION BY column: ...` | A `PARTITION BY` key column does not exist | | `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires an orderable column, but '' is ` | The top-N form was given a column type that cannot be ordered | | `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | | `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | | `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 23e0f298b3..a737d8b809 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -41,7 +41,7 @@ SHOW { COLUMNS FROM tableName - `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live view. - `SHOW CREATE MATERIALIZED VIEW` returns a DDL query that allows you to - recreate a materialized view. + recreate a materialized view, including its retention clauses. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. - `SHOW GROUPS` shows all groups the user belongs or all groups in the system @@ -225,7 +225,9 @@ SHOW CREATE MATERIALIZED VIEW bbo_1s; | CREATE MATERIALIZED VIEW 'bbo_1s' WITH BASE 'market_data' REFRESH IMMEDIATE AS (SELECT timestamp, symbol, last(bids[1][1]) AS bid, last(asks[1][1]) AS ask FROM market_data SAMPLE BY 1s) PARTITION BY DAY; | This returns the `CREATE MATERIALIZED VIEW` statement that would recreate the -materialized view, including its base table, refresh strategy, and partitioning. +materialized view, including its base table, refresh strategy, partitioning, +`TTL`, and `EXPIRE ROWS` policy. When the policy specifies `CLEANUP EVERY`, the +output includes that cadence as well, so the DDL preserves the complete policy. ### SHOW CREATE TABLE From a85ebfde0d8e99c32fb6c7c5935fce9c557825b7 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 14:04:14 +0200 Subject: [PATCH 13/14] docs: correct expire rows cleanup semantics --- documentation/concepts/expire-rows.md | 74 ++++++++++++++++++--------- 1 file changed, 49 insertions(+), 25 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 5d251b0f3b..49dfff4df8 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -193,8 +193,8 @@ SELECT * FROM trades_sized ORDER BY ts; | BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is -`FALSE`), and any `NULL` amount would be kept too (the comparison is `UNKNOWN`, -not `TRUE`). See [NULLs](#nulls). +`FALSE`), and any `NULL` amount would be kept too because a comparison against +`NULL` evaluates to `FALSE` in QuestDB. See [NULLs](#nulls). A predicate on the designated timestamp gives a **rolling retention window**, which is the main use for `WHEN`. It is re-evaluated on every read, so the @@ -322,10 +322,10 @@ at every stage. A row the `WHERE` clause excludes is never copied into the view: | | `WHERE` in the query | `EXPIRE ROWS WHEN` | | --- | --- | --- | -| Storage | Row is never written | Row is written, and occupies disk until a sweep reclaims it | +| Storage | Row is never written | Row is written; only a `FILTER_AND_RECLAIM` policy can reclaim it later | | Read cost | None | The keep-set filter is applied on every read of the view | -| Write cost | None | The cleanup job rewrites partitions on the `CLEANUP EVERY` cadence | -| After a full refresh | Still excluded | Re-materialized from the base, then hidden and swept again | +| Write cost | None | Cleanup can rewrite partitions for `FILTER_AND_RECLAIM` policies | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden; eligible policies sweep it again | | Can be another view's base | Yes | No (a policied view is rejected as a base) | That last row is a hard constraint rather than a preference. If any other view @@ -384,8 +384,9 @@ kept rows are visible **immediately, regardless of whether cleanup has run**. This is what makes results correct at all times: - **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB - filtering is three-valued, so `FALSE` **and** `NULL` are kept (see - [NULLs](#nulls)). + comparisons use two-valued boolean semantics, so a comparison against `NULL` + is `FALSE`. Whether the complete predicate keeps or expires a `NULL` row + depends on operators such as `NOT`, `!=`, and `IS NULL` (see [NULLs](#nulls)). - **`KEEP LATEST`** returns the latest row per key using the designated timestamp. - **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a @@ -397,15 +398,21 @@ time-based predicate) reappears on the next read. ### Physical cleanup (best-effort) -A background job reclaims disk for non-active partitions. A fully-expired -partition is removed. Under a rolling clock-based predicate, a partially-expired -partition is compacted down to its survivors only when the expired-row fraction -reaches `cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults -to `0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff -moves through it. Set the property to `0` to compact on the first expired row, -or to `1` to disable partial-partition compaction; fully-expired partitions are -still removed. The threshold does not delay a fixed, deterministic predicate, -whose expired-row verdicts cannot change with time. +A background job reclaims disk for non-active partitions. It never rewrites the +active logical partition that receives new rows. A young view with only one +partition therefore reclaims no disk yet, even when its policy reports +`FILTER_AND_RECLAIM`. Once data creates a newer active partition, the older one +becomes eligible for cleanup. Read filtering remains effective throughout. + +A fully-expired eligible partition is removed. Under a rolling clock-based +predicate, a partially-expired partition is compacted down to its survivors only +when the expired-row fraction reaches +`cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults to +`0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff moves +through it. Set the property to `0` to compact on the first expired row, or to +`1` to disable partial-partition compaction; fully-expired partitions are still +removed. The threshold does not delay a fixed, deterministic predicate, whose +expired-row verdicts cannot change with time. The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. The read filter is authoritative, so deferred or skipped reclamation only @@ -431,9 +438,14 @@ To observe reclamation, compare the physical row count per partition before and after a sweep: ```questdb-sql title="Physical rows still on disk per partition" -SELECT name, numRows FROM table_partitions('trades_latest'); +SELECT name, numRows FROM table_partitions('trades_recent'); ``` +Use a view whose `expire_enforcement` is `FILTER_AND_RECLAIM`, such as +`trades_recent`, for this check. Its active partition remains unchanged after a +sweep. Insert data into a newer partition before expecting the current active +partition to become eligible for reclamation. + Reclamation **defers while a view is being refreshed continuously** and resumes on a quiet sweep. @@ -441,12 +453,20 @@ on a quiet sweep. ### NULLs -The keep-set is computed with three-valued logic, so a `NULL` value is never -*less than* a group maximum (the comparison is `UNKNOWN`, not `TRUE`). -Therefore: - -- **`KEEP HIGHEST/LOWEST` and value-based `WHEN`** predicates **keep** rows whose - value is `NULL`. +QuestDB comparisons use two-valued boolean semantics: a comparison against +`NULL` evaluates to `FALSE`, not `UNKNOWN`, and `EXPIRE ROWS WHEN` expires a row +only when the complete predicate evaluates to `TRUE`. The complete predicate +therefore determines whether a `NULL` row survives: + +- **A direct comparison such as `amount < 1.5`** is `FALSE` for a `NULL` amount, + so the policy keeps the row. +- **`NOT (amount >= 1.5)`** is `TRUE` for a `NULL` amount because the inner + comparison is `FALSE`, so the policy expires the row. Although this predicate + resembles `amount < 1.5`, the two differ for `NULL` values. +- **`amount != 1.5` and `amount IS NULL`** are also `TRUE` for a `NULL` amount, + so both expire the row. +- **`KEEP HIGHEST/LOWEST`** keeps a `NULL` because its comparison against the + group extreme is `FALSE`. - **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. - **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under @@ -625,8 +645,12 @@ rather than breaking subsequent reads. - **Reads recompute the keep-set.** A relative/window policy computes its keep-set over the whole physical view on every read. `KEEP LATEST` on an [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window - modes (and non-indexed keep-latest) scan the view. A tighter `CLEANUP EVERY` - keeps the physical residue, and therefore the read cost, small. + modes (and non-indexed keep-latest) scan the view. +- **Cleanup tuning applies only to reclaiming policies.** For a monotonic scalar + `WHEN` policy that reports `FILTER_AND_RECLAIM`, a tighter `CLEANUP EVERY` + reduces how long expired rows remain in eligible non-active partitions. It has + no reclamation effect on relative or window policies that report + `FILTER_ONLY`. - **Cleanup defers under continuous refresh.** Reclamation only proceeds when the view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative From e5ea2dd41e4312e2ef1af82d05406b9d27f8b550 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Thu, 27 Aug 2026 14:26:22 +0200 Subject: [PATCH 14/14] docs: align expire rows examples and retention guidance --- documentation/concepts/expire-rows.md | 156 +++++++++--------- documentation/operations/data-retention.md | 21 ++- documentation/query/functions/meta.md | 2 +- .../query/sql/alter-mat-view-set-expire.md | 10 +- documentation/query/sql/create-mat-view.md | 13 +- documentation/query/sql/show.md | 4 +- 6 files changed, 116 insertions(+), 90 deletions(-) diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md index 49dfff4df8..073f7e8021 100644 --- a/documentation/concepts/expire-rows.md +++ b/documentation/concepts/expire-rows.md @@ -27,13 +27,13 @@ on-disk storage is reclaimed afterwards by a background job under a monotonic Every mode keeps a defined set of rows and expires the rest. A row is expired only when the rule selects it for removal. -| Mode | What it keeps | Syntax | Frees disk | -| --------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --------------------- | -| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | -| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON ts] PARTITION BY cols` | No (read filter only) | -| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | -| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (read filter only) | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | `KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a window predicate, so the window `WHEN` is the general escape hatch. @@ -146,17 +146,18 @@ exactly which rows each policy keeps. ```questdb-sql title="Base table and sample data" CREATE TABLE trades ( symbol SYMBOL, + side SYMBOL, price DOUBLE, amount DOUBLE, - ts TIMESTAMP -) TIMESTAMP(ts) PARTITION BY DAY WAL; + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; INSERT INTO trades VALUES - ('BTC', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), - ('BTC', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), - ('BTC', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), - ('ETH', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), - ('ETH', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); + ('BTC', 'buy', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), + ('BTC', 'sell', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), + ('BTC', 'buy', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), + ('ETH', 'sell', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), + ('ETH', 'buy', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); ``` :::note @@ -183,14 +184,14 @@ CREATE MATERIALIZED VIEW trades_sized AS ( SELECT * FROM trades ) EXPIRE ROWS WHEN amount < 1.5; -SELECT * FROM trades_sized ORDER BY ts; +SELECT * FROM trades_sized ORDER BY timestamp; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is `FALSE`), and any `NULL` amount would be kept too because a comparison against @@ -204,15 +205,15 @@ because the defining query rejects `now()`: ```questdb-sql title="Keep the last 1 day" CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < dateadd('d', -1, now()); +) EXPIRE ROWS WHEN timestamp < dateadd('d', -1, now()); ``` (With the 2024 sample timestamps above, every row is already older than a day and would be hidden; use recent data to see rows retained.) -`WHEN ts < dateadd('d', -1, now())` and `WHEN ts < now() - 86400000000` retain -the same rows and both reclaim disk because the cleanup job proves either form -monotonic. +`WHEN timestamp < dateadd('d', -1, now())` and +`WHEN timestamp < now() - 86400000000` retain the same rows and both reclaim +disk because the cleanup job proves either form monotonic. ### Keep latest per key: `KEEP LATEST` @@ -224,18 +225,19 @@ CREATE MATERIALIZED VIEW trades_latest AS ( SELECT * FROM trades ) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; -SELECT * FROM trades_latest ORDER BY ts; +SELECT * FROM trades_latest ORDER BY timestamp; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -The designated timestamp `ts` determines the latest row for each symbol. As new -trades arrive, the kept row advances automatically. `PARTITION BY` may list -multiple key columns. You may write `KEEP LATEST ON ts PARTITION BY symbol`, but -the `ON` column must be the view's designated timestamp. +The designated `timestamp` column determines the latest row for each symbol. As +new trades arrive, the kept row advances automatically. `PARTITION BY` may list +multiple key columns. You may write +`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column must be the +view's designated timestamp. ### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` @@ -249,10 +251,10 @@ CREATE MATERIALIZED VIEW trades_peak AS ( SELECT * FROM trades_peak; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | `KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows @@ -270,12 +272,12 @@ CREATE MATERIALIZED VIEW trades_top2 AS ( SELECT * FROM trades_top2 ORDER BY symbol, price DESC; ``` -| symbol | price | amount | ts | -| ------ | ----- | ------ | --------------------------- | -| BTC | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | -| BTC | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | -| ETH | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | -| ETH | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, so both survive. Ties are broken by the designated timestamp, so the N-th @@ -298,7 +300,7 @@ A row expires when its price is below its symbol's maximum, so only the peak per symbol survives. This produces the same result as `trades_peak` above. From here you can express richer rules, for example keeping rows within 5% of the peak (`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window -(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY ts DESC) > 100`). +(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY timestamp DESC) > 100`). ## `WHERE` filter or `EXPIRE ROWS`? @@ -308,11 +310,12 @@ clock. **Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling window cannot be written as a `WHERE` clause at all: a view's defining query -rejects non-deterministic functions, so `WHERE ts > dateadd('d', -7, now())` is -not accepted. `EXPIRE ROWS WHEN ts < dateadd('d', -7, now())` is the supported -way to say "keep the last 7 days". The read filter re-evaluates `now()` on -every read, so the window rolls forward on its own and the cleanup job reclaims -the disk behind it. This is what the `WHEN` form is for. +rejects non-deterministic functions, so +`WHERE timestamp > dateadd('d', -7, now())` is not accepted. +`EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now())` is the supported way to +say "keep the last 7 days". The read filter re-evaluates `now()` on every read, +so the window rolls forward on its own and the cleanup job reclaims the disk +behind it. This is what the `WHEN` form is for. **Put a deterministic predicate in the `WHERE` clause.** A predicate that depends only on the row's own values, such as `symbol = 'BTC'` or @@ -344,7 +347,7 @@ fixes how long it keeps what it has. ```questdb-sql title="A filter for the subject, a policy for the horizon" CREATE MATERIALIZED VIEW trades_btc_recent AS ( SELECT * FROM trades WHERE symbol = 'BTC' -) EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 1h; +) EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 1h; ``` ### When a deterministic cutoff still belongs in a policy @@ -356,8 +359,10 @@ dropping the view and re-creating it, which re-materializes it from the base. Changing a policy is a metadata operation: ```questdb-sql title="Retuning a retention horizon without a rebuild" -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-06-01T00:00:00.000000Z'; -ALTER MATERIALIZED VIEW trades_recent SET EXPIRE ROWS WHEN ts < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-07-01T00:00:00.000000Z'; ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; ``` @@ -478,14 +483,14 @@ therefore determines whether a `NULL` row survives: ### A `NULL` threshold is rejected A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because -`ts < NULL` is never `TRUE`, so the policy would be inert. QuestDB refuses it at -`CREATE` and `ALTER` time rather than storing a view that silently never -reclaims: +`timestamp < NULL` is never `TRUE`, so the policy would be inert. QuestDB +refuses it at `CREATE` and `ALTER` time rather than storing a view that silently +never reclaims: ```questdb-sql title="Rejected: the threshold is NULL" CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < CAST(NULL AS TIMESTAMP); +) EXPIRE ROWS WHEN timestamp < CAST(NULL AS TIMESTAMP); -- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire ``` @@ -498,17 +503,17 @@ arithmetic threshold can land on that value: -- LONG overflow CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < 4611686018427387904 * 2; +) EXPIRE ROWS WHEN timestamp < 4611686018427387904 * 2; -- INT overflow, reached three orders of magnitude sooner CREATE MATERIALIZED VIEW trades_recent AS ( SELECT * FROM trades -) EXPIRE ROWS WHEN ts < 2147483647 + 1; +) EXPIRE ROWS WHEN timestamp < 2147483647 + 1; ``` Only thresholds that are constant at definition time are checked this way. One -built from a clock, such as `ts < now() - 3600000000`, is evaluated per read and -cannot be checked in advance. +built from a clock, such as `timestamp < now() - 3600000000`, is evaluated per +read and cannot be checked in advance. ### Ties and determinism @@ -516,7 +521,7 @@ cannot be checked in advance. deterministic by construction. `KEEP N` makes the order total by appending the designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if -`(col, ts)` is not already unique). +`(col, timestamp)` is not already unique). ### Combining with TTL @@ -566,10 +571,11 @@ it can **prove** monotonic: - clock-free predicates (`WHEN amount < 1.5`), and - designated-timestamp thresholds of a proven advancing-clock shape: a bare - clock (for example, `ts < now()`), a bare clock minus a non-negative constant - (for example, `ts < now() - 7200000000`), or a fixed-unit look-back `dateadd` - on a bare clock (for example, `ts < dateadd('d', -1, now())`, with units - `s`/`m`/`h`/`d`/`w` and finer). + clock (for example, `timestamp < now()`), a bare clock minus a non-negative + constant (for example, `timestamp < now() - 7200000000`), or a fixed-unit + look-back `dateadd` on a bare clock (for example, + `timestamp < dateadd('d', -1, now())`, with units `s`/`m`/`h`/`d`/`w` and + finer). Anything else **skips cleanup**: calendar units such as `dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets @@ -581,11 +587,11 @@ proven shape. :::warning -A non-monotonic predicate such as `WHEN ts > now()` expires *future* rows that -**un-expire** as `now()` advances. The read filter recomputes `now()` on every -read and stays correct, and the cleanup job skips such a policy rather than -risk physically deleting a row a later read must show. The tradeoff is that its -disk is never reclaimed. Write `WHEN` predicates that expire things in the +A non-monotonic predicate such as `WHEN timestamp > now()` expires *future* +rows that **un-expire** as `now()` advances. The read filter recomputes `now()` +on every read and stays correct, and the cleanup job skips such a policy rather +than risk physically deleting a row a later read must show. The tradeoff is that +its disk is never reclaimed. Write `WHEN` predicates that expire things in the **past** or against fixed thresholds, never rows that the passage of time will later keep. @@ -593,7 +599,9 @@ later keep. ## Inspecting a policy -`SHOW CREATE MATERIALIZED VIEW` renders the clause as written: +`SHOW CREATE MATERIALIZED VIEW` renders the policy in replayable DDL. It omits +`CLEANUP EVERY` when the cadence is the default `1h` and includes it for a +non-default cadence: ```questdb-sql SHOW CREATE MATERIALIZED VIEW trades_latest; @@ -655,9 +663,9 @@ rather than breaking subsequent reads. view is quiescent and fully applied, so a view being refreshed continuously defers reclamation to a quiet sweep. The read filter stays authoritative meanwhile. -- **`KEEP LATEST [ON ts]`.** The optional `ON ts` is accepted for familiarity but - the view's designated timestamp is always used; naming a different column is - rejected. +- **`KEEP LATEST [ON timestamp]`.** The optional `ON timestamp` is accepted for + familiarity, but the view's designated timestamp is always used; naming a + different column is rejected. - **Cleanup eligibility.** See [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check a view's verdict with `materialized_views().expire_enforcement`. diff --git a/documentation/operations/data-retention.md b/documentation/operations/data-retention.md index cb27786907..b35fe7f969 100644 --- a/documentation/operations/data-retention.md +++ b/documentation/operations/data-retention.md @@ -8,14 +8,21 @@ description: How to employ a data retention strategy to delete old data and save The nature of [time-series data](/blog/what-is-time-series-data/) is that the relevance of information diminishes over time. If stale data is no longer required, users can delete old data from QuestDB to either save disk space or adhere to a data retention policy. This is -achieved in QuestDB by removing data partitions from a table. +achieved by removing or tiering table partitions, or by applying row-level +retention to a materialized view. -QuestDB offers four approaches for data retention: +QuestDB offers five approaches for data retention: - **TTL** _(automatic, open source)_: Use [Time To Live (TTL)](/docs/concepts/ttl/) to automatically drop partitions when data ages beyond a specified threshold. This is the simplest approach and is available in both open source and Enterprise editions. +- **EXPIRE ROWS** _(automatic, materialized views only)_: Use + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) for row-granular retention on a + materialized view, such as keeping the latest row per key, the top-N rows per + group, or expiring rows selected by a predicate. The base table is unaffected. + Some policies hide expired rows without reclaiming their disk space, so check + the policy's enforcement mode when storage reduction matters. - **Storage policy** _(automatic, Enterprise only)_: Use a [storage policy](/docs/concepts/storage-policy/) to automate the partition lifecycle — convert to Parquet locally and drop old data on a schedule. This is @@ -30,8 +37,9 @@ QuestDB offers four approaches for data retention: - **Manual**: Use `DROP PARTITION` commands as described on this page for explicit control over which partitions to remove and when. -This page provides a high-level overview of partitioning with examples to drop -data by date. For more details on partitioning, see the +This page summarizes the available retention approaches, then provides +partitioning examples that drop data by date. For more details on partitioning, +see the [partitioning](/docs/concepts/partitions/) page. ## Manual partition management @@ -109,8 +117,9 @@ syntax. Partitions may be dropped by: chronologically. Depending on the types of queries users are performing on a dataset, it may not be desirable to have gaps caused by dropped partitions. - Unlike TTL, `DROP PARTITION` commands must be triggered manually or via - external scheduling (e.g., cron jobs). For fully automated retention, consider - using [TTL](/docs/concepts/ttl/) instead. + external scheduling (e.g., cron jobs). For automated partition retention, + consider [TTL](/docs/concepts/ttl/) or a + [storage policy](/docs/concepts/storage-policy/) instead. ### Example diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index a6d726a146..b489cabf0d 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -312,7 +312,7 @@ FROM materialized_views(); | view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | | --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | | trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | -| trades_recent | valid | trades | 1 | 1 | ts < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | | trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md index a796e2464d..56d53c6543 100644 --- a/documentation/query/sql/alter-mat-view-set-expire.md +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -58,9 +58,17 @@ set, the policy takes effect without rebuilding the view. See ## Examples +These examples use `trades_mirror`, a passthrough materialized view over a +`trades` table with `symbol`, `side`, `price`, `amount`, and designated +`timestamp` columns: + +```questdb-sql title="Create the passthrough view" +CREATE MATERIALIZED VIEW trades_mirror AS (SELECT * FROM trades); +``` + ```questdb-sql title="Rolling 7-day window, with a tighter cleanup cadence" ALTER MATERIALIZED VIEW trades_mirror - SET EXPIRE ROWS WHEN ts < dateadd('d', -7, now()) CLEANUP EVERY 30m; + SET EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 30m; ``` A `WHEN` predicate is the right tool for a cutoff that moves with the clock like diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 57988083b9..efabfc9ed5 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -41,7 +41,7 @@ Where: `GROUP BY`, or a [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table -- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON ts] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. This policy is designed for passthrough views (see below). ## Parameters @@ -84,10 +84,11 @@ Where: ```questdb-sql title="Base table" CREATE TABLE trades ( - timestamp TIMESTAMP, symbol SYMBOL, + side SYMBOL, price DOUBLE, - amount DOUBLE + amount DOUBLE, + timestamp TIMESTAMP ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` @@ -314,7 +315,7 @@ CREATE MATERIALIZED VIEW trades_latest AS ( ``` A `WHEN` predicate is for rules that move with **wall-clock time**, such as a -rolling `ts < dateadd('d', -7, now())` window. The defining query cannot +rolling `timestamp < dateadd('d', -7, now())` window. The defining query cannot express those, because it rejects non-deterministic functions. A predicate that depends only on the row's own values belongs in the query's `WHERE` clause instead, which keeps those rows out of the view entirely; see @@ -332,8 +333,8 @@ EXPIRE ROWS A `WHEN` threshold that is constant at definition time and evaluates to `NULL` is rejected, since it would expire nothing. That covers the explicit -`ts < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the reserved -`NULL` value, such as `ts < 2147483647 + 1`. See +`timestamp < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the +reserved `NULL` value, such as `timestamp < 2147483647 + 1`. See [A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). For filtering and disk-reclamation behavior, see diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index a737d8b809..197d1df8b3 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -226,8 +226,8 @@ SHOW CREATE MATERIALIZED VIEW bbo_1s; This returns the `CREATE MATERIALIZED VIEW` statement that would recreate the materialized view, including its base table, refresh strategy, partitioning, -`TTL`, and `EXPIRE ROWS` policy. When the policy specifies `CLEANUP EVERY`, the -output includes that cadence as well, so the DDL preserves the complete policy. +`TTL`, and `EXPIRE ROWS` policy. The output includes `CLEANUP EVERY` only when +the cadence differs from the default `1h`. ### SHOW CREATE TABLE