diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 65d12a89d..1977726f0 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - New row-level retention for materialized views, with creation syntax in [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/), immediate read filtering, and background reclamation for eligible policies - [Cold storage](/docs/concepts/cold-storage/) - Historical partitions move to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store as Parquet and stay queryable with normal SQL, covering the partition lifecycle, the range-read path, the immutability boundary at `DROP LOCAL`, and the manager and refresher roles - [Operating cold storage](/docs/operations/cold-storage/) - Setup guide and runbook: preparing the object store prefix, configuring the cluster, the two-step manager handoff, remote garbage collection, cold-read metrics, backup interaction, and a troubleshooting table - [Cold storage configuration](/docs/configuration/cold-storage/) - Every `cold.storage.*` setting, covering the store connection, roles, upload, read path, garbage collection, and request timeouts diff --git a/documentation/concepts/expire-rows.md b/documentation/concepts/expire-rows.md new file mode 100644 index 000000000..073f7e802 --- /dev/null +++ b/documentation/concepts/expire-rows.md @@ -0,0 +1,695 @@ +--- +title: Expiring rows (EXPIRE ROWS) +sidebar_label: EXPIRE ROWS +description: + EXPIRE ROWS is a row-level retention policy for passthrough materialized + views. Keep the latest row per key, the top-N per group, or rows matching a + predicate. Policies are recomputed continuously, with expired rows hidden + immediately and reclaimed in the background under the modes that allow it. +--- + +`EXPIRE ROWS` is a row-level retention policy for +[materialized views](/docs/concepts/materialized-views/). Where +[TTL](/docs/concepts/ttl/) drops whole partitions once they age out, `EXPIRE +ROWS` decides retention **row by row**. It can keep the latest row per key, the +top-N per group, rows matching a predicate, and so on. It recomputes the result +continuously as the view refreshes. + +Expired rows disappear from query results **immediately** in every mode. Their +on-disk storage is reclaimed afterwards by a background job under a monotonic +`WHEN` predicate; the relative modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, +`KEEP N`) and window predicates hide rows without freeing disk. See +[The modes](#the-modes) and +[Monotonicity and cleanup safety](#monotonicity-and-cleanup-safety). + +## The modes + +Every mode keeps a defined set of rows and expires the rest. A row is expired +only when the rule selects it for removal. + +| Mode | What it keeps | Syntax | Frees disk | +| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | +| Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | +| Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (read filter only) | +| Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Keep top-N | The `N` highest / lowest rows per group | `EXPIRE ROWS KEEP N HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | +| Window predicate | Rows for which a window predicate is **not** `TRUE` | `EXPIRE ROWS WHEN windowPredicate` | No (read filter only) | + +`KEEP HIGHEST/LOWEST` and `KEEP N` are convenience forms that desugar to a +window predicate, so the window `WHEN` is the general escape hatch. + +The bare `KEEP HIGHEST/LOWEST` form accepts `BYTE`, `SHORT`, `INT`, `LONG`, +`FLOAT`, `DOUBLE`, `DATE`, `TIMESTAMP` and `DECIMAL` columns. The top-N form has +a broader type surface: `KEEP N HIGHEST/LOWEST` ranks with `ORDER BY`, so it +accepts any orderable column type. For example, use `KEEP 1 HIGHEST symbol`, not +`KEEP HIGHEST symbol`, to rank a `SYMBOL` column. This changes tie and `NULL` +behavior: the top-N form keeps exactly one row per group using the designated +timestamp as a descending tiebreaker. An integer or timestamp `NULL` sorts last +and is expired, while a floating-point `NULL` sorts first. The bare form keeps +every row tied at the extreme and every `NULL`. + +For how read filtering and physical reclamation differ between modes, see +[How it works](#how-it-works). You can inspect the behavior selected for a view +through `materialized_views().expire_enforcement`; see +[Inspecting a policy](#inspecting-a-policy). + +The clause is attached to a passthrough `CREATE MATERIALIZED VIEW` (after the +query, and after `PARTITION BY` if present), or set later with +[`ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS`](/docs/query/sql/alter-mat-view-set-expire/): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +| Element | Meaning | +| ------------------ | ---------------------------------------------------------------------------------------- | +| `predicate` | Any boolean expression over the view's columns. A row expires when it evaluates `TRUE`. | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp. | +| `ON timestampCol` | Optional; if given it must name the view's designated timestamp. | +| `HIGHEST\|LOWEST` | Keep rows at the max / min of `col` per group (`N` omitted), or the top `N` per group. | +| `CLEANUP EVERY` | How often the background reclamation job runs for this view: `` with unit `s`/`m`/`h`/`d`/`w`. Defaults to `1h` if omitted. | + +:::note + +`EXPIRE ROWS` is **materialized-view-only**: `CREATE TABLE ... EXPIRE ROWS` is +rejected. It is designed for a **passthrough** (non-aggregating) view, where +`SELECT * FROM base` has no `SAMPLE BY` / `GROUP BY`, the view mirrors base rows +1:1, and reclamation is permanent. An aggregating view is **accepted with a +logged advisory** because a later refresh can regenerate reclaimed rows from +base rows that still exist (see +[Requirements](#requirements)). For base-table retention use +[TTL](/docs/concepts/ttl/) or, on Enterprise, +[storage policies](/docs/concepts/storage-policy/). + +::: + +## When to use EXPIRE ROWS + +Reach for `EXPIRE ROWS` on a passthrough materialized view when you want a +continuously-maintained, pruned copy of a base table: + +- **Current-state-per-key tables**: keep only the latest row per device, + symbol, or session (`KEEP LATEST`). +- **Per-group extremes or leaderboards**: keep the highest/lowest value per + group, or the top-N (`KEEP HIGHEST/LOWEST`, `KEEP N`). +- **Rolling row-level windows**: keep rows newer than a moving cutoff such as + `now() - 7d`, at finer granularity than TTL's whole-partition drops + (`WHEN predicate`). + +The `WHEN` form earns its keep on predicates that involve **wall-clock time**. A +deterministic predicate depends only on the row's own values and selects the +same rows more cheaply as a `WHERE` clause in the view's defining query, which +never copies the excluded rows into the view at all. See +[`WHERE` filter or `EXPIRE ROWS`?](#where-filter-or-expire-rows). + +Use [TTL](/docs/concepts/ttl/) instead when partition-granularity, age-based +retention on a base table is enough. It is cheaper and has no passthrough-view +requirement. + +## Requirements + +`EXPIRE ROWS` is designed for a **passthrough materialized view**: + +- The view query keeps view rows 1:1 with base rows as a projection over a single + table, with or without a `WHERE` filter. See + [which queries are passthrough](/docs/concepts/materialized-views/#which-queries-are-passthrough) + for the full rules. +- The view inherits the base table's + [designated timestamp](/docs/concepts/designated-timestamp/), partitioning and + symbol indexes. + +A passthrough view mirrors its base table 1:1 and refreshes incrementally, so it +is effectively a continuously-maintained replica. `EXPIRE ROWS` prunes that +replica down to the rows you want to keep without touching the base table. + +A **non-passthrough (aggregating) view is accepted with a logged advisory** +rather than rejected: physical reclamation only sticks when base-table retention +is aligned with the expiry horizon because a later incremental or full refresh +can regenerate a reclaimed row from base rows that still exist. + +A policied view must also stand alone: `CREATE MATERIALIZED VIEW` rejects a +defining query that reads a policied view (as its base or in a join), and +`ALTER ... SET EXPIRE` is rejected on a view that other materialized views +derive from because those views would copy expired rows on refresh. + +## Worked examples + +The following walks through every mode on a small fixed dataset so you can see +exactly which rows each policy keeps. + +### Setup + +```questdb-sql title="Base table and sample data" +CREATE TABLE trades ( + symbol SYMBOL, + side SYMBOL, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; + +INSERT INTO trades VALUES + ('BTC', 'buy', 100.0, 1.0, '2024-01-01T10:00:00.000000Z'), + ('BTC', 'sell', 105.0, 2.0, '2024-01-01T11:00:00.000000Z'), + ('BTC', 'buy', 102.0, 1.5, '2024-01-02T09:00:00.000000Z'), + ('ETH', 'sell', 50.0, 3.0, '2024-01-01T10:30:00.000000Z'), + ('ETH', 'buy', 55.0, 1.0, '2024-01-02T08:00:00.000000Z'); +``` + +:::note + +A materialized view starts an **asynchronous initial refresh** on creation, so +it may briefly return no rows. Check progress with +`SELECT view_name, view_status, base_table_txn, refresh_base_table_txn FROM materialized_views();` +The view is up to date when `refresh_base_table_txn = base_table_txn`. With +this small dataset that is effectively instant. + +::: + +### Per-row predicate: `WHEN` + +A per-row predicate expires a row when it evaluates `TRUE`. The example below +uses a deterministic predicate to expire small trades (`amount < 1.5`) because +it makes the keep-set easy to read off the sample data. In production +that rule belongs in the view's `WHERE` clause (`WHERE amount >= 1.5`), which +keeps those rows out of the view entirely; the rolling window further down is +the case `WHEN` exists for. + +```questdb-sql title="Expire rows where amount < 1.5" +CREATE MATERIALIZED VIEW trades_sized AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN amount < 1.5; + +SELECT * FROM trades_sized ORDER BY timestamp; +``` + +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | + +The two `amount = 1.0` rows are expired. `amount = 1.5` is kept (`1.5 < 1.5` is +`FALSE`), and any `NULL` amount would be kept too because a comparison against +`NULL` evaluates to `FALSE` in QuestDB. See [NULLs](#nulls). + +A predicate on the designated timestamp gives a **rolling retention window**, +which is the main use for `WHEN`. It is re-evaluated on every read, so the +visible set rolls forward with the clock, and no `WHERE` clause can express it +because the defining query rejects `now()`: + +```questdb-sql title="Keep the last 1 day" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN timestamp < dateadd('d', -1, now()); +``` + +(With the 2024 sample timestamps above, every row is already older than a day +and would be hidden; use recent data to see rows retained.) + +`WHEN timestamp < dateadd('d', -1, now())` and +`WHEN timestamp < now() - 86400000000` retain the same rows and both reclaim +disk because the cleanup job proves either form monotonic. + +### Keep latest per key: `KEEP LATEST` + +Keep only the most recent row per key to turn the passthrough view into a live, +current-state-per-symbol table: + +```questdb-sql title="Keep the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +SELECT * FROM trades_latest ORDER BY timestamp; +``` + +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | + +The designated `timestamp` column determines the latest row for each symbol. As +new trades arrive, the kept row advances automatically. `PARTITION BY` may list +multiple key columns. You may write +`KEEP LATEST ON timestamp PARTITION BY symbol`, but the `ON` column must be the +view's designated timestamp. + +### Keep extremes per group: `KEEP HIGHEST` / `KEEP LOWEST` + +Keep the rows tied at the group maximum (or minimum) of a column: + +```questdb-sql title="Keep the highest-priced trade per symbol" +CREATE MATERIALIZED VIEW trades_peak AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_peak; +``` + +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | + +`KEEP LOWEST price PARTITION BY symbol` keeps the cheapest instead (BTC `100.0`, +ETH `50.0`). All rows **tied** at the extreme are kept, and `NULL`-valued rows +are kept (a `NULL` is never less than the max). + +### Keep top-N per group: `KEEP N HIGHEST` / `KEEP N LOWEST` + +Keep a per-group leaderboard with the `N` highest (or lowest) rows: + +```questdb-sql title="Keep the 2 highest-priced trades per symbol" +CREATE MATERIALIZED VIEW trades_top2 AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; + +SELECT * FROM trades_top2 ORDER BY symbol, price DESC; +``` + +| symbol | side | price | amount | timestamp | +| ------ | ---- | ----- | ------ | --------------------------- | +| BTC | sell | 105.0 | 2.0 | 2024-01-01T11:00:00.000000Z | +| BTC | buy | 102.0 | 1.5 | 2024-01-02T09:00:00.000000Z | +| ETH | buy | 55.0 | 1.0 | 2024-01-02T08:00:00.000000Z | +| ETH | sell | 50.0 | 3.0 | 2024-01-01T10:30:00.000000Z | + +BTC keeps its two highest (`105`, `102`) and drops `100`; ETH has only two rows, +so both survive. Ties are broken by the designated timestamp, so the N-th +boundary is deterministic. + +### Window predicate: the escape hatch + +`KEEP HIGHEST/LOWEST` and `KEEP N` are shorthand for window predicates. When you +need a rule they do not cover, write the window predicate directly in `WHEN`. +For example, this is exactly what `KEEP HIGHEST price PARTITION BY symbol` +expands to: + +```questdb-sql title="Equivalent to KEEP HIGHEST, written as a window predicate" +CREATE MATERIALIZED VIEW trades_peak_win AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN price < max(price) OVER (PARTITION BY symbol); +``` + +A row expires when its price is below its symbol's maximum, so only the peak per +symbol survives. This produces the same result as `trades_peak` above. From here +you can express richer rules, for example keeping rows within 5% of the peak +(`WHEN price < 0.95 * max(price) OVER (PARTITION BY symbol)`) or a ranked window +(`WHEN row_number() OVER (PARTITION BY symbol ORDER BY timestamp DESC) > 100`). + +## `WHERE` filter or `EXPIRE ROWS`? + +A passthrough view can exclude rows in two places: a `WHERE` clause in its +defining query, or an `EXPIRE ROWS WHEN` predicate. The dividing line is the +clock. + +**Use `EXPIRE ROWS WHEN` for rules that move with wall-clock time.** A rolling +window cannot be written as a `WHERE` clause at all: a view's defining query +rejects non-deterministic functions, so +`WHERE timestamp > dateadd('d', -7, now())` is not accepted. +`EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now())` is the supported way to +say "keep the last 7 days". The read filter re-evaluates `now()` on every read, +so the window rolls forward on its own and the cleanup job reclaims the disk +behind it. This is what the `WHEN` form is for. + +**Put a deterministic predicate in the `WHERE` clause.** A predicate that +depends only on the row's own values, such as `symbol = 'BTC'` or +`amount >= 1.5`, describes the same surviving rows either way, so the two are +near-equivalent in what the view contains, and `WHERE` is the cheaper of the two +at every stage. A row the `WHERE` clause excludes is never copied into the view: + +| | `WHERE` in the query | `EXPIRE ROWS WHEN` | +| --- | --- | --- | +| Storage | Row is never written | Row is written; only a `FILTER_AND_RECLAIM` policy can reclaim it later | +| Read cost | None | The keep-set filter is applied on every read of the view | +| Write cost | None | Cleanup can rewrite partitions for `FILTER_AND_RECLAIM` policies | +| After a full refresh | Still excluded | Re-materialized from the base, then hidden; eligible policies sweep it again | +| Can be another view's base | Yes | No (a policied view is rejected as a base) | + +That last row is a hard constraint rather than a preference. If any other view +will read this one, the policy is not available and the predicate has to go in +the `WHERE` clause. + +The two forms are not exact negations of each other on `NULL`s: `WHERE` keeps a +row only when the predicate is `TRUE`, while `EXPIRE ROWS WHEN` expires a row +only when it is `TRUE`. A `NULL` amount is dropped by `WHERE amount >= 1.5` and +kept by `EXPIRE ROWS WHEN amount < 1.5`. See [NULLs](#nulls). + +The two compose, and on a passthrough view that combination is usually the right +shape: the `WHERE` clause fixes what the view is about, and the `WHEN` policy +fixes how long it keeps what it has. + +```questdb-sql title="A filter for the subject, a policy for the horizon" +CREATE MATERIALIZED VIEW trades_btc_recent AS ( + SELECT * FROM trades WHERE symbol = 'BTC' +) EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 1h; +``` + +### When a deterministic cutoff still belongs in a policy + +One case pulls a deterministic predicate back into `EXPIRE ROWS`: a fixed +threshold you expect to advance by hand. There is no +`ALTER MATERIALIZED VIEW ... AS `, so changing a `WHERE` clause means +dropping the view and re-creating it, which re-materializes it from the base. +Changing a policy is a metadata operation: + +```questdb-sql title="Retuning a retention horizon without a rebuild" +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-06-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent + SET EXPIRE ROWS WHEN timestamp < '2024-07-01T00:00:00.000000Z'; +ALTER MATERIALIZED VIEW trades_recent DROP EXPIRE; +``` + +The rebuild a `WHERE` change forces is not only slow, it can lose data: if the +base table has its own [TTL](/docs/concepts/ttl/), re-creating the view reads a +base that no longer holds everything the view held. A view whose retention +horizon is longer than its base table's cannot afford to be rebuilt, so its +cutoff belongs in a policy. + +A rule that compares rows against each other has no `WHERE` equivalent either: +`KEEP LATEST`, `KEEP N HIGHEST/LOWEST` and window predicates cannot be expressed +in the defining query, because a `LATEST ON` or a window function there makes +the view non-passthrough. + +## How it works + +`EXPIRE ROWS` has two cooperating parts: an authoritative read-time filter and a +best-effort background cleanup. + +### Read-time filter (authoritative) + +Every query against a policied view is transparently rewritten so that only the +kept rows are visible **immediately, regardless of whether cleanup has run**. +This is what makes results correct at all times: + +- **Per-row `WHEN`** keeps rows where the predicate is not `TRUE`. QuestDB + comparisons use two-valued boolean semantics, so a comparison against `NULL` + is `FALSE`. Whether the complete predicate keeps or expires a `NULL` row + depends on operators such as `NOT`, `!=`, and `IS NULL` (see [NULLs](#nulls)). +- **`KEEP LATEST`** returns the latest row per key using the designated + timestamp. +- **`KEEP HIGHEST/LOWEST/N` and window `WHEN`** compute the keep-set with a + window function over the whole view. + +Because the filter is applied at query time, a freshly-refreshed row that should +be expired is hidden the moment it lands, and a row that should reappear (under a +time-based predicate) reappears on the next read. + +### Physical cleanup (best-effort) + +A background job reclaims disk for non-active partitions. It never rewrites the +active logical partition that receives new rows. A young view with only one +partition therefore reclaims no disk yet, even when its policy reports +`FILTER_AND_RECLAIM`. Once data creates a newer active partition, the older one +becomes eligible for cleanup. Read filtering remains effective throughout. + +A fully-expired eligible partition is removed. Under a rolling clock-based +predicate, a partially-expired partition is compacted down to its survivors only +when the expired-row fraction reaches +`cairo.mat.view.row.expiry.cleanup.min.expired.fraction`, which defaults to +`0.5`. This avoids repeatedly rewriting a boundary partition as the cutoff moves +through it. Set the property to `0` to compact on the first expired row, or to +`1` to disable partial-partition compaction; fully-expired partitions are still +removed. The threshold does not delay a fixed, deterministic predicate, whose +expired-row verdicts cannot change with time. + +The job runs at the `CLEANUP EVERY` cadence (default `1h`) and is **best-effort**. +The read filter is authoritative, so deferred or skipped reclamation only +affects disk usage, never query results. + +The job runs only under a monotonic `WHEN` predicate. It skips `KEEP LATEST`, +`KEEP HIGHEST/LOWEST`, `KEEP N` and window policies entirely: a later refresh +can remove the row those modes currently keep, which promotes an older row back +into the keep-set, and the job cannot reconstruct a row it has already deleted. +Those views accumulate their expired rows on disk. + +On QuestDB Enterprise, cleanup runs on the **primary only**, but the reclamation +still replicates: the compaction commits are ordinary WAL transactions, so +replicas reclaim the identical rows by applying them. A read-only replica neither +runs the job nor needs to. Disable the job with +`cairo.mat.view.row.expiry.cleanup.enabled=false` in `server.conf` (reads stay +filtered, but only reclamation stops; the setting does not disable `EXPIRE ROWS` +itself). Cleanup settings are read at startup, so changing this property or the +minimum expired fraction requires a restart. A failing sweep retries after one +second, doubling the per-view retry gap up to a 10-minute cap. + +To observe reclamation, compare the physical row count per partition before and +after a sweep: + +```questdb-sql title="Physical rows still on disk per partition" +SELECT name, numRows FROM table_partitions('trades_recent'); +``` + +Use a view whose `expire_enforcement` is `FILTER_AND_RECLAIM`, such as +`trades_recent`, for this check. Its active partition remains unchanged after a +sweep. Insert data into a newer partition before expecting the current active +partition to become eligible for reclamation. + +Reclamation **defers while a view is being refreshed continuously** and resumes +on a quiet sweep. + +## Semantics + +### NULLs + +QuestDB comparisons use two-valued boolean semantics: a comparison against +`NULL` evaluates to `FALSE`, not `UNKNOWN`, and `EXPIRE ROWS WHEN` expires a row +only when the complete predicate evaluates to `TRUE`. The complete predicate +therefore determines whether a `NULL` row survives: + +- **A direct comparison such as `amount < 1.5`** is `FALSE` for a `NULL` amount, + so the policy keeps the row. +- **`NOT (amount >= 1.5)`** is `TRUE` for a `NULL` amount because the inner + comparison is `FALSE`, so the policy expires the row. Although this predicate + resembles `amount < 1.5`, the two differ for `NULL` values. +- **`amount != 1.5` and `amount IS NULL`** are also `TRUE` for a `NULL` amount, + so both expire the row. +- **`KEEP HIGHEST/LOWEST`** keeps a `NULL` because its comparison against the + group extreme is `FALSE`. +- **`KEEP LATEST`** uses the designated timestamp, which is never `NULL`. +- **`KEEP N` is the exception.** It ranks rows with `row_number()`, and QuestDB + has no `NULLS LAST`, so where a `NULL` lands is **type-dependent**: under + `DESC` a floating-point `NULL` (NaN) sorts first (kept while there is room in + `N`), while an integer/timestamp `NULL` sorts last (expired first). Use + `KEEP HIGHEST/LOWEST` (no `N`) when every `NULL` must be kept regardless of + type. + +### A `NULL` threshold is rejected + +A `WHEN` threshold that evaluates to a constant `NULL` expires nothing because +`timestamp < NULL` is never `TRUE`, so the policy would be inert. QuestDB +refuses it at `CREATE` and `ALTER` time rather than storing a view that silently +never reclaims: + +```questdb-sql title="Rejected: the threshold is NULL" +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN timestamp < CAST(NULL AS TIMESTAMP); +-- invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire +``` + +The check matters most where the `NULL` is not written down. QuestDB stores a +`NULL` `TIMESTAMP`, `LONG` or `INT` as a reserved value at the bottom of the +type's range, and integer arithmetic wraps silently when it overflows, so an +arithmetic threshold can land on that value: + +```questdb-sql title="Also rejected: arithmetic that overflows onto NULL" +-- LONG overflow +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN timestamp < 4611686018427387904 * 2; + +-- INT overflow, reached three orders of magnitude sooner +CREATE MATERIALIZED VIEW trades_recent AS ( + SELECT * FROM trades +) EXPIRE ROWS WHEN timestamp < 2147483647 + 1; +``` + +Only thresholds that are constant at definition time are checked this way. One +built from a clock, such as `timestamp < now() - 3600000000`, is evaluated per +read and cannot be checked in advance. + +### Ties and determinism + +`KEEP HIGHEST/LOWEST` keeps **all** rows tied at the max/min, making the result +deterministic by construction. `KEEP N` makes the order total by appending the +designated timestamp as a tiebreak, so the N-th boundary is deterministic (pair +the base table with [`DEDUP UPSERT KEYS`](/docs/concepts/deduplication/) if +`(col, timestamp)` is not already unique). + +### Combining with TTL + +A view can carry a [TTL](/docs/concepts/ttl/) and an `EXPIRE ROWS` policy at the +same time, and the order is fixed: **TTL removes rows from the view first, then +the policy applies to the rows that stay.** TTL drops whole partitions from the +view's own storage as they age out, and the keep-set is computed over what +remains. + +```questdb-sql title="Highest price per symbol, over a 3-day window" +CREATE MATERIALIZED VIEW trades_peak_3d AS ( + SELECT * FROM trades +) PARTITION BY DAY TTL 3 DAYS + EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +`TTL` goes before `EXPIRE ROWS` in the statement, as it does after any +`PARTITION BY`. + +This view reports the highest price of the **last three days**, so its answer +can go **down** as the window moves: when the day holding a symbol's maximum +ages out, the next-highest price within the window takes over. That is what the +two clauses ask for together. The view is no longer "the highest price ever"; +it is "the highest price still retained". The base table is unaffected; it keeps +whatever its own retention settings keep. + +TTL is also the only control that bounds the size of a `KEEP LATEST`, +`KEEP HIGHEST/LOWEST` or `KEEP N` view, since the cleanup job never reclaims +disk for those modes. + +### Monotonicity and cleanup safety + +Physical deletion is only safe when expiry is **monotonic**: a row that is +expired now must stay expired forever. Two separate things can break that. + +The relative and window modes (`KEEP LATEST`, `KEEP HIGHEST/LOWEST`, `KEEP N`, +window `WHEN`) decide each row's fate by comparing it against the other rows in +the view. A later refresh can remove or replace the row a key currently keeps, +which promotes an older row back into the keep-set. For this reason, the cleanup +job never deletes for these modes, whatever their predicate looks like. + +A scalar `WHEN predicate` judges each row on its own, so it is eligible. It is +arbitrary SQL. QuestDB recognizes `now()`, `now_ns()`, `sysdate()`, +`systimestamp()` and `systimestamp_ns()` as wall-clock functions, and gives each +the same monotonicity proof. The cleanup job reclaims disk only for predicates +it can **prove** monotonic: + +- clock-free predicates (`WHEN amount < 1.5`), and +- designated-timestamp thresholds of a proven advancing-clock shape: a bare + clock (for example, `timestamp < now()`), a bare clock minus a non-negative + constant (for example, `timestamp < now() - 7200000000`), or a fixed-unit + look-back `dateadd` on a bare clock (for example, + `timestamp < dateadd('d', -1, now())`, with units `s`/`m`/`h`/`d`/`w` and + finer). + +Anything else **skips cleanup**: calendar units such as +`dateadd('M', -1, now())` (a month is a variable amount), look-forward offsets +(`dateadd('h', 1, now())`), +further clock arithmetic, non-constant offsets, and arbitrary window `WHEN` +predicates. A skipped policy stays correct at read time (the filter recomputes +on every read), but its disk is not reclaimed until the policy is changed to a +proven shape. + +:::warning + +A non-monotonic predicate such as `WHEN timestamp > now()` expires *future* +rows that **un-expire** as `now()` advances. The read filter recomputes `now()` +on every read and stays correct, and the cleanup job skips such a policy rather +than risk physically deleting a row a later read must show. The tradeoff is that +its disk is never reclaimed. Write `WHEN` predicates that expire things in the +**past** or against fixed thresholds, never rows that the passage of time will +later keep. + +::: + +## Inspecting a policy + +`SHOW CREATE MATERIALIZED VIEW` renders the policy in replayable DDL. It omits +`CLEANUP EVERY` when the cadence is the default `1h` and includes it for a +non-default cadence: + +```questdb-sql +SHOW CREATE MATERIALIZED VIEW trades_latest; +-- ... EXPIRE ROWS KEEP LATEST PARTITION BY symbol +``` + +The [`materialized_views()`](/docs/query/functions/meta/) function exposes the +policy in the `expire_clause`, `expire_cleanup_every` and `expire_enforcement` +columns (all `NULL` when no policy is set): + +```questdb-sql title="List EXPIRE ROWS policies" +SELECT view_name, expire_clause, expire_cleanup_every, expire_enforcement +FROM materialized_views(); +``` + +| view_name | expire_clause | expire_cleanup_every | expire_enforcement | +| ------------- | ------------------------------- | -------------------- | ------------------ | +| trades_sized | amount < 1.5 | 1h | FILTER_AND_RECLAIM | +| trades_latest | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | +| trades_top2 | KEEP 2 HIGHEST price ... | 1h | FILTER_ONLY | + +`expire_enforcement` is the verdict the cleanup job acts on: + +- `FILTER_AND_RECLAIM`: reads hide the expired rows and the job deletes them + from disk. +- `FILTER_ONLY`: reads hide the expired rows and they stay on disk. Every + relative and window policy reports this, as does a `WHEN` predicate that + cannot be proven monotonic. + +## Changing or removing a policy + +Set, change, or drop a policy on an existing passthrough view. See +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/): + +```questdb-sql +-- set or replace the policy +ALTER MATERIALIZED VIEW trades_latest SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; + +-- remove it (keeps all rows again) +ALTER MATERIALIZED VIEW trades_latest DROP EXPIRE; +``` + +`SET EXPIRE ROWS` validates the new policy against the view's columns before +applying it, so an invalid predicate or an unknown column is rejected up front +rather than breaking subsequent reads. + +## Limitations and operational notes + +- **Reads recompute the keep-set.** A relative/window policy computes its + keep-set over the whole physical view on every read. `KEEP LATEST` on an + [indexed](/docs/concepts/deep-dive/indexes/) symbol key is cheap; the window + modes (and non-indexed keep-latest) scan the view. +- **Cleanup tuning applies only to reclaiming policies.** For a monotonic scalar + `WHEN` policy that reports `FILTER_AND_RECLAIM`, a tighter `CLEANUP EVERY` + reduces how long expired rows remain in eligible non-active partitions. It has + no reclamation effect on relative or window policies that report + `FILTER_ONLY`. +- **Cleanup defers under continuous refresh.** Reclamation only proceeds when the + view is quiescent and fully applied, so a view being refreshed continuously + defers reclamation to a quiet sweep. The read filter stays authoritative + meanwhile. +- **`KEEP LATEST [ON timestamp]`.** The optional `ON timestamp` is accepted for + familiarity, but the view's designated timestamp is always used; naming a + different column is rejected. +- **Cleanup eligibility.** See + [monotonicity and cleanup safety](#monotonicity-and-cleanup-safety), and check + a view's verdict with `materialized_views().expire_enforcement`. +- **Reserved column name.** The window/keep modes compute the keep-set through a + synthetic boolean column named `__qdb_re_keep`; a policy is rejected on a view + that exposes a column with that name. +- **No line comments in the clause.** The clause text is stored verbatim and + embedded into generated SQL, so `--` comments are rejected inside an + `EXPIRE ROWS` clause; terminated block comments (`/* ... */`) are fine. +- **Compacting a Parquet partition rewrites it as native storage.** When cleanup + compacts a *partially*-expired partition held in Parquet, the partition + reverts to native QuestDB storage until the Parquet-conversion job re-converts + it. Reclamation correctness is unaffected. + +## Related documentation + +- [Materialized views](/docs/concepts/materialized-views/): the view type + `EXPIRE ROWS` runs on +- [Passthrough views](/docs/concepts/materialized-views/#passthrough-views): the + non-aggregating views `EXPIRE ROWS` applies to +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/): full create + syntax, including the `EXPIRE ROWS` clause +- [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/): + set, change, or drop a policy +- [Time To Live (TTL)](/docs/concepts/ttl/): partition-level retention by age +- [Storage policy](/docs/concepts/storage-policy/): graduated partition + lifecycle (Enterprise) diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index cf7ed4ac2..62c5397c5 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -12,6 +12,11 @@ their results at query time, materialized views persist their data to disk, making them particularly efficient for expensive aggregate queries that are run frequently. +Most materialized views aggregate, bucketing base rows with `SAMPLE BY` or a +time-based `GROUP BY`. A view can also be a +[passthrough view](#passthrough-views), which projects base rows one-for-one +instead of summarising them. + ## What are materialized views for? Let's say your application ingests trade data into a table like this: @@ -116,9 +121,10 @@ reads on a smaller, pre-aggregated dataset. ### Not suited for: data enrichment -Materialized views support JOINs, but `SAMPLE BY` (aggregation) is mandatory. -This means you can enrich aggregated results with data from other tables, but -you cannot keep raw (non-aggregated) rows while adding enrichment columns. +Materialized views support JOINs only in an aggregating query. A +[passthrough view](#passthrough-views) keeps raw rows, but it must read a single +table. So neither shape lets you keep raw rows while adding columns from another +table. For example, joining aggregated trades with instrument metadata works: @@ -157,6 +163,183 @@ refresh. Changes to joined tables do not trigger updates. **Coming soon**: We are actively developing a new type of materialized view that will support data enrichment use cases. Stay tuned for updates. +## Passthrough views + +Not every materialized view aggregates. A **passthrough view** projects base +rows one-for-one instead of bucketing them, so the view is a +continuously-maintained copy of its base table, optionally narrowed to a subset +of columns, a subset of rows, or both. Its query has no `SAMPLE BY` and no +time-based `GROUP BY`: + +```questdb-sql title="Passthrough view: a maintained, filtered copy of trades" +CREATE MATERIALIZED VIEW trades_btc AS ( + SELECT timestamp, symbol, price, amount + FROM trades + WHERE symbol = 'BTC' +); +``` + +Reach for one when you want a maintained *subset* of a large table rather than a +summary of it: + +- **A narrowed replica**: one symbol, one tenant, one region, or a handful of + columns out of a wide table, kept current automatically and queried without + the base table's scan cost. +- **A row-level retention target**: attach an + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only + some of the view's rows. The base table is left alone. + +Passthrough views refresh incrementally like any other materialized view, and +accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and +`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets +for a period to align to. + +### Use cases + +On its own, a passthrough view is a maintained copy of the base table. Add an +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a +maintained *subset*: you describe which rows are worth keeping, and QuestDB +keeps that set current as new data arrives. The three examples below are drawn +from sensor telemetry and from capital markets, and each keeps a different kind +of subset. + +#### IoT: The current reading from every sensor + +A building management platform records temperature and humidity from tens of +thousands of sensors, and its operations screen shows the newest reading from +each one. + +Sensors report at their own pace. Some send a reading every second, others go +quiet for days. Against the base table that screen runs +`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it has found +a row for even the quietest sensor, and over a long history that is most of the +table. + +A view that keeps only the newest row per sensor answers the same question from +a handful of rows: + +```questdb-sql title="Latest reading per sensor" +CREATE MATERIALIZED VIEW sensor_current AS ( + SELECT * FROM sensor_readings +) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id; +``` + +`sensor_current` holds one row per `sensor_id` and moves forward on its own as +readings arrive. Superseded rows stop appearing in queries but stay on disk, so +the view keeps growing at the same rate as the base table. Give it a +[TTL](/docs/concepts/ttl/) to cap its size. + +#### Finance: Options that have not expired yet + +A market maker quotes an options chain where contracts expire every Friday, and +the pricing screen must never show a contract that has already expired. + +That rule cannot live in the view's query. The query runs when rows are written +into the view, and it may not call `now()`, so there is no way to say "expiry is +still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` predicate is +evaluated on every read, which is what this case needs: + +```questdb-sql title="Options that have not expired yet" +CREATE MATERIALIZED VIEW options_live AS ( + SELECT * FROM options_quotes +) EXPIRE ROWS WHEN expiry < now(); +``` + +Contracts leave `options_live` as their expiry passes. There is no job to +schedule and nothing to re-create. + +One caveat, about disk rather than about results. QuestDB deletes expired rows +only when it can tell that a row, once expired, can never qualify again. It can +tell that for a cutoff on the view's designated timestamp, which only moves +forward. `expiry` is a different column, so QuestDB takes the safe route: +expired contracts stop appearing in queries straight away, but their rows stay +on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see +[when expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety). + +#### Finance: The largest trades per symbol + +A surveillance desk watches for block trades and wants the ten biggest prints +for every instrument on hand at all times. + +A trade is a fact that does not change once it has happened, so "the ten biggest +so far" is a set that only gets refined as larger trades arrive. That is what a +top-N policy maintains: + +```questdb-sql title="Ten largest trades per symbol" +CREATE MATERIALIZED VIEW trades_largest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol; +``` + +`trades_largest` holds ten rows per symbol however far the base table grows, and +the desk reads it directly instead of ranking the base table on every query. +When an eleventh large trade arrives, the smallest of the ten drops out. Trades +tied on `amount` at the tenth place are separated by the designated timestamp, +with the newer one staying. + +Ten rows per symbol are visible, but the view still stores every base row it has +taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/) +is what bounds that, at the price of changing the question the view answers from +"the biggest so far" to "the biggest still retained". See +[combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). + +The ranking covers everything the view holds rather than a recent window, so +`KEEP N` fits records that stay true once written. For a value that gets +superseded later, such as a resting order that is then cancelled, `KEEP LATEST` +is the mode that tracks the current version. + +### What a passthrough view inherits + +A passthrough view takes its shape from the base table instead of from a +`SAMPLE BY` clause: + +- **Designated timestamp and partitioning** come from the base table. The + projection has to keep the designated timestamp; a query that drops it is + rejected with `materialized view query is required to have designated + timestamp`. `PARTITION BY` and `TTL` can still be stated explicitly. +- **Symbol indexes are inherited.** A base column declared `SYMBOL INDEX` stays + indexed in the view, under whatever alias the projection gives it, so an + indexed lookup on the view costs what it costs on the base. An aggregating + view never inherits an index, because its rows are not base rows. + +### Which queries are passthrough + +The rule is that view rows stay 1:1 with base rows. A projection over a single +table qualifies, with or without a filter: + +| Query | | +| ----- | --- | +| `SELECT * FROM trades` | Passthrough | +| `SELECT timestamp, symbol, price FROM trades` | Passthrough: column subset | +| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough: aliases are fine | +| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough: a row-local expression | +| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough: a filter only removes rows | +| `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view | +| `SELECT DISTINCT ...` | Rejected | +| `... LATEST ON timestamp PARTITION BY symbol` | Rejected | +| `JOIN`, `UNION` | Rejected | +| `row_number() OVER (...)` and other window functions | Rejected | +| `LIMIT` | Rejected | +| `ORDER BY` a non-timestamp column | Rejected: the view loses its designated timestamp | + +Everything in the rejected group produces output that depends on rows *other +than* the one being emitted. An incremental refresh sees only the newly-arrived +slice of the base table, so it cannot compute those correctly. A `LIMIT 100` +would admit 100 rows per refresh rather than 100 in total, and a `row_number()` +would restart within each slice. + +A query that is neither passthrough nor aggregating is rejected at creation +time. Most report `materialized view query requires a sampling interval, use +SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to +aggregate but named no bucket. `LIMIT` and window functions have their own +messages. + +To keep only some of a passthrough view's rows over time, such as the latest per +key, the top-N per group, or rows matching a predicate, attach an +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also +covers +[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows). + ## Creating a materialized view ### Basic syntax @@ -214,7 +397,9 @@ interval: ### The query -Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query. +An aggregating materialized view uses a `SAMPLE BY` or time-based `GROUP BY` +query. (The other shape is a [passthrough view](#passthrough-views), which does +not aggregate; the rules below apply to aggregating views.) **Supported:** @@ -526,7 +711,9 @@ rows. Materialized view queries: -- Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column +- Must either aggregate with `SAMPLE BY` / `GROUP BY` on a designated timestamp + column, or be a [passthrough](#passthrough-views) projection over a single + table - Must not use `FROM-TO`, `FILL`, or `ALIGN TO FIRST OBSERVATION` - Must not use non-deterministic functions (`now()`, `rnd_uuid4()`) - Must use join conditions compatible with incremental refresh diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index 9c1ee9f40..4a9040b7f 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -89,6 +89,8 @@ A view's TTL is independent of its base table's TTL. For full syntax, see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#ttl-time-to-live) and [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/). +For row-level retention on passthrough materialized views, such as keeping the +latest row per key, see [`EXPIRE ROWS`](/docs/concepts/expire-rows/). ## How TTL works diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index 49f73b12c..4d22ddc6b 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -3,9 +3,9 @@ title: Materialized views description: Configuration settings for materialized views in QuestDB. --- -These settings control materialized view SQL support and the background refresh -job. Materialized views can use dedicated worker threads or share the server's -common pool. +These settings control materialized view SQL support, background refresh, and +row-expiry cleanup. Materialized views can use dedicated worker threads or share +the server's common pool. ## cairo.mat.view.enabled @@ -22,6 +22,25 @@ Enables or disables SQL support and the refresh job for materialized views. When disabled, SQL executed by the materialized view refresh job always runs single-threaded. +## cairo.mat.view.row.expiry.cleanup.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables the background job that reclaims rows removed by an eligible +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Disabling the job does not +disable read filtering, so expired rows remain hidden from query results. + +## cairo.mat.view.row.expiry.cleanup.min.expired.fraction + +- **Default**: `0.5` +- **Reloadable**: no + +Minimum fraction of expired rows required before the background cleanup job +compacts a partially expired partition. Set this property to `0` to compact on +the first expired row, or to `1` to disable partial-partition compaction. Fully +expired partitions are still removed. + ## mat.view.refresh.worker.affinity - **Default**: equal to the CPU core count diff --git a/documentation/operations/data-retention.md b/documentation/operations/data-retention.md index cb2778690..b35fe7f96 100644 --- a/documentation/operations/data-retention.md +++ b/documentation/operations/data-retention.md @@ -8,14 +8,21 @@ description: How to employ a data retention strategy to delete old data and save The nature of [time-series data](/blog/what-is-time-series-data/) is that the relevance of information diminishes over time. If stale data is no longer required, users can delete old data from QuestDB to either save disk space or adhere to a data retention policy. This is -achieved in QuestDB by removing data partitions from a table. +achieved by removing or tiering table partitions, or by applying row-level +retention to a materialized view. -QuestDB offers four approaches for data retention: +QuestDB offers five approaches for data retention: - **TTL** _(automatic, open source)_: Use [Time To Live (TTL)](/docs/concepts/ttl/) to automatically drop partitions when data ages beyond a specified threshold. This is the simplest approach and is available in both open source and Enterprise editions. +- **EXPIRE ROWS** _(automatic, materialized views only)_: Use + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) for row-granular retention on a + materialized view, such as keeping the latest row per key, the top-N rows per + group, or expiring rows selected by a predicate. The base table is unaffected. + Some policies hide expired rows without reclaiming their disk space, so check + the policy's enforcement mode when storage reduction matters. - **Storage policy** _(automatic, Enterprise only)_: Use a [storage policy](/docs/concepts/storage-policy/) to automate the partition lifecycle — convert to Parquet locally and drop old data on a schedule. This is @@ -30,8 +37,9 @@ QuestDB offers four approaches for data retention: - **Manual**: Use `DROP PARTITION` commands as described on this page for explicit control over which partitions to remove and when. -This page provides a high-level overview of partitioning with examples to drop -data by date. For more details on partitioning, see the +This page summarizes the available retention approaches, then provides +partitioning examples that drop data by date. For more details on partitioning, +see the [partitioning](/docs/concepts/partitions/) page. ## Manual partition management @@ -109,8 +117,9 @@ syntax. Partitions may be dropped by: chronologically. Depending on the types of queries users are performing on a dataset, it may not be desirable to have gaps caused by dropped partitions. - Unlike TTL, `DROP PARTITION` commands must be triggered manually or via - external scheduling (e.g., cron jobs). For fully automated retention, consider - using [TTL](/docs/concepts/ttl/) instead. + external scheduling (e.g., cron jobs). For automated partition retention, + consider [TTL](/docs/concepts/ttl/) or a + [storage policy](/docs/concepts/storage-policy/) instead. ### Example diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 6152826dd..b489cabf0 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -262,26 +262,58 @@ Returns a `table` including the following information: - `view_table_dir_name` - view directory name - `invalidation_reason` - message explaining why the view was marked as invalid - `view_status` - view status: 'valid', 'refreshing', or 'invalid' +- `refresh_period_hi` - end of the last refreshed period, for a view defined + with a `REFRESH PERIOD` - `refresh_base_table_txn` - the last base table transaction used to refresh the materialized view - `base_table_txn` - the last committed transaction in the base table -- `refresh_limit_value` - how many units back in time the refresh limit goes +- `refresh_limit` - how many units back in time the refresh limit goes - `refresh_limit_unit` - how long each unit is +- `timer_time_zone` - time zone the refresh timer and period boundaries use - `timer_start` - start date for the scheduled refresh timer -- `timer_interval_value` - how many interval units between each refresh +- `timer_interval` - how many interval units between each refresh - `timer_interval_unit` - how long each unit is +- `period_length` - length of one refresh period (`PERIOD (LENGTH ...)`) +- `period_length_unit` - how long each period-length unit is +- `period_delay` - delay before a completed period is refreshed + (`PERIOD (DELAY ...)`) +- `period_delay_unit` - how long each period-delay unit is +- `refresh_avg_commit_nanos` - moving average of one refresh commit, in + nanoseconds. Held in memory only, so it resets on restart +- `refresh_avg_scan_sample_nanos` - moving average of one refresh iteration's + base-table scan, in nanoseconds. In memory only +- `refresh_avg_scan_range_ts_units` - moving average of the timestamp range one + refresh iteration covers, in the base table's timestamp unit. In memory only +- `refresh_gap_threshold_ts_units` - timestamp gap below which the refresh job + merges two adjacent intervals instead of paying for a second commit. `0` means + merging is disabled +- `expire_clause` - the view's + [`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy as written, or + `NULL` when the view has no policy +- `expire_cleanup_every` - how often the cleanup job runs for the policy, or + `NULL` +- `expire_enforcement` - `FILTER_AND_RECLAIM` when the cleanup job frees disk + for the policy, `FILTER_ONLY` when reads hide the expired rows but they stay + on disk, `NULL` when the view has no policy **Examples:** +`materialized_views()` on its own returns every column listed above. The example +below projects a readable subset over three views: an aggregating one with no +retention policy, and two passthrough views with an +[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. + ```questdb-sql title="List all materialized views" -materialized_views(); +SELECT view_name, view_status, base_table_name, refresh_base_table_txn, + base_table_txn, expire_clause, expire_cleanup_every, expire_enforcement +FROM materialized_views(); ``` -| view_name | refresh_type | base_table_name | last_refresh_start_timestamp | last_refresh_finish_timestamp | view_sql | view_table_dir_name | invalidation_reason | view_status | refresh_base_table_txn | base_table_txn | refresh_limit_value | refresh_limit_unit | timer_start | timer_interval_value | timer_interval_unit | -| ---------------- | ------------ | --------------- | ---------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | ------------------- | ----------- | ---------------------- | -------------- | ------------------- | ------------------ | ----------- | -------------------- | ------------------- | -| trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | -| trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | - +| view_name | view_status | base_table_name | refresh_base_table_txn | base_table_txn | expire_clause | expire_cleanup_every | expire_enforcement | +| --------------- | ----------- | --------------- | ---------------------- | -------------- | ----------------------------------- | -------------------- | ------------------ | +| trades_OHLC_15m | valid | trades | 1 | 1 | null | null | null | +| trades_recent | valid | trades | 1 | 1 | timestamp < dateadd('d', -7, now()) | 30m | FILTER_AND_RECLAIM | +| trades_latest | valid | trades | 1 | 1 | KEEP LATEST PARTITION BY symbol | 1h | FILTER_ONLY | ## memory_metrics @@ -758,20 +790,20 @@ Returns a `table` with the following columns: ### Basic table information -| Column | Type | Description | -| --------------------- | ------- | ------------------------------------------------------------------ | -| `id` | INT | Internal table ID | -| `table_name` | STRING | Table name | -| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | -| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | -| `dedup` | BOOLEAN | Whether deduplication is enabled | -| `ttlValue` | INT | TTL (Time-To-Live) value | -| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | -| `matView` | BOOLEAN | Whether this is a materialized view | -| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | -| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | -| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | +| Column | Type | Description | +| ---------------------- | ------- | ------------------------------------------------------------------ | +| `id` | INT | Internal table ID | +| `table_name` | STRING | Table name | +| `designatedTimestamp` | STRING | Name of the designated timestamp column, or `null` | +| `partitionBy` | STRING | Partition strategy: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `walEnabled` | BOOLEAN | Whether WAL (Write-Ahead Log) is enabled | +| `dedup` | BOOLEAN | Whether deduplication is enabled | +| `ttlValue` | INT | TTL (Time-To-Live) value | +| `ttlUnit` | STRING | TTL unit: `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR` | +| `matView` | BOOLEAN | Whether this is a materialized view | +| `directoryName` | STRING | Directory name on disk (includes ` (->)` suffix for symlinks) | +| `maxUncommittedRows` | INT | Table's `maxUncommittedRows` setting | +| `o3MaxLag` | LONG | Table's `o3MaxLag` setting in microseconds | :::note @@ -841,6 +873,17 @@ These columns are populated on **replicas only** via replication download tracki On primary instances, these columns will be `0` or `false`. +### Row-expiry policy + +| Column | Type | Description | +| ---------------------- | ------ | -------------------------------------------------------------- | +| `expire_clause` | STRING | Materialized view's `EXPIRE ROWS` policy as written, or `null` | +| `expire_cleanup_every` | STRING | Cleanup cadence for the policy, or `null` | + +Unlike `materialized_views()`, `tables()` does not expose +`expire_enforcement`; use `materialized_views()` to inspect whether a policy is +`FILTER_AND_RECLAIM` or `FILTER_ONLY`. + ### Data precision and limitations These values are approximations, not precise real-time metrics: diff --git a/documentation/query/sql/alter-mat-view-set-expire.md b/documentation/query/sql/alter-mat-view-set-expire.md new file mode 100644 index 000000000..56d53c654 --- /dev/null +++ b/documentation/query/sql/alter-mat-view-set-expire.md @@ -0,0 +1,136 @@ +--- +title: ALTER MATERIALIZED VIEW SET EXPIRE +sidebar_label: SET EXPIRE +description: + ALTER MATERIALIZED VIEW SET EXPIRE ROWS / DROP EXPIRE SQL keyword reference + documentation. +--- + +Sets, replaces, or removes an [`EXPIRE ROWS`](/docs/concepts/expire-rows/) +row-retention policy on a materialized view. It is designed for +**passthrough views**; see the concept page. For filtering and disk-reclamation +behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). + +## Syntax + +``` +ALTER MATERIALIZED VIEW viewName SET EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] + +ALTER MATERIALIZED VIEW viewName DROP EXPIRE +``` + +## Parameters + +| Parameter | Description | +| ---------------- | --------------------------------------------------------------------------------- | +| `viewName` | Name of the passthrough materialized view to modify | +| `WHEN predicate` | Per-row (or window) predicate; a row expires when it evaluates `TRUE` | +| `KEEP LATEST` | Keep the latest row per `PARTITION BY` key, by the designated timestamp | +| `KEEP [N] HIGHEST\|LOWEST col` | Keep the rows at the max/min of `col` per group, or the top `N` | +| `CLEANUP EVERY` | Background reclamation cadence as ``, where `unit` is `s`, `m`, `h`, `d`, or `w`. Defaults to `1h` if omitted | + +Without `N`, the keep column must be `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, +`DOUBLE`, `DATE`, `TIMESTAMP` or `DECIMAL`. `KEEP N HIGHEST/LOWEST` ranks with +`ORDER BY` and accepts any orderable column type. + +For the full description of each mode and its semantics, see the +[Expiring rows](/docs/concepts/expire-rows/) concept page. + +## When to use + +- Add a retention policy to a passthrough view created without one +- Switch a view between modes (e.g. from a value predicate to `KEEP LATEST`) +- Tune the `CLEANUP EVERY` cadence +- Remove a policy with `DROP EXPIRE` so the view keeps all rows again + +## How it works + +`SET EXPIRE ROWS` validates the new policy against the view's columns first +(compiling the predicate / checking the key columns), so an invalid predicate or +an unknown column is rejected immediately rather than breaking later reads. Once +set, the policy takes effect without rebuilding the view. See +[How it works](/docs/concepts/expire-rows/#how-it-works). + +## Examples + +These examples use `trades_mirror`, a passthrough materialized view over a +`trades` table with `symbol`, `side`, `price`, `amount`, and designated +`timestamp` columns: + +```questdb-sql title="Create the passthrough view" +CREATE MATERIALIZED VIEW trades_mirror AS (SELECT * FROM trades); +``` + +```questdb-sql title="Rolling 7-day window, with a tighter cleanup cadence" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS WHEN timestamp < dateadd('d', -7, now()) CLEANUP EVERY 30m; +``` + +A `WHEN` predicate is the right tool for a cutoff that moves with the clock like +this one. A deterministic predicate such as `amount < 1.5` is accepted too, but +it selects the same rows more cheaply as a `WHERE` clause in the view's defining +query. See +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). + +```questdb-sql title="Keep the latest row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +```questdb-sql title="Keep the highest-priced row per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Keep the 2 highest-priced rows per symbol" +ALTER MATERIALIZED VIEW trades_mirror + SET EXPIRE ROWS KEEP 2 HIGHEST price PARTITION BY symbol; +``` + +```questdb-sql title="Remove the policy" +ALTER MATERIALIZED VIEW trades_mirror DROP EXPIRE; +``` + +## Behavior + +| Aspect | Description | +| ----------------------- | ---------------------------------------------------------------------------- | +| Passthrough recommended | An aggregating view is accepted with a logged advisory: a later refresh can regenerate reclaimed rows, so align base-table retention with the expiry horizon | +| No dependent views | Rejected when other materialized views derive from this view (they would copy expired rows on refresh) | +| Validation | The policy is checked against the view's columns before it is applied | +| Replication | The policy and the reclamation it drives replicate as normal WAL traffic | + +## Permissions (Enterprise) + +Changing the policy requires the `ALTER MATERIALIZED VIEW` permission: + +```questdb-sql title="Grant alter permission" +GRANT ALTER MATERIALIZED VIEW ON trades_mirror TO user1; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `materialized view does not exist` | View with the specified name doesn't exist | +| `cannot set an EXPIRE ROWS policy on '...': it is the base of N view(s), including '', which would copy expired rows on refresh` | Other views derive from this view; the message names the first dependent | +| `EXPIRE ROWS KEEP LATEST ON must name the designated timestamp ...` | `ON` names a column other than the designated timestamp | +| `invalid EXPIRE ROWS KEEP LATEST PARTITION BY column: ...` | A `PARTITION BY` key column does not exist | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires a BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP or DECIMAL column, but '' is ; use KEEP HIGHEST/LOWEST to rank an orderable column of any type` | The bare `KEEP HIGHEST/LOWEST` form was given an unsupported column type; use a supported numeric/date type or the top-N form | +| `EXPIRE ROWS KEEP HIGHEST/LOWEST requires an orderable column, but '' is ` | The top-N form was given a column type that cannot be ordered | +| `EXPIRE ROWS KEEP / window retention cannot be used on a view with a column named '__qdb_re_keep'` | The view exposes a column named like the reserved keep column | +| `invalid EXPIRE ROWS predicate: ...` | The predicate does not parse, bind, or type-check against the view's columns | +| `invalid EXPIRE ROWS predicate: the threshold is NULL, so no row can ever expire` | A `WHEN` threshold that is constant at definition time evaluates to `NULL`, e.g. `CAST(NULL AS TIMESTAMP)` or arithmetic that overflows | +| `permission denied` | Missing `ALTER MATERIALIZED VIEW` permission (Enterprise) | + +## See also + +- [Expiring rows (EXPIRE ROWS) concept](/docs/concepts/expire-rows/) +- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) +- [Materialized views concept](/docs/concepts/materialized-views/) +- [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 0dd973953..efabfc9ed 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -18,17 +18,31 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] viewName [ START timestamp ] [ TIME ZONE timezone ] [ PERIOD ( LENGTH length [ TIME ZONE tz ] [ DELAY delay ] ) ] [ PERIOD ( SAMPLE BY INTERVAL ) ] ] -AS [ ( ] query [ ) ] -[ TIMESTAMP ( columnRef ) ] -[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) - [ TTL n timeUnit ] ] -[ OWNED BY ownerName ] +AS +{ query +| ( query ) + [ TIMESTAMP ( columnRef ) ] + [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) + [ TTL n timeUnit ] ] + [ EXPIRE ROWS expirePolicy [ CLEANUP EVERY duration ] ] + [ OWNED BY ownerName ] +} ``` +Parentheses around `query` are optional only when the query ends the statement. +They are required when any trailing clause follows the query, including +`TIMESTAMP`, `PARTITION BY`, `TTL`, `EXPIRE ROWS`, or `OWNED BY`. + Where: + - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` +- `query`: Either an aggregating query with `SAMPLE BY` or a time-based + `GROUP BY`, or a + [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection + over a single table +- `expirePolicy`: `WHEN predicate | KEEP LATEST [ON timestamp] PARTITION BY cols | KEEP [N] (HIGHEST|LOWEST) col [PARTITION BY cols]`. + This policy is designed for passthrough views (see below). ## Parameters @@ -39,17 +53,18 @@ Where: | `WITH BASE` | Specify base table (required for JOINs) | | `REFRESH` | Refresh strategy (default: `IMMEDIATE`) | | `DEFERRED` | Skip initial refresh on creation | -| `query` | A `SAMPLE BY` or time-based `GROUP BY` query | +| `query` | An aggregating (`SAMPLE BY` / time-based `GROUP BY`) or [passthrough](/docs/concepts/materialized-views/#passthrough-views) query | | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | +| `EXPIRE ROWS` | Row-level retention for passthrough views (see below) | | `OWNED BY` | Assign ownership (Enterprise) | ## Rules and defaults | Rule | Description | | ---- | ----------- | -| Query must aggregate | Requires `SAMPLE BY` or `GROUP BY` with designated timestamp | +| Query must aggregate or be passthrough | Either `SAMPLE BY` / `GROUP BY` with a designated timestamp, or a 1:1 [passthrough](/docs/concepts/materialized-views/#passthrough-views) projection over a single table | | Default refresh | `IMMEDIATE` (refreshes after each base table transaction) | | WITH BASE required | Must specify when query contains JOINs | | PARTITION BY sizing | Should be larger than or equal to `SAMPLE BY` interval | @@ -69,10 +84,11 @@ Where: ```questdb-sql title="Base table" CREATE TABLE trades ( - timestamp TIMESTAMP, symbol SYMBOL, + side SYMBOL, price DOUBLE, - amount DOUBLE + amount DOUBLE, + timestamp TIMESTAMP ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` @@ -278,6 +294,62 @@ Time units: `HOURS`, `DAYS`, `WEEKS`, `MONTHS`, `YEARS` The view's TTL is independent of the base table's TTL. See [TTL documentation](/docs/concepts/ttl/) for details. +## EXPIRE ROWS + +Attach a row-level retention policy with `EXPIRE ROWS`. Unlike `TTL` (which drops +whole partitions by age), `EXPIRE ROWS` keeps a defined set of rows: the latest +per key, the top-N per group, or rows matching a predicate. It recomputes that +set continuously as the view refreshes. + +`EXPIRE ROWS` is designed for +[**passthrough (non-aggregating) views**](/docs/concepts/materialized-views/#passthrough-views) +(a projection over a single table with no `SAMPLE BY` / `GROUP BY`, whose rows +stay 1:1 with the base). An aggregating view is accepted with a logged advisory +(a later refresh can regenerate reclaimed rows). The defining query must not +read another policied view, as its base or in a join: + +```questdb-sql title="Passthrough view that keeps the latest row per symbol" +CREATE MATERIALIZED VIEW trades_latest AS ( + SELECT * FROM trades +) EXPIRE ROWS KEEP LATEST PARTITION BY symbol; +``` + +A `WHEN` predicate is for rules that move with **wall-clock time**, such as a +rolling `timestamp < dateadd('d', -7, now())` window. The defining query cannot +express those, because it rejects non-deterministic functions. A predicate that +depends only on the row's own values belongs in the query's `WHERE` clause +instead, which keeps those rows out of the view entirely; see +[`WHERE` filter or `EXPIRE ROWS`?](/docs/concepts/expire-rows/#where-filter-or-expire-rows). + +The clause goes after the query (and after `PARTITION BY` if present): + +``` +EXPIRE ROWS + { WHEN predicate + | KEEP LATEST [ ON timestampColumn ] PARTITION BY col [, col ...] + | KEEP [ N ] ( HIGHEST | LOWEST ) col [ PARTITION BY col [, col ...] ] } + [ CLEANUP EVERY duration ] +``` + +A `WHEN` threshold that is constant at definition time and evaluates to `NULL` +is rejected, since it would expire nothing. That covers the explicit +`timestamp < CAST(NULL AS TIMESTAMP)` and arithmetic that overflows onto the +reserved `NULL` value, such as `timestamp < 2147483647 + 1`. See +[A `NULL` threshold is rejected](/docs/concepts/expire-rows/#a-null-threshold-is-rejected). + +For filtering and disk-reclamation behavior, see +[How `EXPIRE ROWS` works](/docs/concepts/expire-rows/#how-it-works). Change or +remove a policy with +[`ALTER MATERIALIZED VIEW SET EXPIRE`](/docs/query/sql/alter-mat-view-set-expire/). + +A view can carry both `TTL` and `EXPIRE ROWS`. `TTL` comes first in the +statement and first in effect: it removes rows from the view, and the +`EXPIRE ROWS` policy then applies to the rows that stay. See +[Combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl). + +See the [Expiring rows](/docs/concepts/expire-rows/) concept page for +all modes, worked examples, and semantics (NULLs, ties, monotonicity). + ## Complete example Putting it all together: diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 23e0f298b..197d1df8b 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -41,7 +41,7 @@ SHOW { COLUMNS FROM tableName - `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live view. - `SHOW CREATE MATERIALIZED VIEW` returns a DDL query that allows you to - recreate a materialized view. + recreate a materialized view, including its retention clauses. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. - `SHOW GROUPS` shows all groups the user belongs or all groups in the system @@ -225,7 +225,9 @@ SHOW CREATE MATERIALIZED VIEW bbo_1s; | CREATE MATERIALIZED VIEW 'bbo_1s' WITH BASE 'market_data' REFRESH IMMEDIATE AS (SELECT timestamp, symbol, last(bids[1][1]) AS bid, last(asks[1][1]) AS ask FROM market_data SAMPLE BY 1s) PARTITION BY DAY; | This returns the `CREATE MATERIALIZED VIEW` statement that would recreate the -materialized view, including its base table, refresh strategy, and partitioning. +materialized view, including its base table, refresh strategy, partitioning, +`TTL`, and `EXPIRE ROWS` policy. The output includes `CLEANUP EVERY` only when +the cadence differs from the default `1h`. ### SHOW CREATE TABLE diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 84ffb0548..23ae94729 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -343,6 +343,7 @@ module.exports = { "query/sql/alter-mat-view-set-refresh", "query/sql/alter-mat-view-set-refresh-limit", "query/sql/alter-mat-view-set-ttl", + "query/sql/alter-mat-view-set-expire", ], }, { @@ -626,6 +627,7 @@ module.exports = { "concepts/deduplication", "concepts/delivery-semantics", "concepts/ttl", + "concepts/expire-rows", "concepts/storage-policy", { id: "concepts/cold-storage",