Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation.

### New

- [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - New row-level retention for materialized views, with creation syntax in [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/), immediate read filtering, and background reclamation for eligible policies
- [Cold storage](/docs/concepts/cold-storage/) - Historical partitions move to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store as Parquet and stay queryable with normal SQL, covering the partition lifecycle, the range-read path, the immutability boundary at `DROP LOCAL`, and the manager and refresher roles
- [Operating cold storage](/docs/operations/cold-storage/) - Setup guide and runbook: preparing the object store prefix, configuring the cluster, the two-step manager handoff, remote garbage collection, cold-read metrics, backup interaction, and a troubleshooting table
- [Cold storage configuration](/docs/configuration/cold-storage/) - Every `cold.storage.*` setting, covering the store connection, roles, upload, read path, garbage collection, and request timeouts
Expand Down
695 changes: 695 additions & 0 deletions documentation/concepts/expire-rows.md

Large diffs are not rendered by default.

197 changes: 192 additions & 5 deletions documentation/concepts/materialized-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ their results at query time, materialized views persist their data to disk,
making them particularly efficient for expensive aggregate queries that are run
frequently.

Most materialized views aggregate, bucketing base rows with `SAMPLE BY` or a
time-based `GROUP BY`. A view can also be a
[passthrough view](#passthrough-views), which projects base rows one-for-one
instead of summarising them.

## What are materialized views for?

Let's say your application ingests trade data into a table like this:
Expand Down Expand Up @@ -116,9 +121,10 @@ reads on a smaller, pre-aggregated dataset.

### Not suited for: data enrichment

Materialized views support JOINs, but `SAMPLE BY` (aggregation) is mandatory.
This means you can enrich aggregated results with data from other tables, but
you cannot keep raw (non-aggregated) rows while adding enrichment columns.
Materialized views support JOINs only in an aggregating query. A
[passthrough view](#passthrough-views) keeps raw rows, but it must read a single
table. So neither shape lets you keep raw rows while adding columns from another
table.

For example, joining aggregated trades with instrument metadata works:

Expand Down Expand Up @@ -157,6 +163,183 @@ refresh. Changes to joined tables do not trigger updates.
**Coming soon**: We are actively developing a new type of materialized view that
will support data enrichment use cases. Stay tuned for updates.

## Passthrough views

Not every materialized view aggregates. A **passthrough view** projects base
rows one-for-one instead of bucketing them, so the view is a
continuously-maintained copy of its base table, optionally narrowed to a subset
of columns, a subset of rows, or both. Its query has no `SAMPLE BY` and no
time-based `GROUP BY`:

```questdb-sql title="Passthrough view: a maintained, filtered copy of trades"
CREATE MATERIALIZED VIEW trades_btc AS (
SELECT timestamp, symbol, price, amount
FROM trades
WHERE symbol = 'BTC'
);
```

Reach for one when you want a maintained *subset* of a large table rather than a
summary of it:

- **A narrowed replica**: one symbol, one tenant, one region, or a handful of
columns out of a wide table, kept current automatically and queried without
the base table's scan cost.
- **A row-level retention target**: attach an
[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy to keep only
some of the view's rows. The base table is left alone.

Passthrough views refresh incrementally like any other materialized view, and
accept the same `REFRESH IMMEDIATE` (the default), `REFRESH MANUAL` and
`REFRESH EVERY` strategies. `REFRESH PERIOD` is rejected: there are no buckets
for a period to align to.

### Use cases

On its own, a passthrough view is a maintained copy of the base table. Add an
[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy and it becomes a
maintained *subset*: you describe which rows are worth keeping, and QuestDB
keeps that set current as new data arrives. The three examples below are drawn
from sensor telemetry and from capital markets, and each keeps a different kind
of subset.

#### IoT: The current reading from every sensor

A building management platform records temperature and humidity from tens of
thousands of sensors, and its operations screen shows the newest reading from
each one.

Sensors report at their own pace. Some send a reading every second, others go
quiet for days. Against the base table that screen runs
`LATEST ON ts PARTITION BY sensor_id`, which reads backwards until it has found
a row for even the quietest sensor, and over a long history that is most of the
table.

A view that keeps only the newest row per sensor answers the same question from
a handful of rows:

```questdb-sql title="Latest reading per sensor"
CREATE MATERIALIZED VIEW sensor_current AS (
SELECT * FROM sensor_readings
) EXPIRE ROWS KEEP LATEST PARTITION BY sensor_id;
```

`sensor_current` holds one row per `sensor_id` and moves forward on its own as
readings arrive. Superseded rows stop appearing in queries but stay on disk, so
the view keeps growing at the same rate as the base table. Give it a
[TTL](/docs/concepts/ttl/) to cap its size.

#### Finance: Options that have not expired yet

A market maker quotes an options chain where contracts expire every Friday, and
the pricing screen must never show a contract that has already expired.

That rule cannot live in the view's query. The query runs when rows are written
into the view, and it may not call `now()`, so there is no way to say "expiry is
still in the future" in a `WHERE` clause. An `EXPIRE ROWS WHEN` predicate is
evaluated on every read, which is what this case needs:

```questdb-sql title="Options that have not expired yet"
CREATE MATERIALIZED VIEW options_live AS (
SELECT * FROM options_quotes
) EXPIRE ROWS WHEN expiry < now();
```

Contracts leave `options_live` as their expiry passes. There is no job to
schedule and nothing to re-create.

One caveat, about disk rather than about results. QuestDB deletes expired rows
only when it can tell that a row, once expired, can never qualify again. It can
tell that for a cutoff on the view's designated timestamp, which only moves
forward. `expiry` is a different column, so QuestDB takes the safe route:
expired contracts stop appearing in queries straight away, but their rows stay
on disk. Add a [TTL](/docs/concepts/ttl/) if you want that space back, and see
[when expired rows are deleted from disk](/docs/concepts/expire-rows/#monotonicity-and-cleanup-safety).

#### Finance: The largest trades per symbol

A surveillance desk watches for block trades and wants the ten biggest prints
for every instrument on hand at all times.

A trade is a fact that does not change once it has happened, so "the ten biggest
so far" is a set that only gets refined as larger trades arrive. That is what a
top-N policy maintains:

```questdb-sql title="Ten largest trades per symbol"
CREATE MATERIALIZED VIEW trades_largest AS (
SELECT * FROM trades
) EXPIRE ROWS KEEP 10 HIGHEST amount PARTITION BY symbol;
```

`trades_largest` holds ten rows per symbol however far the base table grows, and
the desk reads it directly instead of ranking the base table on every query.
When an eleventh large trade arrives, the smallest of the ten drops out. Trades
tied on `amount` at the tenth place are separated by the designated timestamp,
with the newer one staying.

Ten rows per symbol are visible, but the view still stores every base row it has
taken in, because a top-N policy never frees disk. A [TTL](/docs/concepts/ttl/)
is what bounds that, at the price of changing the question the view answers from
"the biggest so far" to "the biggest still retained". See
[combining with TTL](/docs/concepts/expire-rows/#combining-with-ttl).

The ranking covers everything the view holds rather than a recent window, so
`KEEP N` fits records that stay true once written. For a value that gets
superseded later, such as a resting order that is then cancelled, `KEEP LATEST`
is the mode that tracks the current version.

### What a passthrough view inherits

A passthrough view takes its shape from the base table instead of from a
`SAMPLE BY` clause:

- **Designated timestamp and partitioning** come from the base table. The
projection has to keep the designated timestamp; a query that drops it is
rejected with `materialized view query is required to have designated
timestamp`. `PARTITION BY` and `TTL` can still be stated explicitly.
- **Symbol indexes are inherited.** A base column declared `SYMBOL INDEX` stays
indexed in the view, under whatever alias the projection gives it, so an
indexed lookup on the view costs what it costs on the base. An aggregating
view never inherits an index, because its rows are not base rows.

### Which queries are passthrough

The rule is that view rows stay 1:1 with base rows. A projection over a single
table qualifies, with or without a filter:

| Query | |
| ----- | --- |
| `SELECT * FROM trades` | Passthrough |
| `SELECT timestamp, symbol, price FROM trades` | Passthrough: column subset |
| `SELECT timestamp, symbol AS ticker FROM trades` | Passthrough: aliases are fine |
| `SELECT timestamp, price * amount AS notional FROM trades` | Passthrough: a row-local expression |
| `SELECT * FROM trades WHERE symbol = 'BTC'` | Passthrough: a filter only removes rows |
| `... SAMPLE BY 1h`, or `GROUP BY` on a timestamp | Aggregating view |
| `SELECT DISTINCT ...` | Rejected |
| `... LATEST ON timestamp PARTITION BY symbol` | Rejected |
| `JOIN`, `UNION` | Rejected |
| `row_number() OVER (...)` and other window functions | Rejected |
| `LIMIT` | Rejected |
| `ORDER BY` a non-timestamp column | Rejected: the view loses its designated timestamp |

Everything in the rejected group produces output that depends on rows *other
than* the one being emitted. An incremental refresh sees only the newly-arrived
slice of the base table, so it cannot compute those correctly. A `LIMIT 100`
would admit 100 rows per refresh rather than 100 in total, and a `row_number()`
would restart within each slice.

A query that is neither passthrough nor aggregating is rejected at creation
time. Most report `materialized view query requires a sampling interval, use
SAMPLE BY or GROUP BY timestamp_floor()`: the query looked like it meant to
aggregate but named no bucket. `LIMIT` and window functions have their own
messages.

To keep only some of a passthrough view's rows over time, such as the latest per
key, the top-N per group, or rows matching a predicate, attach an
[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. That page also
covers
[when to put a predicate in the view's `WHERE` clause instead](/docs/concepts/expire-rows/#where-filter-or-expire-rows).

## Creating a materialized view

### Basic syntax
Expand Down Expand Up @@ -214,7 +397,9 @@ interval:

### The query

Materialized views require a `SAMPLE BY` or time-based `GROUP BY` query.
An aggregating materialized view uses a `SAMPLE BY` or time-based `GROUP BY`
query. (The other shape is a [passthrough view](#passthrough-views), which does
not aggregate; the rules below apply to aggregating views.)

**Supported:**

Expand Down Expand Up @@ -526,7 +711,9 @@ rows.

Materialized view queries:

- Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp column
- Must either aggregate with `SAMPLE BY` / `GROUP BY` on a designated timestamp
column, or be a [passthrough](#passthrough-views) projection over a single
table
- Must not use `FROM-TO`, `FILL`, or `ALIGN TO FIRST OBSERVATION`
- Must not use non-deterministic functions (`now()`, `rnd_uuid4()`)
- Must use join conditions compatible with incremental refresh
Expand Down
2 changes: 2 additions & 0 deletions documentation/concepts/ttl.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ A view's TTL is independent of its base table's TTL. For full syntax, see
[CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#ttl-time-to-live)
and
[ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/).
For row-level retention on passthrough materialized views, such as keeping the
latest row per key, see [`EXPIRE ROWS`](/docs/concepts/expire-rows/).

## How TTL works

Expand Down
25 changes: 22 additions & 3 deletions documentation/configuration/materialized-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ title: Materialized views
description: Configuration settings for materialized views in QuestDB.
---

These settings control materialized view SQL support and the background refresh
job. Materialized views can use dedicated worker threads or share the server's
common pool.
These settings control materialized view SQL support, background refresh, and
row-expiry cleanup. Materialized views can use dedicated worker threads or share
the server's common pool.

## cairo.mat.view.enabled

Expand All @@ -22,6 +22,25 @@ Enables or disables SQL support and the refresh job for materialized views.
When disabled, SQL executed by the materialized view refresh job always runs
single-threaded.

## cairo.mat.view.row.expiry.cleanup.enabled

- **Default**: `true`
- **Reloadable**: no

Enables the background job that reclaims rows removed by an eligible
[`EXPIRE ROWS`](/docs/concepts/expire-rows/) policy. Disabling the job does not
disable read filtering, so expired rows remain hidden from query results.

## cairo.mat.view.row.expiry.cleanup.min.expired.fraction

- **Default**: `0.5`
- **Reloadable**: no

Minimum fraction of expired rows required before the background cleanup job
compacts a partially expired partition. Set this property to `0` to compact on
the first expired row, or to `1` to disable partial-partition compaction. Fully
expired partitions are still removed.

## mat.view.refresh.worker.affinity

- **Default**: equal to the CPU core count
Expand Down
21 changes: 15 additions & 6 deletions documentation/operations/data-retention.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,21 @@ description: How to employ a data retention strategy to delete old data and save
The nature of [time-series data](/blog/what-is-time-series-data/) is that the relevance of information diminishes
over time. If stale data is no longer required, users can delete old data from
QuestDB to either save disk space or adhere to a data retention policy. This is
achieved in QuestDB by removing data partitions from a table.
achieved by removing or tiering table partitions, or by applying row-level
retention to a materialized view.

QuestDB offers four approaches for data retention:
QuestDB offers five approaches for data retention:

- **TTL** _(automatic, open source)_: Use
[Time To Live (TTL)](/docs/concepts/ttl/) to automatically drop partitions when
data ages beyond a specified threshold. This is the simplest approach and is
available in both open source and Enterprise editions.
- **EXPIRE ROWS** _(automatic, materialized views only)_: Use
[`EXPIRE ROWS`](/docs/concepts/expire-rows/) for row-granular retention on a
materialized view, such as keeping the latest row per key, the top-N rows per
group, or expiring rows selected by a predicate. The base table is unaffected.
Some policies hide expired rows without reclaiming their disk space, so check
the policy's enforcement mode when storage reduction matters.
- **Storage policy** _(automatic, Enterprise only)_: Use a
[storage policy](/docs/concepts/storage-policy/) to automate the partition
lifecycle — convert to Parquet locally and drop old data on a schedule. This is
Expand All @@ -30,8 +37,9 @@ QuestDB offers four approaches for data retention:
- **Manual**: Use `DROP PARTITION` commands as described on this page for
explicit control over which partitions to remove and when.

This page provides a high-level overview of partitioning with examples to drop
data by date. For more details on partitioning, see the
This page summarizes the available retention approaches, then provides
partitioning examples that drop data by date. For more details on partitioning,
see the
[partitioning](/docs/concepts/partitions/) page.

## Manual partition management
Expand Down Expand Up @@ -109,8 +117,9 @@ syntax. Partitions may be dropped by:
chronologically. Depending on the types of queries users are performing on a
dataset, it may not be desirable to have gaps caused by dropped partitions.
- Unlike TTL, `DROP PARTITION` commands must be triggered manually or via
external scheduling (e.g., cron jobs). For fully automated retention, consider
using [TTL](/docs/concepts/ttl/) instead.
external scheduling (e.g., cron jobs). For automated partition retention,
consider [TTL](/docs/concepts/ttl/) or a
[storage policy](/docs/concepts/storage-policy/) instead.

### Example

Expand Down
Loading
Loading