perf: use compact pruning for large string IN lists

[sunchao]

Why are the changes needed?

Which issue does this PR close?

Related to #8668 and #8609; follows #24074. This PR is stacked on #24525, which must merge first.

Rationale for this change

Queries often select a sparse set of string identifiers. Parquet min/max statistics can make these queries much cheaper by ruling out row groups or pages that cannot contain any requested identifier. For example, this query asks for 21 IDs, spaced ten apart:

SET datafusion.execution.parquet.max_in_list_size = 1024;

SELECT *
FROM events
WHERE customer_id IN (
  'id000', 'id010', 'id020', 'id030', 'id040', 'id050', 'id060',
  'id070', 'id080', 'id090', 'id100', 'id110', 'id120', 'id130',
  'id140', 'id150', 'id160', 'id170', 'id180', 'id190', 'id200'
);

A row group whose values fall between id003 and id007 cannot contain a match. The default pruning limit is 20, so this list is not eligible for the IN min/max rewrite unless the caller raises the limit. #24074 made that limit configurable. With a raised limit, DataFusion can already reject this row group, but it does so by constructing a growing expression tree resembling:

(min <= id000 AND id000 <= max)
OR (min <= id010 AND id010 <= max)
OR ...

For hundreds or thousands of identifiers, building and evaluating that tree can become expensive in its own right. Replacing the list with one enclosing range, [id000, id200], would be cheaper, but would lose the gaps: that broad range overlaps [id003, id007] even though none of the requested IDs is present there.

The aim is to keep the useful pruning precision of the existing per-value checks while making large lists cheaper to represent and evaluate. In the included local microbenchmark, evaluating 1,024 values against 4,096 intervals falls from 68.8 ms to 0.198 ms. This measures pruning work only, not end-to-end query speedup.

What changes were proposed in this PR?

What changes are included in this PR?

Eligible large string lists are stored as a sorted, deduplicated set of values inside the pruning predicate, rather than expanded into one comparison branch per value. For each inclusive statistics interval, DataFusion finds the first requested value at or after the interval's minimum, then checks whether that value is also at or before its maximum.

In the example, the first requested ID at or after id003 is id010. Since id010 > id007, the row group can be skipped. An interval such as [id019, id021] must be kept because it could contain id020. This takes a binary search per interval after sorting the values once per constructed predicate, and the expression tree no longer grows with the number of IDs.

The result remains a conservative pruning decision. An overlapping interval means only that a match is possible; the original IN expression still performs exact row filtering. The original literal information also remains available to other pruning mechanisms, including Bloom filters. Missing or inverted bounds cannot prove that data is safe to skip.

The existing limit continues to control eligibility. Its default remains 20, setting it to zero disables the IN min/max rewrite, and lists beyond the configured cap remain ineligible. Only eligible positive, non-null literal string lists larger than 20 take the compact path. NOT IN, NULL-containing lists, and unsupported expressions keep their existing handling. Page-index pruning now receives the same configured cap as row-group pruning, so raising the limit can benefit both.

The dependency on #24525 matters for correctness: an interval search is only meaningful when the stored bounds use the same comparison order as the query. That companion PR handles legacy or unrecognized Parquet byte-array ordering. It must land before the newly enabled large-list page-pruning path here.

Are there any user-facing changes?

Users who raise datafusion.execution.parquet.max_in_list_size get cheaper min/max pruning for eligible large string lists, and page pruning now honors that setting. The configuration default, exact query results, and existing public APIs are unchanged. This is a focused optimization for literal string lists, not a general rewrite of every large IN expression.

How was this PR tested?

Are these changes tested?

The pruning-crate suite passed 93 tests. The standalone Parquet regressions use lists of 20, 21, 256, and 1,024 values and check both exact query results and scan/pruning metrics. They cover gaps inside the list's enclosing range, row-group pruning, page-only pruning, and the default and zero-cap controls.

Two correctness regressions exercise the less obvious interactions. A direct physical-source test uses NOT IN (..., NULL), row-filter pushdown, and LIMIT 1, so logical optimizer folding cannot hide an incorrect decision to bypass filtering. A real-file test combines this PR with #24525 and verifies that compact page pruning cannot lose a matching row when the footer's ordering is missing or unknown. Against unchanged Apache f1f0449a, the positive row-group/page tests fail as expected; the NOT IN (..., NULL) control passes.

The benchmark compares the actual raised-cap IN path on Apache f1f0449a and this patch, using separate build directories and checking that both return the same nontrivial pruning results. Local results on an Apple M5 Max (18 CPUs, 128 GiB), Rust 1.97.0, release-nonlto, 20 samples:

Values	Predicate construction, main → PR	Evaluate 4,096 intervals, main → PR
20	31.1 → 27.4 µs	177.6 → 165.7 µs
21	30.5 → 3.52 µs	183.8 → 98.1 µs
256	356.6 → 21.4 µs	5.53 → 0.144 ms
1,024	1.466 → 0.081 ms	68.8 → 0.198 ms

A balanced explicit OR tree is included as another comparison: at 1,024 values it takes 8.03 ms to evaluate the same intervals. These measurements isolate pruning overhead; no end-to-end workload improvement is claimed.

Formatting, all-targets/all-features Clippy with warnings denied, and ./dev/rust_lint.sh passed on the combined stack. The extended workspace run passed 10,674 Rust tests, with eight ignored, and all 503 SQL-logic files.

Validation commands

cargo test --locked --profile ci -p datafusion-pruning

cargo test --locked --profile ci -p datafusion \
  --test parquet_integration string_in_list_pruning

cargo bench --locked --profile release-nonlto -p datafusion-pruning \
  --bench string_in_list_pruning -- \
  --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot

RUST_BACKTRACE=1 cargo test --locked --profile ci \
  --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
  --workspace --lib --tests --bins \
  --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.65022% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.46%. Comparing base (6e66a85) to head (df65b7c).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/pruning/src/string_in_list.rs	73.22%	28 Missing and 6 partials ⚠️
datafusion/pruning/src/pruning_predicate.rs	90.03%	5 Missing and 25 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24526      +/-   ##
==========================================
+ Coverage   81.44%   81.46%   +0.01%     
==========================================
  Files        1118     1119       +1     
  Lines      399602   400126     +524     
  Branches   399602   400126     +524     
==========================================
+ Hits       325460   325947     +487     
+ Misses      55146    55141       -5     
- Partials    18996    19038      +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sunchao]

cc @alamb

[alamb]

FYI I think @geoffreyclaude has many optimizations for IN list in flight

• Further improve performance of IN list evaluation #19241

@geoffreyclaude can you help review this PR and let me know if it conflicts with what you have planned?

(BTW @sunchao is the original author of the arrow-rs parquet reader, one of the original contributors of DataFusion Comet, and a long time friend of the project)

[sunchao]

Thanks @alamb for the intro :) It's great to be back to the DataFusion community!

cc @viirya @comphead @andygrove too - once integrated and enabled, this PR lets Comet’s native Parquet scans efficiently skip row groups and pages using large string IN filters, reducing pruning CPU overhead and unnecessary I/O.

[viirya]

Nice result — replacing the OR tree with a sorted-domain intersection keeps the gap precision while making large lists cheap to represent and evaluate, and the eligibility gate reads conservatively: positive IN only, string (or dictionary-of-string) types only, any non-string literal in the list bails out, oversized string buffers degrade to all-NULL, and the result is combined with a non-null check so an all-NULL container is handled by the null-count branch. Good catch on page-level pruning too — it now honours the same max_in_list_size as row-group pruning rather than silently using the default.

Re @alamb's question about whether this conflicts with #19241: as far as I can tell it doesn't — the two work at different layers. #19241 optimizes row-level IN evaluation (per-row bitmap / hash membership), and this PR explicitly leaves InListExpr as the row filter ("this expression is used only for pruning; the original IN remains the row filter"). This PR works at the container-pruning layer, deciding from parquet min/max statistics whether a row group or page needs to be read at all. They compose rather than overlap: fewer containers read, and the rows that are read still go through whatever the fastest row filter is.

The one place I'd flag for @geoffreyclaude is Utf8View/dictionary handling: #24088 (all-inline Utf8View/BinaryView) and #24658 (avoid dictionary filter copies) touch those on the row-filter side, while this PR adds its own Utf8View/dictionary normalization here — including the u32-offset overflow guard and preserving dictionary values' logical nulls. Probably an opportunity for shared helpers rather than a conflict, but worth a look from both sides.

A few questions inline.

[viirya]

The set is sorted with values.sort_unstable() (Rust String/str ordering) and compared here via as_bytes(), while the interval bounds come from parquet statistics. Parquet specifies unsigned byte-wise ordering for BYTE_ARRAY/UTF8 and Rust's str Ord is also byte-wise, so these line up — but it is the load-bearing assumption of the whole optimization and it's currently implicit. Could a comment state it?

Relatedly: is there any path where the column's ColumnOrder is unset or defines a different collation, where these min/max shouldn't be trusted for an ordered comparison? The existing per-value min <= v AND v <= max path has the same dependency, so this may be pre-existing rather than something this PR introduces — I mostly want to know whether it's been considered.

[viirya]

Following up on my own question: I see now that #24525 — the PR this one is stacked on — is precisely about this, distinguishing the legacy signed-byte min/max from the unsigned ordering Arrow uses, and it calls itself "the correctness prerequisite for #24526". So the ordering question is already handled upstream in the stack; sorry for asking you to re-explain it.

What's still worth confirming is narrower: does the compact path here sit behind the same ordering gate that #24525 introduces? Since StringInListPruningExpr does its own byte comparisons on the statistics arrays rather than going through the per-value comparison expressions, I wanted to check that a column whose statistics ordering is unusable is rejected before it reaches this expression, rather than only being filtered on the path #24525 touched.

[sunchao]

Yes—the compact expression reads the same gated min/max columns as the per-value expressions; it does not access raw Parquet metadata. The row-group adapter withholds bounds for unusable footer ordering and masks deprecated byte-array bounds to NULL. The page adapter also withholds untrusted bounds, and runtime row-group pruning reuses the row-group adapter.

The distinction is that we suppress unusable bounds, not the entire predicate: the compact expression can still run with NULL bounds and return UNKNOWN, keeping the container unless independent statistics, such as an all-null count, safely exclude it.

The 22-value page regression explicitly asserts IN_SET_INTERSECTS and verifies that the matching az row survives missing/unknown column order.

Agreed about making the ordering assumption explicit: Rust string ordering and these byte comparisons both use unsigned lexicographic UTF-8 byte order. Generic PruningStatistics providers retain the existing responsibility to supply bounds in the comparison order, or mark them unavailable. A short comment here would make that contract clearer.

[viirya]

These single-bound arms exclude via values.last() / values.first() and otherwise fall through to None (unknown, so the container is kept). That looks right to me, but the asymmetry with the two-bound case is worth a comment: when only max is known the interval is unbounded below, so values.first() > max is the only thing that can rule it out — no gap reasoning is possible. Is that the intended reading, i.e. is first() > max the complete exclusion condition here?

[sunchao]

Yes, that is the intended reading. With only max, the possible interval is unbounded below, so a nonempty sorted domain is disjoint exactly when values.first() > max. Symmetrically, with only min, it is disjoint exactly when values.last() < min. Equality must keep the container; otherwise we return UNKNOWN. Gaps between requested values cannot exclude an interval without its other endpoint.

The interval regression checks both exclusions and equality at the endpoints for lists of 20, 21, 256, and 10,000 values. Agreed that spelling out the unbounded-interval interpretation would improve the comment.

[viirya]

IN_SET_INTERSECTS returns NULL for the degenerate min > max case and for unknown bounds, which pruning should treat as "cannot rule out" and keep the container. Is that pinned down by a test end-to-end — a container whose statistics are inverted or missing, asserted to be kept rather than pruned? string_in_list_pruning.rs covers a lot of ground and I couldn't tell whether the inverted-statistics case is among them.

[sunchao]

Yes at the final container-decision level, although these cases are in pruning_predicate.rs, not the Parquet integration module:

• large_string_in_list_handles_unicode_and_unknown_bounds includes inverted bounds min="z", max="m" and asserts that the container is kept.
• large_string_in_list_preserves_dictionary_nulls checks missing bounds from NULL keys/values and inverted bounds min="zz", max=""; those containers are also kept.

Both call PruningPredicate::prune and assert the final Boolean decisions, so they cover UNKNOWN becoming “keep,” rather than only checking the expression's NULL result.

There is not a dedicated full Parquet scan test with an inverted footer. Separately, the compact page-order regression reads a real Parquet fixture and verifies that matching rows survive missing/unknown statistics ordering.

[viirya]

The window is len() > MAX_IN_LIST_SIZE && len() <= max_in_list_size, so lists at or below the default keep the per-value path. Since the compact form looks both cheaper and (as far as I can tell) exactly as precise as the OR tree, what's the reason for keeping the lower bound rather than using the compact form for every eligible positive string list? If it's to avoid perturbing existing plans and EXPLAIN output for small lists, that would be worth saying in the comment.

[sunchao]

The lower bound preserves the existing pruning-expression representation for lists of at most 20 values and keeps this PR focused on the cost of large lists when callers explicitly raise the cap. The value 20 comes from the existing default; it is not an established performance crossover.

Using the compact form for smaller eligible string lists is worth considering separately. The current benchmark keeps the 20-value case on the legacy path, so it does not establish that compact construction/normalization is cheaper across the small-list cases. A focused comparison could justify broadening the optimization later. Agreed that the scope/default-behavior rationale should be stated next to this condition.

[comphead]

@adriangb cc

[comphead]

do we need to pass schema by val?

[sunchao]

Good point, I'll change it to pass &SchemaRef instead.

[sunchao]

Addressed in df65b7c. new_with_max_in_list_size now takes &SchemaRef, and the opener passes its existing reference. The public new signature is unchanged.

I also added the agreed ordering, single-bound, and threshold comments. Formatting, full all-targets/all-features Clippy with warnings denied, and ./dev/rust_lint.sh passed, as did 10,796 Rust tests (8 ignored) and all 505 SQL logic files. The extended rerun used a 65,536 open-file limit after the initial run hit the environment's 1,024 limit.

[comphead]

👍

[comphead]

Thanks @sunchao the PR looks solid to me, nice test coverage, duplicates collapsed.

For benches it shows dramatic speed up after values > 20, which is supposedly related to MAX_IN_LIST_SIZE?

[sunchao]

For benches it shows dramatic speed up after values > 20, which is supposedly related to MAX_IN_LIST_SIZE?

Yes, as MAX_IN_LIST_SIZE grows, the old implementation does more work. For each row group or page:

• Old approach: Evaluate a comparison branch for every requested value: “Does this value fall between min and max?” Work grows roughly as N.
• Compact approach: Sort the values once, then binary-search for the first value ≥ min and check whether it is ≤ max. Work per interval grows roughly as log₂(N).

Therefore this is especially useful when we want to push down a large number of sparse IDs or keys to Parquet.

[geoffreyclaude]

FYI I think @geoffreyclaude has many optimizations for IN list in flight

• Further improve performance of IN list evaluation #19241

@geoffreyclaude can you help review this PR and let me know if it conflicts with what you have planned?

(BTW @sunchao is the original author of the arrow-rs parquet reader, one of the original contributors of DataFusion Comet, and a long time friend of the project)

Thanks for the ping @alamb. I took a high level look at this and I don't see any conflict with #19241: rather it's complimentary as @viirya noted.
There might indeed be opportunities for factorizing common code, but that should probably be done once both streams are committed and stable to avoid annoying conflicts.
I'll do a more in-depth pass on the PR and see if there are any in-line comments to make. But overall it looks really nice!

[geoffreyclaude]

minor documentation suggestion: could the type-level comment mention that evaluate returns one Boolean per min/max statistics interval? IIUC, true means the IN set intersects the interval, false it is disjoint, and NULL means the bounds are incomplete or invalid.

[geoffreyclaude]

My Codex review pointed me to apache/arrow-rs#10510 which seems to fix this issue. Can we just add a short TODO to review again once the Arrow dependency is upgraded?

[adriangb]

This is a very cool change! Did you base this on some existing system / literature or is it a novel optimization?

There's a couple interesting things worth discussing with the limits:

1. The PR / documentation could be a bit clearer around what the limits are and how things behave after this PR. My understanding is that the regime is now len() < min(20, limit) is always dense, min(20, limit) < len() < limit uses the new representation if the array is strings, limit < len() disables pruning completely. I had to think about it / read the code and description a couple of times and ask an agent to verify.
2. There is a footgun in the min(20, limit) < len() < limit regime for non-strings: we'd still build the large OR tree for e.g. x in (1, 2, 3..., N) (more on this below). So a user setting a larger limit so that they can use this optimization is exposed to degenerate behavior that they may not control (if they run user provided queries).
3. The proposal discards the option of vmin <= c1_max AND c1_min <= vmax citing that it would loose pruning power. This is true, but in cases like ULIDs the remaining pruning power might still do a lot of work. And at the len() > limit regime which currently has no pruning coverage it's pretty much a free win (it's O(1) to evaluate and build no matter the length of the list).

Regarding type support: is there any reason we don't support other types? Ints in particular seem straightforward to support. Byte types are pretty much an identical implementation to string types. Would you plan these as followups or are there reasons to not implement them?

Regarding NOT IN and NULL-containing lists: is there a fix we could make on main to allow these to be supported? We have some special handling of nulls in identify_fully_matched_row_groups, we could expand it. I'm not sure if that is the source of the issue or it's more widespread.

[adriangb]

This seems to me like the kind of thing where we would want a PagePruningAccessPlanFilterBuilder in the future if we add any more options / constructors. But maybe this is fine for now.

[adriangb]

Indeed seems like this is all a thin wrapper around a builder, might be a sign...

[adriangb]

Have we analyzed if this is the best representation? E.g. could we use a StringArray instead?

[adriangb]

These both walk every value. It might be worth measuring what impact that has at the extremes, it would depend on how much these expression trees are walk / if there's any dedup or expression optimization run on them. We could always build a digest of the values upfront. Probably fine to leave for now.

[comphead]

Nice PR, thanks @sunchao

I would be honest, I checked the same in DuckDB and seems this implementation is better

┌──────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────┐
  │                      │                                         DuckDB                                          │                    DataFusion PR                     │
  ├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
  │ Container test       │ Inclusive [min,max] byte-interval membership per constant                               │ Same interval, tested via sorted-domain intersection │
  ├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
  │ Order                │ Unsigned memcmp + length tiebreak (string_type.hpp:217)                                 │ Rust &[u8] Ord                                       │
  ├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
  │ IN evaluation        │ Linear scan of the list per container, 2 memcmp/value, early-out (string_stats.cpp:704) │ Sort+dedup once, one partition_point per container   │
  ├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
  │ Cost per container   │ O(n)                                                                                    │ O(log n)                                             │
  ├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
  │ Collation in zonemap │ None (byte-wise)                                                                        │ None (byte-wise)                                     │
  └──────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────┘

[viirya]

Took another pass over the latest head (df65b7cc). Two of the three comments from the earlier threads have landed — the unsigned-byte-order assumption at the partition_point call and the unbounded-interval reasoning on the single-bound arms both read clearly now.

I also went and checked the ordering-gate claim from our earlier thread rather than leaving it as a question, and it holds: RowGroupPruningStatistics::min_values/max_values route through mask_untrusted_byte_array_stats, which nullifs untrusted byte-array bounds per row group, and has_untrusted_min_max_order withholds the column entirely. So the compact expression really does read the same gated columns as the per-value path, and unusable bounds arrive as NULL rather than as misordered values. Thanks for the pointers — that was the part I couldn't tell from the diff alone.

Worth noting for other reviewers: LiteralGuarantee::analyze runs on the original predicate, so bloom-filter / contained() pruning is unaffected by the rewrite. Given #8668 is in the same neighbourhood, that seemed worth stating explicitly.

Overall this looks ready to me. The algorithm is precision-equivalent to the OR tree (sorted deduped domain plus a binary search for the first value at or above min, then one comparison against max), the eligibility gate is conservative on every axis I checked (positive IN only, string or dictionary-of-string only, any non-string literal in the list bails out, column name must match), and every degenerate case I could think of resolves to "keep the container": inverted bounds, missing bounds, dictionary values that are NULL behind valid keys, and string buffers too large for Utf8View's offsets. The tests cover those, and the correctness prerequisite (#24525) is already merged. The page-level max_in_list_size fix is a nice bonus — that path was silently using the default before.

Approving. Two comment-level follow-ups inline, neither blocking. Separately, codecov reports 73% patch coverage on string_in_list.rs; from reading the tests the risky branches (inverted, missing, oversized, dictionary nulls) are covered, so I assume the gap is boilerplate like Display/Hash/with_new_children, but flagging it in case something substantive is untested.

[viirya]

This comment states a contract on statistics providers generally, but the enforcement it points at is Parquet-specific. I checked the other providers that can reach this expression — eligibility keys only on the schema type being a string, so it carries no knowledge of who supplied the bounds:

• PartitionPruningStatistics — bounds are partition values DataFusion produced itself, so they are in Arrow order by construction.
• PrunableStatistics (built in file_pruner.rs from a file's ColumnStatistics) — bounds come from whatever the TableProvider reported, with no ordering gate on that path.

The exposure isn't new; the per-value min <= v AND v <= max path has the same dependency. But since this is the first place the requirement is written down, the comment should say which providers are known to satisfy it and which are taken on trust — as written it reads as if the invariant is enforced everywhere. (If you think it belongs closer to the source, PruningStatistics::min_values might be the better home for it.)

[viirya]

This is the one remaining item from the earlier thread — you agreed the scope/default-behaviour rationale should sit next to this condition, and it doesn't appear to be in df65b7cc yet. A sentence saying the lower bound is a scope/compatibility choice rather than a measured threshold would stop the next reader assuming 20 is tuned.