IN LIST: optimize Utf8View and BinaryView filters - #24088
Draft
geoffreyclaude wants to merge 3 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24088 +/- ##
==========================================
+ Coverage 81.32% 81.40% +0.08%
==========================================
Files 1117 1120 +3
Lines 396269 399153 +2884
Branches 396269 399153 +2884
==========================================
+ Hits 322260 324925 +2665
- Misses 55186 55209 +23
- Partials 18823 19019 +196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 5, 2026 07:02
ec9cb2e to
65b67ba
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listCPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 6, 2026 16:01
65b67ba to
a44ee0d
Compare
geoffreyclaude
changed the title
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 9, 2026 06:21
a44ee0d to
b0443b6
Compare
11 tasks
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
10 times, most recently
from
August 11, 2026 16:29
61cc7a6 to
156d76b
Compare
github-actions
Bot
added
auto detected api change
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
2 times, most recently
from
August 12, 2026 10:18
6a5284e to
f8ae3ad
Compare
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Contributor
|
I noticed this is marked as draft -- please ping me when ready to review |
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
9 times, most recently
from
August 13, 2026 11:42
cc8aacf to
00dd083
Compare
Contributor
Author
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 00dd083 (00dd083) to 7ff49f9 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"
changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 00dd083 (00dd083) to 7ff49f9 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"
changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Contributor
Author
@alamb thank you! I'm reworked a bit the ordering of PRs, the next logical one to me should be #24283 , and it should now be ready for review. The next two are pretty much ready as well, just keeping them in draft so we can rebase over main for easier review as we go. |
Treat inline Utf8View and BinaryView values as 128-bit primitive keys and pass them to the shared primitive filter selector. This reuses direct comparisons for short lists and the primitive hash-set path for larger lists without reading backing buffers. Lists containing a non-inline value continue to use the general filter. Exact view types, dictionaries, slices, nulls, IN, and NOT IN keep their existing behavior.
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 24, 2026 15:54
00dd083 to
b214a31
Compare
alamb
pushed a commit
to alamb/datafusion
that referenced
this pull request
Aug 26, 2026
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#24283](apache#24283). - Next in stack: [apache#24088](apache#24088). ## Rationale for this change Today, `FixedSizeBinary` `IN LIST` expressions use the general filter, even when their width matches a primitive type with a faster specialized filter. This makes repeated evaluation slower than necessary for common fixed-width values. A `FixedSizeBinary(N)` value is exactly `N` bytes long. For widths 1, 2, 4, 8, and 16, DataFusion already has fast `IN LIST` filters for primitive values of the same size. This PR lets `FixedSizeBinary` reuse them. For example, a 4-byte value can be read as a `UInt32` lookup key. This does not turn the value into a number or perform arithmetic. The list and input use the same mapping, so two keys match exactly when the original bytes match. Machine byte order may change the numeric interpretation, but it does not change equality. The filter depends on the width and the number of non-null values in the list: | Width | Lookup key | Small lists | Larger lists | |---:|---|---|---| | 1 byte | `UInt8` | direct comparison through 16 values | bitmap | | 2 bytes | `UInt16` | direct comparison through 8 values | bitmap | | 4 bytes | `UInt32` | direct comparison through 32 values | hash set | | 8 bytes | `UInt64` | direct comparison through 16 values | hash set | | 16 bytes | `Decimal128` (`i128` storage) | direct comparison through 4 values | hash set | apache#24283 provides this choice, so this PR only needs to map each supported width to the matching primitive key. Other widths keep using the general filter. Arrow buffers are normally aligned for the selected primitive key and can be read without copying. If a buffer is not aligned for that key type, the list is copied once when the filter is built, while an unaligned input is copied before each evaluation. After unwrapping any dictionary, the input must have the same `FixedSizeBinary(N)` type as the list. ## What changes are included in this PR? - Adds the faster path for `FixedSizeBinary` widths 1, 2, 4, 8, and 16. - Reuses the primitive filters from apache#24283. - Preserves slices, dictionaries, nulls, `IN`, and `NOT IN`. - Keeps unsupported widths on the general filter. - Expands `FixedSizeBinary` benchmark coverage from 8 to 14 cases, including both match rates for an input unaligned for `i128` access. ## Are these changes tested? Tests cover every supported width around its direct-comparison limit; bitmap and hash-set hits and misses; slices and dictionaries; input and list nulls; `IN` and `NOT IN`; wrong widths and Arrow types; unsupported widths; and aligned and unaligned buffers. ## Are there any user-facing changes? No. SQL results and public APIs are unchanged. ## Benchmark snapshot The benchmark repeatedly evaluates an 8,192-row batch; filter construction is outside the timed loop. Each selected configuration is measured with 0% and 50% matches: - `fsb1/list=16` exercises the direct-comparison boundary for 1-byte keys. - `fsb2/list=64` exercises the bitmap path. - `fsb16/list=4` exercises the direct-comparison boundary for 16-byte keys. - `fsb16/list=64`, `list=256`, and `list=10000` exercise hash-set scaling. - `fsb16/list=64/input=unaligned` exercises the per-evaluation input-copy path for buffers unaligned for `i128` access. Lower is better. An [automated run](apache#24102 (comment)) compared this PR's [benchmark-only first commit](apache@a1c8762) with the [completed implementation](apache@77498a0). The benchmark source is identical in both states, so the comparison isolates the production changes. Across this selected case set, speedups range from 1.51x to 12.11x, and the geometric mean is 3.17x (68.5% lower execution time). The aligned and unaligned `fsb16/list=64` cases contain identical values, so their results show the per-evaluation input-copy overhead: +1.6 us (12%) with 0% matches and +3.5 us (17%) with 50% matches. | Benchmark | Before | After | Change | |---|---:|---:|---:| | `fixed_size_binary/fsb1/list=16/match=0%` | 25.4 us | 4.7 us | -81.5% (5.40x faster) | | `fixed_size_binary/fsb1/list=16/match=50%` | 56.9 us | 4.7 us | -91.7% (12.11x faster) | | `fixed_size_binary/fsb2/list=64/match=0%` | 24.6 us | 5.3 us | -78.5% (4.64x faster) | | `fixed_size_binary/fsb2/list=64/match=50%` | 58.2 us | 5.3 us | -90.9% (10.98x faster) | | `fixed_size_binary/fsb16/list=4/match=0%` | 22.2 us | 11.6 us | -47.7% (1.91x faster) | | `fixed_size_binary/fsb16/list=4/match=50%` | 56.0 us | 11.6 us | -79.3% (4.83x faster) | | `fixed_size_binary/fsb16/list=64/match=0%` | 22.3 us | 13.2 us | -40.8% (1.69x faster) | | `fixed_size_binary/fsb16/list=64/match=50%` | 55.4 us | 20.8 us | -62.5% (2.66x faster) | | `fixed_size_binary/fsb16/list=256/match=0%` | 22.4 us | 13.1 us | -41.5% (1.71x faster) | | `fixed_size_binary/fsb16/list=256/match=50%` | 53.0 us | 20.5 us | -61.3% (2.59x faster) | | `fixed_size_binary/fsb16/list=10000/match=0%` | 24.4 us | 14.3 us | -41.4% (1.71x faster) | | `fixed_size_binary/fsb16/list=10000/match=50%` | 56.5 us | 19.4 us | -65.7% (2.91x faster) | | `fixed_size_binary/fsb16/list=64/match=0%/input=unaligned` | 22.3 us | 14.8 us | -33.6% (1.51x faster) | | `fixed_size_binary/fsb16/list=64/match=50%/input=unaligned` | 55.7 us | 24.3 us | -56.4% (2.29x faster) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Note
Until #24102 merges, GitHub's Files changed tab also shows the earlier PRs in the stack. The direct diff from #24102 to this PR shows only this PR's byte-view changes.
Rationale for this change
Arrow represents every
Utf8ViewandBinaryViewvalue with a 16-byte view. When a value is at most 12 bytes long, the complete value is stored directly in that view:Equal short values therefore have the same 128-bit view. DataFusion can look up that view directly, without reading another buffer or using Arrow's general value comparison.
This PR uses that faster path when every non-null value in the
INlist is at most 12 bytes long. The byte-view adapter treats each inline view as aDecimal128key and passes it to the shared primitive selector from #24283, the same selector used by #24102:Decimal128is only a 16-byte container here; no decimal operations are performed.A longer input value cannot match a short list value because its encoded length is different, so it can be rejected without reading its backing bytes. If the list itself contains a value longer than 12 bytes, DataFusion keeps using the general filter.
Null list entries do not affect this choice, but are still recorded for SQL null behavior. After unwrapping any dictionary, the input and list must use the same view type:
Utf8Viewis not mixed withBinaryView, and regularUtf8andBinarykeep their existing paths.What changes are included in this PR?
Utf8ViewandBinaryView.IN, andNOT IN.Are these changes tested?
Tests cover both view types and both faster paths; the 4/5-value and 12/13-byte boundaries; long inputs and lists that require the general filter; exact-type routing; slices and dictionaries; input and list nulls; and
INandNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Automated benchmark snapshot
Automated run on an aarch64 Neoverse V2 runner, comparing #24102 with this PR at exact commit
00dd083e. The benchmark evaluates 8,192-row batches after filter construction. Lower is better.All 15 reported rows improved, with a 35.9% geometric-mean time reduction.
utf8view/short_8b/list=4/match=0%utf8view/short_8b/list=4/match=50%utf8view/short_8b/list=16/match=0%utf8view/short_8b/list=16/match=50%utf8view/short_8b/list=64/match=0%utf8view/short_8b/list=64/match=50%utf8view/short_8b/list=256/match=0%utf8view/short_8b/list=256/match=50%utf8view/len_12b/list=16/match=0%utf8view/len_12b/list=16/match=50%utf8view/len_12b/list=64/match=0%utf8view/len_12b/list=64/match=50%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_INnulls/utf8view/short_8b/list=16/match=50%/nulls=50%