IN LIST: centralize primitive filters and optimize Decimal128 - #24283
Merged
alamb merged 5 commits intoAug 18, 2026
Conversation
This was referenced Aug 12, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24283 +/- ##
==========================================
+ Coverage 80.91% 81.14% +0.23%
==========================================
Files 1102 1112 +10
Lines 377102 386940 +9838
Branches 377102 386940 +9838
==========================================
+ Hits 305143 313998 +8855
- Misses 53769 54439 +670
- Partials 18190 18503 +313 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-primitive-selector-decimal
branch
7 times, most recently
from
August 13, 2026 10:14
00885e0 to
65b1e9f
Compare
Use PrimitiveHashSetFilter<T, K> for integer and floating-point arrays. Integer filters keep their native key type by default; Float32 and Float64 use their existing bitwise wrapper keys, preserving signed-zero and NaN-payload behavior. The key type and conversion are selected statically, with no function pointer or dynamic dispatch in the lookup loop. Filter selection, dictionary handling, null handling, IN, and NOT IN behavior remain unchanged. Decimal128 routing remains in the following commit.
Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.
geoffreyclaude
force-pushed
the
codex/in-list-primitive-selector-decimal
branch
from
August 13, 2026 10:46
65b1e9f to
510bf3e
Compare
geoffreyclaude
marked this pull request as ready for review
August 14, 2026 11:15
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
alamb
approved these changes
Aug 18, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thanks @geoffreyclaude -- this looks good to me
| // (Some, false, true) => values: valid & !contains, nulls: valid | ||
| // (None, false, false) => values: contains, nulls: none | ||
| // (None, false, true) => values: !contains, nulls: none | ||
| // Truth table for `value [NOT] IN (set)` with SQL three-valued logic: |
alamb
pushed a commit
to alamb/datafusion
that referenced
this pull request
Aug 26, 2026
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#24283](apache#24283). - Next in stack: [apache#24088](apache#24088). ## Rationale for this change Today, `FixedSizeBinary` `IN LIST` expressions use the general filter, even when their width matches a primitive type with a faster specialized filter. This makes repeated evaluation slower than necessary for common fixed-width values. A `FixedSizeBinary(N)` value is exactly `N` bytes long. For widths 1, 2, 4, 8, and 16, DataFusion already has fast `IN LIST` filters for primitive values of the same size. This PR lets `FixedSizeBinary` reuse them. For example, a 4-byte value can be read as a `UInt32` lookup key. This does not turn the value into a number or perform arithmetic. The list and input use the same mapping, so two keys match exactly when the original bytes match. Machine byte order may change the numeric interpretation, but it does not change equality. The filter depends on the width and the number of non-null values in the list: | Width | Lookup key | Small lists | Larger lists | |---:|---|---|---| | 1 byte | `UInt8` | direct comparison through 16 values | bitmap | | 2 bytes | `UInt16` | direct comparison through 8 values | bitmap | | 4 bytes | `UInt32` | direct comparison through 32 values | hash set | | 8 bytes | `UInt64` | direct comparison through 16 values | hash set | | 16 bytes | `Decimal128` (`i128` storage) | direct comparison through 4 values | hash set | apache#24283 provides this choice, so this PR only needs to map each supported width to the matching primitive key. Other widths keep using the general filter. Arrow buffers are normally aligned for the selected primitive key and can be read without copying. If a buffer is not aligned for that key type, the list is copied once when the filter is built, while an unaligned input is copied before each evaluation. After unwrapping any dictionary, the input must have the same `FixedSizeBinary(N)` type as the list. ## What changes are included in this PR? - Adds the faster path for `FixedSizeBinary` widths 1, 2, 4, 8, and 16. - Reuses the primitive filters from apache#24283. - Preserves slices, dictionaries, nulls, `IN`, and `NOT IN`. - Keeps unsupported widths on the general filter. - Expands `FixedSizeBinary` benchmark coverage from 8 to 14 cases, including both match rates for an input unaligned for `i128` access. ## Are these changes tested? Tests cover every supported width around its direct-comparison limit; bitmap and hash-set hits and misses; slices and dictionaries; input and list nulls; `IN` and `NOT IN`; wrong widths and Arrow types; unsupported widths; and aligned and unaligned buffers. ## Are there any user-facing changes? No. SQL results and public APIs are unchanged. ## Benchmark snapshot The benchmark repeatedly evaluates an 8,192-row batch; filter construction is outside the timed loop. Each selected configuration is measured with 0% and 50% matches: - `fsb1/list=16` exercises the direct-comparison boundary for 1-byte keys. - `fsb2/list=64` exercises the bitmap path. - `fsb16/list=4` exercises the direct-comparison boundary for 16-byte keys. - `fsb16/list=64`, `list=256`, and `list=10000` exercise hash-set scaling. - `fsb16/list=64/input=unaligned` exercises the per-evaluation input-copy path for buffers unaligned for `i128` access. Lower is better. An [automated run](apache#24102 (comment)) compared this PR's [benchmark-only first commit](apache@a1c8762) with the [completed implementation](apache@77498a0). The benchmark source is identical in both states, so the comparison isolates the production changes. Across this selected case set, speedups range from 1.51x to 12.11x, and the geometric mean is 3.17x (68.5% lower execution time). The aligned and unaligned `fsb16/list=64` cases contain identical values, so their results show the per-evaluation input-copy overhead: +1.6 us (12%) with 0% matches and +3.5 us (17%) with 50% matches. | Benchmark | Before | After | Change | |---|---:|---:|---:| | `fixed_size_binary/fsb1/list=16/match=0%` | 25.4 us | 4.7 us | -81.5% (5.40x faster) | | `fixed_size_binary/fsb1/list=16/match=50%` | 56.9 us | 4.7 us | -91.7% (12.11x faster) | | `fixed_size_binary/fsb2/list=64/match=0%` | 24.6 us | 5.3 us | -78.5% (4.64x faster) | | `fixed_size_binary/fsb2/list=64/match=50%` | 58.2 us | 5.3 us | -90.9% (10.98x faster) | | `fixed_size_binary/fsb16/list=4/match=0%` | 22.2 us | 11.6 us | -47.7% (1.91x faster) | | `fixed_size_binary/fsb16/list=4/match=50%` | 56.0 us | 11.6 us | -79.3% (4.83x faster) | | `fixed_size_binary/fsb16/list=64/match=0%` | 22.3 us | 13.2 us | -40.8% (1.69x faster) | | `fixed_size_binary/fsb16/list=64/match=50%` | 55.4 us | 20.8 us | -62.5% (2.66x faster) | | `fixed_size_binary/fsb16/list=256/match=0%` | 22.4 us | 13.1 us | -41.5% (1.71x faster) | | `fixed_size_binary/fsb16/list=256/match=50%` | 53.0 us | 20.5 us | -61.3% (2.59x faster) | | `fixed_size_binary/fsb16/list=10000/match=0%` | 24.4 us | 14.3 us | -41.4% (1.71x faster) | | `fixed_size_binary/fsb16/list=10000/match=50%` | 56.5 us | 19.4 us | -65.7% (2.91x faster) | | `fixed_size_binary/fsb16/list=64/match=0%/input=unaligned` | 22.3 us | 14.8 us | -33.6% (1.51x faster) | | `fixed_size_binary/fsb16/list=64/match=50%/input=unaligned` | 55.7 us | 24.3 us | -56.4% (2.29x faster) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
#23014 adds fast direct comparisons for short primitive
INlists. For longer lists, DataFusion uses a bitmap, a hash set, or another specialized filter when one exists, and otherwise uses the general filter.#24102 needs to make that same choice after reading
FixedSizeBinaryvalues as same-width primitive keys. Today, the choice is part of the native primitive path, so #24102 would have to repeat its limits and fallback rules.This PR moves that choice beside the primitive filters and makes it reusable. The limits count only non-null list values. A separate commit uses one
PrimitiveHashSetFilter<T, K>for the existing integer and floating-point hash-set paths. Integers keep their native keys;Float32andFloat64keep their bitwise keys, including distinct signed zeros and NaN encodings. Filter selection and SQL behavior are unchanged.The only new lookup path is for
Decimal128. Lists with at most four non-null values still use direct comparisons. Larger lists now use a hash set over the storedi128values instead of the general Arrow filter. Values are not rescaled, and the existing precision, scale, and null rules are unchanged. #24102 also uses this path for 16-byteFixedSizeBinaryvalues.What changes are included in this PR?
PrimitiveHashSetFilter<Decimal128Type>forDecimal128lists with more than four non-null values.Decimal128benchmark cases at list sizes 5 and 64.Are these changes tested?
Tests cover the direct-comparison boundary, an all-null list, and bitwise float hash keys, including signed zero and distinct NaN encodings. Existing end-to-end Decimal tests cover hits, misses, nulls,
IN, andNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Benchmark snapshot
The benchmark uses list size 5, the first size above the direct-comparison limit, and a larger size of 64. Each size is measured with 0% and 50% matches. Filter construction is outside the timed loop. Lower is better.
An automated run compared this PR's benchmark-only first commit with the completed implementation. The benchmark source is identical in both states, so the comparison isolates the production changes.
All four cases improved, with a 34.2% geometric-mean reduction in execution time.
primitive/decimal128/large_list/list=5/match=0%primitive/decimal128/large_list/list=5/match=50%primitive/decimal128/large_list/list=64/match=0%primitive/decimal128/large_list/list=64/match=50%