IN LIST: Handle dictionary inputs once - #24662
Draft
geoffreyclaude wants to merge 4 commits into
Draft
Conversation
Cover plain and dictionary Int32 inputs from one row through the standard 8,192-row batch. The cases use both short and longer IN lists so fixed per-batch costs are visible without losing the usual vectorized comparison.
geoffreyclaude
force-pushed
the
codex/in-list-dictionary-dispatch
branch
from
August 25, 2026 12:56
df02fd2 to
7055e43
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-dictionary-dispatch (7055e43) to c396e6a diff Run configurationrun benchmark in_list
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"Results will be posted here when complete File an issue against this benchmark runner |
Wrap the selected filter when the input is dictionary encoded, instead of teaching every filter implementation how to unpack dictionary keys. Plain inputs continue to use their filter directly. Flatten nested dictionary values before choosing a filter, and cover dictionary key types, nested dictionaries, nulls, timestamps, decimals, and scalar inputs.
geoffreyclaude
force-pushed
the
codex/in-list-dictionary-dispatch
branch
from
August 25, 2026 13:09
7055e43 to
33a3f38
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-dictionary-dispatch (33a3f38) to c396e6a diff Run configurationrun benchmark in_list
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-dictionary-dispatch (7055e43) to c396e6a diff Run configurationrun benchmark in_list
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"CPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-dictionary-dispatch (33a3f38) to c396e6a diff Run configurationrun benchmark in_list
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"CPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24662 +/- ##
==========================================
+ Coverage 81.32% 81.43% +0.11%
==========================================
Files 1117 1120 +3
Lines 396269 399920 +3651
Branches 396269 399920 +3651
==========================================
+ Hits 322260 325690 +3430
+ Misses 55186 55178 -8
- Partials 18823 19052 +229 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-dictionary-dispatch (33a3f38) to c396e6a diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-dictionary-dispatch (33a3f38) to c396e6a diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "c396e6ae465a96725363e7715702d140ca97a2ca"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Contributor
|
❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Dictionary handling is currently repeated in each concrete
IN LISTfilter. That makesdatafusion-physical-exprsubstantially larger to compile and link.What changes are included in this PR?
Dictionary-encoded inputs use one small wrapper around the selected filter. The filter checks each dictionary value once, then maps those results through the dictionary keys. Plain inputs continue to call their selected filter directly.
Nested dictionary list values are flattened before choosing a filter. This preserves SQL null behavior while still allowing the final value type to use its specialized filter.
The PR also adds small-batch Criterion cases for both plain and dictionary inputs.
Measurements
Measured locally on an M1 Max against the exact #24102 head plus the benchmark-only commit.
datafusion-physical-exprLLVM IRStaticFilter::containsLLVM IR__textFocused Criterion benchmarks found no material regression. In the automated AArch64 run, dictionary inputs had 15–26% lower latency at batch sizes 1–64, while 8,192-row and plain-input cases were unchanged (full results).
Are these changes tested?
Yes. Tests cover every dictionary key type, nested dictionaries, dictionary nulls, unreferenced null values, timestamp timezones, decimal metadata, and nested dictionary scalar inputs.
cargo test -p datafusion-physical-expr --all-features(1,694 passed, 1 ignored; 13 doctests passed)cargo clippy --all-targets --all-features -- -D warningscargo fmt --allAre there any user-facing changes?
No. This keeps the existing
IN LISTbehavior while reducing generated code.