bench: scalar functions over dictionary-encoded columns - #24586
bench: scalar functions over dictionary-encoded columns#24586radmirnovii wants to merge 1 commit into
Conversation
neilconway
left a comment
There was a problem hiding this comment.
Scalar functions over dictionary-encoded columns have no benchmark coverage:
neither the hand-written dictionary arms (e.g. reverse's, from #23930)
#23930 added benchmarks for reverse over dictionary-encoded columns, which are partially duplicated by this PR, albeit in a different crate (datafusion-functions vs datafusion-physical-expr). Can you take a look at the existing benchmarks and consolidate them together with your additions?
| // group would quietly measure a warm dictionary under a cold name. | ||
| for distinct in [8usize, 256, 512, ROWS] | ||
| .into_iter() | ||
| .filter(|_| *dictionary_calls) |
There was a problem hiding this comment.
Seems a bit odd to write the conditional this way, since *dictionary_calls is invariant over the inner loop. Write if *dictionary_calls { ... } instead? Here and below.
a8c5aec to
e4894fc
Compare
e4894fc to
40f66fd
Compare
|
Consolidated into |
40f66fd to
b94975d
Compare
Which issue does this PR close?
Refs #20935 and #19458.
Rationale for this change
Scalar functions over dictionary-encoded columns have no benchmark coverage:
neither the hand-written dictionary arms (e.g.
reverse's, from #23930) northe cast away from the encoding that every function without preservation pays
(e.g.
encode). This lands the baseline first, so follow-up work generalizingthe arms shows its delta on stable benchmark ids.
What changes are included in this PR?
One criterion benchmark:
reverseover a dictionary of its own per batch (cold) and one sharedacross batches the way a Parquet column chunk delivers them (
warm) — todaythe two cost the same, since an arm cannot reuse anything across batches;
cast_away: the dictionary cast to its value type and the function calledonce per row — what any function without encoding preservation pays;
flat: the same rows with no encoding anywhere, as a bound.encodecannot be called over a dictionary today, so it has nodictionary-typed groups yet.
Are these changes tested?
cargo bench -p datafusion-physical-expr --bench scalar_function_dictionaryruns clean.
Are there any user-facing changes?
No.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NysGeXTG5opiJKBApsAe5H