Skip to content

fix: GROUP BY sort record carries only needed columns, not full row - #670

Merged
iheitlager merged 1 commit into
mainfrom
fix/665-group-by-sort-record-narrowing
Aug 30, 2026
Merged

fix: GROUP BY sort record carries only needed columns, not full row#670
iheitlager merged 1 commit into
mainfrom
fix/665-group-by-sort-record-narrowing

Conversation

@iheitlager

Copy link
Copy Markdown
Member

Summary

Verification

SELECT bucket, COUNT(*), SUM(x) FROM bench_data GROUP BY bucket's sorter MakeRecord is now MakeRecord|0|2 (bucket, x) — byte-for-byte matching sqlite3 3.53.4's own EXPLAIN output for the same query (also mkrec(r[11..12]), 2-wide), where it was previously 6-wide with 4 Null placeholders.

Test plan

  • Added group_by_excludes_unreferenced_columns_from_the_sort_record — a 4-column fixture with one column referenced nowhere, asserting the sort record's MakeRecord width is exactly 3 (GROUP BY key + plain arbitrary-row column + aggregate argument, excluding the unreferenced column), plus correctness against the real oracle.
  • cargo test --release — full suite green (980 tests, 0 failed).
  • make test-corpus — 380 tests, byte-exact against the real oracle, all pass.
  • make test-parity — 19 pass, 5 skipped (unimplemented V-blocks), 0 failed.
  • cargo clippy --all-targets / cargo fmt --check — clean.
  • cargo bench --bench crud -- read_group_by_agg — ~3.3% faster on the standard 6-column bench fixture (modest, since the fixture isn't very wide — the correctness win, matching oracle's opcode shape exactly, is the more solid deliverable; the remaining ~2.7x gap vs oracle on this scenario has a different root cause outside this ticket's scope).

Closes #665

spend: ~1x estimate — more involved than the issue's "small" estimate, since true record narrowing required a compacted synthetic schema/index-translation layer rather than just reusing #506's existing needed-columns analysis.

…665)

#506 already skipped the real per-row Column/Rowid read for a column
compile_grouped_scan's needed-columns analysis found unused, but kept
the sort record itself full-width (a cheap Null placeholder in place
of each skipped column) — every downstream reader of the pass-2 pseudo
cursor assumed a 1:1 position match with schema's original column
layout, so truly narrowing the record would have broken that mapping.

This closes that gap: pass 1's MakeRecord/SorterInsert now includes
only needed_order's columns (no Null padding), pass 2 resolves GROUP BY
keys and aggregate arguments against a compacted synthetic TableSchema
(so name-based Scope/compile_value resolution translates automatically),
and the couple of by-index reads (the GROUP BY key's OrderByTarget::Column
case, and the arbitrary-row snapshot loop) translate through an explicit
original-index -> compacted-position map. flush_group's own synthetic
schema/snapshot_regs stay full-width and untouched — that's a separate,
per-group (not per-row) pseudo cursor unaffected by this ticket.

hash.rs's unwired #631 spike keeps its own pre-#665 copy of the old
Null-padded helper, since threading the same compaction through a dead
code path isn't worth it.

Verified against oracle: `read_group_by_agg`'s sorter MakeRecord is now
2-wide (bucket, x), byte-for-byte matching sqlite3's own EXPLAIN. Full
test suite, corpus (380 tests), and parity suites all pass unchanged.

Closes #665

spend: ~1x estimate (more involved than the issue's "small" estimate —
true record narrowing required a compacted synthetic schema/index-
translation layer, not just skipping the analysis #506 already had)
@iheitlager
iheitlager force-pushed the fix/665-group-by-sort-record-narrowing branch from e84ee35 to 22c1750 Compare August 30, 2026 12:15
@iheitlager
iheitlager merged commit 00cbc2f into main Aug 30, 2026
6 checks passed
@iheitlager
iheitlager deleted the fix/665-group-by-sort-record-narrowing branch August 30, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: GROUP BY/aggregate sorter records are full-row-width instead of narrowed to needed columns

1 participant