Skip to content

Account for transformer-retained batches in symmetric hash join #24364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kosiew wants to merge 25 commits into
base: main
Choose a base branch
from
+833 −32
Open

Account for transformer-retained batches in symmetric hash join #24364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
716b46c
**feat(transformer, stream): add RecordBatchMemoryCounter for retaine…
kosiew Aug 14, 2026
958cc74
refactor(transformer): deduplicate retained-batch accounting in priva…
kosiew Aug 14, 2026
71eb7e9
fix(symmetric_hash_join): preserve retained batch memory accounting
kosiew Aug 14, 2026
4e9965c
feat: improve recursive unique ArrayRef overhead accounting and cover…
kosiew Aug 14, 2026
22f6a66
fix(stream-boundary): add nested shared-child regression test for Noo…
kosiew Aug 14, 2026
188fea5
docs: document BatchTransformer::count_memory shared counter and dedu…
kosiew Aug 14, 2026
709a488
feat(memory): add table-driven shared-child cases for List, Map, Unio…
kosiew Aug 14, 2026
fb97c27
feat: add test‑private assertion helper and remove post‑construction …
kosiew Aug 14, 2026
792fa41
test(symmetric_hash_join): add tests for transformer output exhaustio…
kosiew Aug 14, 2026
a31dc42
feat(pool): replace fixed pool limit with dynamic peak profiling and …
kosiew Aug 15, 2026
b039d6c
fix(pool): adjust limit to observed peak - 1 and clarify ownership co…
kosiew Aug 15, 2026
3106c2a
docs(common/utils): update documentation to clarify array counting in…
kosiew Aug 15, 2026
c83dedb
test: replace noisy whole‑exec OOM test, add exact stream‑boundary bo…
kosiew Aug 15, 2026
4769cb7
Fix whole-stream OOM assertion
kosiew Aug 15, 2026
bb6e5ef
fix(datafusion/physical-plan/src/joins/symmetric_hash_join): adjust m…
kosiew Aug 15, 2026
48ef0d5
feat: bound union once, dedup transformer test assertion, add change_…
kosiew Aug 15, 2026
52e5f00
refactor: change_deltas() now projects under one mutex lock, removing…
kosiew Aug 15, 2026
eb20ffd
refactor(join): remove duplicate tests and simplify recording pool
kosiew Aug 18, 2026
00f20ed
fix(symmetric_hash_join): restore direct stream alias tests and exact…
kosiew Aug 18, 2026
f5270d3
Merge branch 'main' into memcalc-03-23393
kosiew Aug 18, 2026
901b57a
fix: fix reservation leak and stream memory metric reset, add retaine…
kosiew Aug 18, 2026
200662b
fix(runend,union): remove object‑identity branches/tests, add size_wi…
kosiew Aug 20, 2026
a112718
fix(stream, traversal): preserve reservation/metric in terminal strea…
kosiew Aug 26, 2026
db7a6f9
fix: resolve test gaps
kosiew Aug 26, 2026
d922d3d
fix(common/utils/memory): use std::iter::once for iter_on_single_items
kosiew Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading