Skip to content

refactor: separate aggregate group completion from input ordering - #24697

Draft
xavlee wants to merge 1 commit into
apache:mainfrom
xavlee:refactor/aggregate-group-completion-mode
Draft

refactor: separate aggregate group completion from input ordering#24697
xavlee wants to merge 1 commit into
apache:mainfrom
xavlee:refactor/aggregate-group-completion-mode

Conversation

@xavlee

@xavlee xavlee commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR relate to?

Rationale for this change

InputOrderMode records how aggregate input is ordered relative to its grouping expressions and also selects the runtime mechanism that determines when groups are complete. This PR gives the runtime capability its own private representation, allowing aggregate execution to consume group-completion evidence while InputOrderMode continues to describe ordering.

What changes are included?

  • Introduce private GroupCompletionMode::{None, Partial, Full} variants.
  • Derive GroupCompletionMode from InputOrderMode during AggregateExec construction.
  • Pass GroupCompletionMode through ordered aggregate tables, streams, and spill replay paths.
  • Use InputOrderMode for required ordering, output ordering, and maintains_input_order metadata.
  • Keep GroupOrdering::try_new(&InputOrderMode) as the public compatibility entry point.
  • Add an unbounded-input regression fixture that establishes the current Linear -> None -> Final behavior and observes the aggregate remaining pending before EOF.

The conversion remains one-to-one in this PR:

InputOrderMode::Linear                    -> GroupCompletionMode::None
InputOrderMode::PartiallySorted(indices) -> GroupCompletionMode::Partial(indices)
InputOrderMode::Sorted                    -> GroupCompletionMode::Full

This mapping preserves the existing execution paths and plan properties.

Stack

This is the first of three commits for #24438. It establishes the private aggregate capability and the blocking Linear baseline. The final consumer commit updates that same regression into paired contiguous and default-empty cases.

Are these changes tested?

Yes:

  • linear_aggregate_waits_for_input_end asserts Linear, None, Final, the ordinary hash stream, and pending output before EOF
  • cargo test -p datafusion-physical-plan --lib (1,785 tests)
  • strict clippy for datafusion-physical-plan, all targets and features
  • cargo fmt --all -- --check

Are there any user-facing changes?

This is an internal refactor. GroupCompletionMode is crate-private, and GroupOrdering::try_new(&InputOrderMode) remains the public entry point.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 26, 2026
@codecov-commenter

codecov-commenter commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.04969% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.44%. Comparing base (4fcaa01) to head (0e05631).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/aggregates/mod.rs 64.65% 36 Missing and 5 partials ⚠️
...tafusion/physical-plan/src/aggregates/order/mod.rs 83.33% 2 Missing ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs 50.00% 0 Missing and 1 partial ⚠️
...ysical-plan/src/aggregates/ordered_final_stream.rs 92.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24697      +/-   ##
==========================================
- Coverage   81.45%   81.44%   -0.02%     
==========================================
  Files        1118     1119       +1     
  Lines      399685   400533     +848     
  Branches   399685   400533     +848     
==========================================
+ Hits       325576   326207     +631     
- Misses      55103    55245     +142     
- Partials    19006    19081      +75     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xavlee
xavlee force-pushed the refactor/aggregate-group-completion-mode branch from 689f76d to 0e05631 Compare August 26, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants