feat: stream exact group-contiguous aggregates

[xavlee]

Which issue does this PR close?

• Closes Supporting analytics over large amounts of pre‑partitioned data #24438.

Rationale for this change

Large pre-partitioned datasets can contain many more logical runs than available CPU cores. A source can combine several runs into each DataFusion output partition while preserving the invariant that every complete grouping tuple occupies one contiguous range. Capturing that invariant lets the first aggregate emit completed groups as the stream advances and bound its live group state to the active tuple range.

For example, a source can certify (key, date_bin(time)) after validating its logical run boundaries. The combined stream may reset tuple values between runs while every distinct tuple still appears in one range.

What changes are included?

This three-commit stack implements the following path:

DataSource / custom ExecutionPlan
    group_contiguous_exprs = complete composite tuple
                |
                +--> ProjectionExec maps the complete tuple
                |
                +--> CooperativeExec preserves the row sequence
                |
                +--> first AggregateExec consumes an exact grouping-tuple match

AggregateExec compares the asserted tuple with the complete GROUP BY tuple using input equivalence properties. Matching supports equivalent expressions and tuple permutation.

An exact match establishes GroupCompletionMode::Full. InputOrderMode continues to describe the input ordering, so an unsorted input retains InputOrderMode::Linear. The aggregate selects DataFusion's existing full group-completion tables and streams, reports incremental emission according to its input pipeline, and emits each group when the next tuple begins.

The first aggregate consumes the assertion and produces the default-empty assertion on its output. Its benefits_from_input_partitioning result protects the contiguous input path when that assertion supplies the stronger completion capability. Aggregates using ordering-derived completion retain their existing partitioning calculation.

Stack

1. refactor: separate aggregate group completion from input ordering #24697 introduces the private group-completion capability and establishes the Linear -> None -> Final blocking regression.
2. feat: add narrow group-contiguous source property #24698 introduces the source, projection, and cooperative propagation lifecycle while retaining that baseline.
3. This PR's final commit (b9900193f) adds the exact-match aggregate consumer and updates the baseline into paired asserted and default-empty cases.

This draft carries all three commits so reviewers can inspect the integrated behavior. As the prerequisite PRs merge, the branch can be rebased to the remaining commit.

Relationship to #24501

This PR establishes within-stream group completion. #24501 establishes cross-partition distribution satisfaction for Range([timestamp]) projected through date_bin and date_trunc. Together, those capabilities support a single-stage partitioned streaming aggregate when the source supplies both facts.

Are these changes tested?

Coverage includes:

• the refactor: separate aggregate group completion from input ordering #24697 blocking baseline updated into paired cases where both inputs remain InputOrderMode::Linear, the asserted input becomes Full and emits before EOF, and the default-empty input remains None and pending;
• non-monotonic logical-run resets using a real date_bin expression;
• exact tuple matching with permutation and equivalence support;
• complete-tuple and grouping-set boundaries;
• output ordering, emission type, and assertion consumption;
• ordinary and contiguous benefits_from_input_partitioning behavior;
• same-PlanProperties child replacement;
• DataSourceExec, ProjectionExec, and CooperativeExec forwarding; and
• the default-empty execution-plan path.

Validation on the three-commit stack:

• cargo test -p datafusion-physical-plan --lib (1,789 tests)
• cargo test -p datafusion-datasource --lib (175 tests)
• strict clippy for both affected crates, all targets and features
• strict rustdoc for both affected crates
• cargo-semver-checks for both affected crates
• range_sorted_time_bin_agg.slt on a combined branch with feat: skip hash shuffle for date_bin/date_trunc on Range([timestamp]) #24501

Are there any user-facing changes?

ExecutionPlan and DataSource implementors can declare a complete tuple through group_contiguous_exprs. An exact aggregate grouping match activates incremental group completion within each input partition.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.02428% with 158 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.43%. Comparing base (4fcaa01) to head (b990019).
⚠️ Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/aggregates/mod.rs	79.80%	48 Missing and 33 partials ⚠️
datafusion/datasource/src/source.rs	52.70%	34 Missing and 1 partial ⚠️
datafusion/physical-plan/src/test.rs	55.31%	20 Missing and 1 partial ⚠️
datafusion/physical-plan/src/projection.rs	81.57%	4 Missing and 10 partials ⚠️
datafusion/physical-plan/src/coop.rs	76.92%	0 Missing and 3 partials ⚠️
...tafusion/physical-plan/src/aggregates/order/mod.rs	83.33%	2 Missing ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs	50.00%	0 Missing and 1 partial ⚠️
...ysical-plan/src/aggregates/ordered_final_stream.rs	92.85%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24497      +/-   ##
==========================================
- Coverage   81.45%   81.43%   -0.03%     
==========================================
  Files        1118     1119       +1     
  Lines      399685   401024    +1339     
  Branches   399685   401024    +1339     
==========================================
+ Hits       325576   326582    +1006     
- Misses      55103    55321     +218     
- Partials    19006    19121     +115     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alamb]

This PR models group contiguity as its own correctness property. It is intentionally neither an ordering guarantee nor an output-distribution guarantee.

Why does it need a new property? I think this notion is designed to be covered y the existing ordering / monotonic analyses

[alamb]

Oh, I see, somehow the external system knows the data is not sorted but is non overlapping

I think this is going to be really hard to manage / ensure through the plan -- we will need to ensure that every operator properly reports if it will propagate this property or not

I am not sure this is something we want to complicate datafusion with

[alamb]

@NGA-TRAN can you help evaluate this PR for its impact and if we will be able to keep this property in tact?

[xavlee]

Hi Andrew, thanks for taking an initial look. Apologies that this is still verymuch a draft.

I agree that propagating another physical property through the plan would be a little complicated. I was hoping we would narrow the group_contiguous_exprs assertion s.t. it:

• is declared explicitly by the data source
• defaults to absent on every execution operator
• may pass only through ProjectionExec when every expression maps
• is consumed by the only the first AggregateExec (and not present on the aggregate output)

[NGA-TRAN]

@alamb : I added a comment in the ticket

@NGA-TRAN can you help evaluate this PR for its impact

The impact of this PR is huge for telemetry use cases of AI frontiers as I described in the comment above

and if we will be able to keep this property in tact?

This property, like some properties, will be no longer available after certain operators so I think it would wok the same. I agree the propagation is a bit more complicated than usual but we work together to split this PR into smaller ones and will look into design carefully to avoid a lot of side effect. I think we would be able to make the design simpler