Skip to content

feat: add narrow group-contiguous source property - #24698

Draft
xavlee wants to merge 2 commits into
apache:mainfrom
xavlee:feat/issue-24438-group-contiguous-property
Draft

feat: add narrow group-contiguous source property#24698
xavlee wants to merge 2 commits into
apache:mainfrom
xavlee:feat/issue-24438-group-contiguous-property

Conversation

@xavlee

@xavlee xavlee commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR relate to?

Rationale for this change

Sources that combine ordered, disjoint logical runs can know that each complete grouping tuple occupies one contiguous range in an output stream, even when tuple values reset between runs. This PR gives sources a precise way to express that fact and defines its propagation through the first row-transparent operators in the physical plan.

Property contract

ExecutionPlan::group_contiguous_exprs describes one complete composite tuple. Within each output partition, every distinct tuple occupies at most one contiguous range of rows. Tuple values may appear in any order.

The source owns this correctness assertion. AggregateExec can use it to emit a tuple after the stream advances to the next tuple.

What changes are included?

The property follows this explicit lifecycle:

  1. DataSource::group_contiguous_exprs declares the tuple, with an empty default.
  2. DataSourceExec exposes the source declaration and includes it in explain output.
  3. ProjectionExec projects the complete tuple when every component maps to the output schema; an incomplete mapping produces the empty default.
  4. CooperativeExec delegates the tuple because it preserves the input row sequence.
  5. Other execution plans inherit the empty ExecutionPlan default.

group_contiguous_exprs is represented separately from PlanProperties, making forwarding an explicit operator decision. ProjectionExec recomputes it during child replacement, including the same-PlanProperties fast path.

Stack

This draft carries #24697 as its first commit and the property implementation as its second commit (6bd697fe8). Once #24697 merges, this branch can be rebased to the property commit.

The Linear -> None -> Final aggregate regression introduced by #24697 retains its baseline behavior in this propagation layer. The following consumer commit updates that test to exercise the new assertion.

Are these changes tested?

Tests cover:

Validation:

  • cargo test -p datafusion-physical-plan --lib
  • cargo test -p datafusion-datasource --lib
  • strict clippy and rustdoc for both crates
  • cargo-semver-checks for both crates

Are there any user-facing changes?

ExecutionPlan and DataSource implementors receive default-empty group_contiguous_exprs methods and can override them to declare a complete contiguous tuple.

@codecov-commenter

codecov-commenter commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.44920% with 118 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.43%. Comparing base (4fcaa01) to head (6bd697f).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/aggregates/mod.rs 64.65% 36 Missing and 5 partials ⚠️
datafusion/datasource/src/source.rs 52.70% 34 Missing and 1 partial ⚠️
datafusion/physical-plan/src/test.rs 55.31% 20 Missing and 1 partial ⚠️
datafusion/physical-plan/src/projection.rs 81.57% 4 Missing and 10 partials ⚠️
datafusion/physical-plan/src/coop.rs 76.92% 0 Missing and 3 partials ⚠️
...tafusion/physical-plan/src/aggregates/order/mod.rs 83.33% 2 Missing ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs 50.00% 0 Missing and 1 partial ⚠️
...ysical-plan/src/aggregates/ordered_final_stream.rs 92.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24698      +/-   ##
==========================================
- Coverage   81.45%   81.43%   -0.03%     
==========================================
  Files        1118     1119       +1     
  Lines      399685   400740    +1055     
  Branches   399685   400740    +1055     
==========================================
+ Hits       325576   326337     +761     
- Misses      55103    55305     +202     
- Partials    19006    19098      +92     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xavlee
xavlee force-pushed the feat/issue-24438-group-contiguous-property branch from b2ceb1a to 6bd697f Compare August 26, 2026 19:20
@github-actions github-actions Bot added datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants