Skip to content

fix: refuse memory-limited NestedLoopJoin fallback for left-emission joins with a multi-partition probe side - #24675

Open
ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:nlj-gate-left-family
Open

fix: refuse memory-limited NestedLoopJoin fallback for left-emission joins with a multi-partition probe side#24675
ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:nlj-gate-left-family

Conversation

@ranflarion

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

The memory-limited fallback returns wrong rows for join types whose final emission reads the visited-left bitmap (LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK) when the probe side has more than one partition. The fallback rebuilds a per-partition JoinLeftData per left chunk with probe_threads_counter == 1 and a per-partition bitmap, so every partition believes it is the last to finish probing and emits its own unmatched set, from a bitmap that only saw its own right rows. Unmatched rows are emitted once per partition, and rows matched only in another partition are emitted as unmatched.

Proof, using this file's own build_table / prepare_join_filter / multi_partitioned_join_collect fixtures (4 right partitions) with a 100-byte memory limit versus unbounded:

for join_type in [JoinType::Left, JoinType::LeftAnti] {
    // unbounded memory -> correct; 100-byte pool -> memory-limited fallback
    let (_, correct, _) = multi_partitioned_join_collect(left, right, &join_type, Some(filter), ample_ctx).await?;
    let (_, wrong, _)   = multi_partitioned_join_collect(left, right, &join_type, Some(filter), tight_ctx).await?;
}
// Left:     correct=19  memory_limited=49
// LeftAnti: correct=1   memory_limited=31

The existing gate covers only FULL. This extends it to every need_produce_result_in_final type, so an over-budget build side fails with ResourcesExhausted instead of returning wrong rows. Right-side emission types (RIGHT, RIGHT SEMI, RIGHT ANTI, RIGHT MARK) keep the fallback: each partition owns its right rows exclusively, so their bitmaps are complete per partition. Proper cross-partition coordination of the left bitmap, which would re-enable the fallback for these types, is #22038.

What changes are included in this PR?

The full_join_multi_partition condition in NestedLoopJoinExec::execute becomes need_produce_result_in_final(self.join_type) && right_partition_count > 1, with the comment updated to describe the failure mode. test_overallocation moves LEFT/LEFT SEMI/LEFT ANTI/LEFT MARK from the succeed-via-fallback group (which collected but never checked the row values) into the must-OOM group alongside FULL.

Are these changes tested?

test_overallocation now asserts the ResourcesExhausted refusal for all five gated join types with a multi-partition probe side, and still asserts fallback success for Inner and the right-emission types. All 42 nested_loop_join tests pass; ./dev/rust_lint.sh is clean.

Are there any user-facing changes?

Left-family nested loop joins with a multi-partition probe side whose build side exceeds the memory budget now fail with ResourcesExhausted instead of returning incorrect results. No API changes.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Correctness guard: disable memory-limited NestedLoopJoin fallback for LEFT-family joins with multi-partition right side

1 participant