[Dataflow Streaming] [Multi Key] Flush streaming sinks at key boundaries and bundle completion - #39961
Draft
arunpandianp wants to merge 2 commits into
Draft
[Dataflow Streaming] [Multi Key] Flush streaming sinks at key boundaries and bundle completion#39961arunpandianp wants to merge 2 commits into
arunpandianp wants to merge 2 commits into
Conversation
…ies and bundle completion When multi-key bundling is enabled in Dataflow Streaming Engine, outputs (productions) emitted during a key's processElement lifecycle must be attributed to that specific key's WorkItemCommitRequest, while outputs emitted during DoFn finishBundle must be attributed to the bundle-level MultiKeyWorkItemCommitRequest. This change: 1. Adds finishKey(@nullable Object key) to the SinkWriter interface and implements it across streaming sinks (WindmillSink, PubsubSink, PubsubDynamicSink) and non-streaming sinks. 2. In WindmillSink, PubsubSink, and PubsubDynamicSink: - When multi-key bundles are enabled: flushes to the active key's commit builder in finishKey; flushes remaining finishBundle outputs to bundle-level outputs (bundleOutputMessages / bundlePubsubMessages) in close(). - When multi-key bundles are disabled: finishKey does not flush; close() flushes all outputs directly into the single key's commit request. - Updates abort() to cleanly discard buffered messages and reset internal streams without flushing. 3. In StreamingModeExecutionContext and StreamingWorkScheduler, tracks bundleOutputMessages and bundlePubsubMessages and attaches them to MultiKeyWorkItemCommitRequest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.