Answer every rebalance callback raised during shutdown - #2835
Draft
delthas wants to merge 1 commit into
Draft
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 1 file with indirect coverage changes
@@ Coverage Diff @@
## improvement/BB-833/stop-consuming-on-shutdown #2835 +/- ##
=================================================================================
- Coverage 75.84% 75.80% -0.04%
=================================================================================
Files 200 200
Lines 13976 13983 +7
=================================================================================
Hits 10600 10600
- Misses 3366 3373 +7
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This was referenced Sep 1, 2026
librdkafka requires every rebalance callback to be answered, and the shutdown path answered none of them. Each case fails differently, and each leaves the client parked in the rebalance, so the disconnect wedges on it and the process can never exit. A revoke was left for close() to answer by un-assigning later. That holds for the revoke unsubscribe() itself raises, but one arriving after close() has already un-assigned has nothing left to answer it. Answering here does not cut the drain short: close() still waits for the in-flight work, the partitions are just handed back sooner. A grant was accepted, which left the disconnect an assignment to revoke all over again. It is declined instead -- but assign() is refused once disconnect() has started, since the binding gates it on isConnected() while permitting unassign() for the whole close, so the decline falls back to unassign(). librdkafka coerces an assign into a full unassign during termination anyway. Measured across 8 full lib-suite runs per arm on CI: whenever the disconnect bound fired the process failed to exit, 40 times out of 40. Answering the callbacks takes the suite from 1 of 8 runs exiting to 8 of 8, matching 9.5 itself, with no message lost and nothing committed past unprocessed work across 240 iterations. Issue: BB-833
delthas
force-pushed
the
improvement/BB-833/answer-rebalance-callbacks
branch
from
September 1, 2026 09:15
a5595c4 to
44ebdf4
Compare
delthas
marked this pull request as ready for review
September 1, 2026 09:35
francoisferrand
requested changes
Sep 1, 2026
francoisferrand
left a comment
francoisferrand
left a comment
Contributor
There was a problem hiding this comment.
could this have been created by the first PR, which changes the unsubscribe sequence?
If we are not subscribed, we should not get assigned: so doing it in the shutdown avoids the issue? Or is it just the race condition where we unsubscribe right before processing the assign partitions event?
Comment on lines
+827
to
+832
| // assign() is refused once disconnect() has started -- the | ||
| // binding gates it on isConnected(), which is already false -- | ||
| // while unassign() is explicitly permitted while closing, and | ||
| // librdkafka coerces an assign into a full unassign during | ||
| // termination anyway. Leaving the callback unanswered is what | ||
| // parks the client in the rebalance. |
Contributor
There was a problem hiding this comment.
based on this, shoud we not just always call unassign() in that case, instead of trying to assign([]) first?
| return; | ||
| } | ||
|
|
||
| this._setDrain(null); |
Contributor
There was a problem hiding this comment.
should not be needed, c.f. first PR in the stack: a new assign cannot happen during an earlier unassign, so setDrain should stay managed in a single place (during partition revoke).
delthas
marked this pull request as draft
September 1, 2026 14:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
librdkafka requires every rebalance callback to be answered, and the shutdown path answered none of them. Each case fails differently, and each leaves the client parked in the rebalance, so the disconnect wedges on it and the process can never exit.
flowchart TD R["rebalance callback raised<br/>after the shutdown began"] --> Q{"which"} Q -->|revoke| U["un-assign — answers it"] Q -->|grant| A["decline with an empty assign"] A --> F{"disconnect already started?"} F -->|"yes — assign is refused"| U2["un-assign — still permitted"] F -->|no| D["declined, nothing to revoke later"] U --> OK["client leaves the rebalance,<br/>disconnect completes"] U2 --> OK D --> OKChanges
A revoke was left for
close()to answer by un-assigning later. That holds for the revokeunsubscribe()itself raises, but one arriving afterclose()has already un-assigned has nothing left to answer it. Answering here does not cut the drain short:close()still waits for the in-flight work, the partitions are just handed back sooner, and the offsets that drain exists to commit (BB-758) are unaffected.A grant was accepted, which left the disconnect an assignment to revoke all over again. It is declined instead — but
assign()is refused oncedisconnect()has started, since the binding gates it onisConnected()while permittingunassign()for the whole close, so the decline falls back tounassign(). librdkafka coerces an assign into a full unassign during termination anyway.Verification
Three unit tests: a grant during shutdown is declined, a grant is still answered when
assign()is refused mid-close, and a revoke during shutdown is answered. Each was checked against the previous implementation to confirm it fails there.This is the defect that decides whether the process exits at all, and it was found by measurement rather than review. Across 8 full runs of the lib suite per arm on CI:
The bounded disconnect was returning from
close()looking successful while leaving a client whose destructor blocks. Answering the callbacks is what stops it firing, so the bound is a backstop rather than load-bearing.The pod-level census found the grant case independently: the callback still accepted a partition grant after
close()had handed the partitions back, so the disconnect had to revoke them again, wedged, hit its 5 s bound, and returned without aLeaveGroup— the surviving members then waited outsession.timeout.ms. That was 6 of 69 iterations; declining took the SIGKILL rate to zero and halved the residual. No message was lost and nothing was committed past unprocessed work across 240 iterations.End-to-end measurement of the whole stack is in #2819.
Issue: BB-833