Leave the consumer group explicitly on shutdown

[delthas]

Stacked on #2818 (BB-835). Review that one first — this branch contains its commit, and the base will move to development/9.5 once it merges.

close() unsubscribed and then waited for the rebalance callback to un-assign before disconnecting. librdkafka delivers no such callback when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — so close() never returned and the pod was SIGKILLed with the member still registered at the broker.

sequenceDiagram
    participant C as BackbeatConsumer
    participant K as Kafka
    Note over C: SIGTERM during a rebalance
    C->>K: unsubscribe()
    Note right of K: postponed — a rebalance<br/>is already in progress
    C->>C: wait for 'unassign' … forever
    Note over C: SIGKILL at the grace period,<br/>no LeaveGroup ever sent
    Note over K: member still registered,<br/>may be elected leader of the next<br/>generation and never SyncGroup

Loading

The group then holds zero partitions until session.timeout.ms (45 s) evicts the member. During a rolling update a rebalance is in progress essentially by construction, since the new pod joins before the old one is told to stop.

Changes

Release the partitions and drop the subscription before closing, so the close path has nothing to hand back to us and the LeaveGroup goes out whatever state the group is in:

before   unsubscribe → wait for a revoke callback → [ drain → commit → unassign ] → disconnect → wait 'disconnected'
after    drain → commit → unsubscribe → unassign → disconnect → wait 'disconnected'

The bracketed steps only ran if librdkafka delivered a revoke callback. It delivers none when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — in both cases the wait never ends. The same steps now run unconditionally, in close() itself.

The order of the last two matters, and the mechanism is a flag rather than the assignment list. Only rd_kafka_cgrp_unsubscribe() sets F_LEAVE_ON_UNASSIGN_DONE, and only unassign_done() — reached from WAIT_UNASSIGN_CALL — consults it to send the LeaveGroup. Un-assigning first therefore clears the assignment without any state change and the LeaveGroup is never armed; unsubscribe() then fires a revoke at us and parks in WAIT_UNASSIGN_CALL, leaving the LeaveGroup gated on a callback round trip that disconnect() is simultaneously blocking on. Unsubscribing first puts us in the one join-state where unassign() is meaningful, so our own call completes it and sends the LeaveGroup before disconnect() is reached.

Draining also means we must stop fetching. Nothing did: every completed task re-armed _tryConsume(), so the pipeline refilled as fast as it emptied and the departure waited on work that arrived after the shutdown began. Against a 3000 message backlog:

concurrency	task	close()	tasks started after close
10	200 ms	6210 ms → 175 ms	301 → 0
10	50 ms	9482 ms → 31 ms	1864 → 0

The same guard ends the self-rescheduling consume loop, which otherwise kept polling a closed client for the lifetime of the process.

• A revoke arriving once the shutdown has started is answered on the spot. librdkafka requires every rebalance callback to be answered, and leaving it for close() to answer by un-assigning later is not enough: one raised after close() has already un-assigned has nothing left to answer it, so the client stays in the rebalance and the disconnect wedges on it. Answering does not cut the drain short — close() still waits for the in-flight work, the partitions are simply handed back sooner, and the offsets that drain exists to commit (BB-758) are unaffected.
• A deferred un-assign from before the shutdown is treated as superseded, and a partition grant arriving mid-close no longer tears down the drain close() installed. Both previously left close() waiting on a callback that could never fire.
• close() no longer waits on an in-flight backlog publish, which could otherwise keep it rescheduling itself every second indefinitely.
• A partition grant arriving after close() has released the partitions is declined rather than accepted, so the disconnect is not handed an assignment it must revoke all over again.
• The final disconnect is bounded, as a backstop only. Measured, that bound firing predicted the process failing to exit in 40 of 40 CI runs — it returned from close() looking successful while leaving a client whose destructor blocks. Answering the rebalance callback above is what stops it firing; the bound is no longer load-bearing.

In-flight work is still drained before the partitions are released, so offsets are committed exactly as before, and that wait keeps the bound it already had through the revoke path (max.poll.interval.ms - 1000) — a wedged task delays the departure no longer than it does today. Draining is skipped once the client is disconnected, since there is then nothing to commit and no partitions to give back.

That bound is inherited, not chosen: at the default max.poll.interval.ms it is ~299 s, far longer than a pod's grace period, so a wedged task is still killed rather than departing cleanly. Replacing it with a deadline derived from the grace period is the budget work, deliberately left out here — BB-854 shortens the drain first.

Verification

Unit tests for the call ordering, completion with no assignment held, the drain wait, the offset-publish skip, watchdog cleanup, and a revoke arriving mid-drain. Each was checked against the previous implementation to confirm it fails there.

Two functional tests against a real broker. The second reproduces the incident: a newcomer joins, and the member being closed has already released its partitions and is waiting to rejoin, so the rebalance is still in progress and nothing will revoke back to it.

	before	after
close with the group settled	3994 ms	3994 ms
close during a rebalance	close() never returns	10008 ms

The first passes either way — it guards the unsubscribe → unassign ordering, since reverting that gates the LeaveGroup on a callback disconnect() is blocking on and the takeover falls off the 45 s cliff. The second is the regression guard.

Whether the process then exits was measured separately, 8 full runs of the lib suite per tree: 9.5 exits 8/8, this branch 8/8. Before the callback fix it was 1/8, and every wedged run had the disconnect bound firing first.

Beyond that, a pod-level census on real CI runners: a pod is terminated in each of four states, and what both pods actually processed is reconciled against what was produced. 288 iterations per round, two arms measured by identical harness code with only lib/ differing.

	this branch	before
exits without being SIGKILLed	100%	0%
median takeover	3.3 s	16.3 s
takeover over 10 s	9%	100%
takeover over 30 s	4%	12%
messages lost	0	0

n = 62 / 77 valid samples. Across the two rounds (290 valid samples) no iteration lost a message, and none ever committed an offset past work it had not finished — the property that matters more than the timing.

The census also found a defect that review had not: the rebalance callback still accepted a partition grant after close() had handed the partitions back, so the disconnect had to revoke them again, wedged, hit its 5 s bound, and returned without a LeaveGroup — the surviving members then waited out session.timeout.ms. That was 6 of 69 iterations; declining the grant took the SIGKILL rate to zero and halved the residual.

What remains is ~4% of departures still landing at 40-45 s, i.e. eviction rather than a departure. That signature is the orphaned member id tracked upstream as BB-843, not this path, but that attribution is a hypothesis rather than something these runs establish.

Issue: BB-833

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.91%. Comparing base (6c6acda) to head (3a58ca4).

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/BackbeatConsumer.js	96.45% <100.00%> (+2.65%)	⬆️
lib/constants.js	100.00% <ø> (ø)

... and 3 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	80.27% <ø> (ø)
Core Library	82.24% <100.00%> (+0.94%)	⬆️
Ingestion	70.09% <ø> (ø)
Lifecycle	80.46% <ø> (ø)
Oplog Populator	85.83% <ø> (ø)
Replication	62.01% <ø> (ø)
Bucket Scanner	85.76% <ø> (ø)

@@                          Coverage Diff                           @@
##           improvement/BB-835/rebalance-guard    #2819      +/-   ##
======================================================================
+ Coverage                               75.51%   75.91%   +0.40%     
======================================================================
  Files                                     200      200              
  Lines                                   13946    13990      +44     
======================================================================
+ Hits                                    10531    10621      +90     
+ Misses                                   3405     3359      -46     
  Partials                                   10       10              

Flag	Coverage Δ
api:retry	9.06% <0.00%> (-0.03%)	⬇️
api:routes	8.82% <0.00%> (-0.03%)	⬇️
bucket-scanner	85.76% <ø> (ø)
ft_test:queuepopulator	11.05% <7.81%> (+1.87%)	⬆️
ingestion	12.21% <1.56%> (-0.04%)	⬇️
lib	9.18% <96.87%> (+0.37%)	⬆️
lifecycle	19.45% <60.93%> (+0.12%)	⬆️
notification	1.01% <0.00%> (-0.01%)	⬇️
oplogPopulator	0.13% <0.00%> (-0.01%)	⬇️
replication	19.08% <60.93%> (+0.20%)	⬆️
unit	55.53% <100.00%> (+0.29%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.