backup: keep a concurrently renewed session alive on a failed renewal - #1207
backup: keep a concurrently renewed session alive on a failed renewal#1207bootjp wants to merge 2 commits into
Conversation
A renewal that fails part way through its fan-out released every group pin and forgot the session unconditionally. If another renewal had already succeeded, that teardown left its caller holding a token whose pins were gone, with retention free to compact the versions underneath it -- a backup silently invalid while the caller believed it renewed. The session now carries a generation that every accepted renewal advances, and a failing attempt only cleans up while the generation it started from is still current. A failure with no concurrent success behaves exactly as before. Also compensate an ambiguous capacity reservation. A capacity rejection is definitive, but any other proposal error may be a reservation that committed with only the response lost, and BeginBackup returned without unreserving it -- holding one of the few global active-backup slots until its TTL for a backup no caller ever received. Unreserve is idempotent and keyed by the pin, so compensating costs nothing when the reservation never landed. Claude-Session: https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedNext included review available in 26 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude finished @bootjp's task in 2m 45s —— View job Claude finished @bootjp's taskTodo:
ReviewI traced both fixes end-to-end against the FSM apply paths and the existing test helpers; both hold up. Generation guard (
Ambiguous-reservation compensation (
Tests (
Overall: the fix matches the two Codex findings from #1056 precisely, the reasoning in the PR body checks out against the code, and the new tests are well-targeted (revert-checked, as claimed). I don't see any correctness gaps. Only a cosmetic branch-structure nit above, not worth a follow-up on its own. Note: I was not able to run |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82e8b2bd92
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if live && !s.forgetBackupSessionAtGeneration(tok.pinID, generation) { | ||
| return |
There was a problem hiding this comment.
If EndBackup removes the session while this renewal is in flight, forgetBackupSessionAtGeneration returns false because the entry is absent, and this early return skips all compensation. A reserve or partial pin fan-out from the renewal can commit after EndBackup's release/unreserve and then remain active until the new TTL, blocking compaction and consuming one of the limited backup slots. Only a still-live session with a different generation proves that another renewal owns the pins; an absent or closing session still needs release/unreserve compensation.
Useful? React with 👍 / 👎.
| // slots until its TTL for a backup no caller ever received. Unreserve is | ||
| // idempotent and keyed by this pin, so compensating costs nothing when | ||
| // the reservation never landed. | ||
| s.compensateBackupRelease(controlGroup, nil, pinID) |
There was a problem hiding this comment.
This compensation is bypassed whenever backupCapacityReservationFull sees any gRPC ResourceExhausted status, even though the forwarding path preserves that status for transport or proxy failures as well as the definitive ErrTooManyActiveBackups apply response. If a local attempt committed the reservation but a subsequent forwarded retry fails with infrastructure-level ResourceExhausted, BeginBackup returns without unreserving the new pin and consumes a backup slot until its TTL. Restrict the definitive-capacity case to the actual capacity error, or issue the idempotent unreserve for ambiguous ResourceExhausted responses too.
Useful? React with 👍 / 👎.
| if live && !s.forgetBackupSessionAtGeneration(tok.pinID, generation) { | ||
| return |
There was a problem hiding this comment.
When concurrent renewals request different TTLs, a failed long-TTL attempt can first extend the control reservation and some group pins to nearly an hour, while a short-TTL attempt completes and advances the generation. The failed attempt then takes this early return, but the live session records only the successful attempt's shorter deadline; after that token/session expires, the failed attempt's reservation and partial pins remain orphaned until their longer deadline, consuming capacity and blocking compaction. Preserving the successful renewal therefore also needs to reconcile or later clean up deadlines installed by the failed attempt rather than treating every generation change as sufficient cleanup.
Useful? React with 👍 / 👎.
1 participant
Follow-up to #1056, which merged at
afe538e4with two Codex findings still open against that commit.adapter/admin_backup.go:440— P1, renewal raceA renewal that fails part way through its fan-out released every group pin and forgot the session unconditionally. If another renewal had already succeeded, that teardown left its caller holding a token whose pins were gone, with retention free to compact the versions underneath it — a backup silently invalid while the caller believed it renewed.
The report offered "serialize renewals per pin, or make cleanup conditional on the failing attempt still owning the session generation". Serializing alone does not fix it: even fully serialized, a later failed renewal still tears down an earlier successful one. So this takes the second option.
backupSessioncarries agenerationthat every accepted renewal advances inextendBackupSession,RenewBackupcaptures it before the fan-out, andabandonFailedRenewalcleans up only while that generation is still current.This is the same hazard class as the existing
closeBackupSessioncomment ("leaving the session live across those proposals let an overlapping renewal commit a fresh Pin after the Release").adapter/admin_backup.go:344— P2, ambiguous reservationBackupReservecommits, the response is lost or the context expires,proposeBackupAllerrors, andBeginBackupreturns without proposingBackupUnreserve. The unacknowledged reservation holds one of the few global active-backup slots until its TTL for a backup no caller received.A capacity rejection stays uncompensated — that one is definitive, nothing was reserved. Every other error is ambiguous, and
compensateBackupRelease(controlGroup, nil, pinID)already does exactly the right thing: with no data groups it proposes only the unreserve, which is idempotent and keyed by this pin.Behaviour change
TestRenewBackupStillReleasesWhenItOwnsTheSession).BeginBackupnow proposes one extra unreserve on non-capacity reservation failures.Risk
Confined to the renewal and reservation error paths. The generation is process-local session state, not replicated, so no wire or on-disk format changes.
Tests
adapter/admin_backup_renew_race_test.go:TestRenewBackupKeepsAConcurrentlyRenewedSession— the concurrent success is injected from inside the failing renewal's own fan-out, after it captured the generation:onProposefires on the reserve preceding the failing pin and calls the realextendBackupSession, which is what a successful renewal ends with. Asserts the session survives and that no release or unreserve was proposed.TestRenewBackupStillReleasesWhenItOwnsTheSession— the ordinary failure path still tears down.TestBeginBackupUnreservesAmbiguousReservationFailures— a non-capacity reserve failure proposes the unreserve.Both fixes revert-checked: reverting the generation guard fails the first test ("the concurrently renewed session must still be live"), reverting the compensation fails the third ("an ambiguous reservation must be compensated").
Verification
go test -race ./adapter/ ./kv/ ./internal/backup/— all pass (adapter 642s, kv 13s, backup 2.2s).golangci-lint ./adapter/...0 issues.Self-review
backupStateMu;forgetBackupSessionAtGenerationdoes the compare and the delete in one critical section, so two failing renewals cannot both claim the session. No new lock ordering.RenewBackupis supposed to provide.RenewBackup/BeginBackuprather than the helpers.https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE