fix(swim): fast-restart rejoin — persist incarnation, echo refutations, ping liveness

[EnRaiha]

Closes the SWIM fast-restart rejoin stick (epic #165, Medium #9): a node that crashes while peers hold Dead(A, N) restarts at incarnation 0, announces Alive(0) exactly once, and can diverge forever when the probabilistic refutation round-trip fails.

• F4 (primary): persist local incarnation in the catalog (KEY_SWIM_INCARNATION); bootstrap resumes at persisted+1 so the first announcement dominates any lingering Dead rumour; every self-refutation bump persisted fire-and-forget.
• F1: refutations echoed deterministically on Ack + forwarded PingReq piggyback.
• F2: a ping IS liveness evidence — sender Alive claim applied before ack; self-advertise rate-limited to 500ms.
• F3: fresh Alive apply cancels the pending suspicion timer (kills the re-kill race).

6 regression tests incl. deterministic recovery loop (A holds Dead(B,5) → echo → self-refute → converge Alive). Verification: swim 129/129, cluster lib 1023, clippy -D warnings 0, maya-gate L1 clean.

Closes #165 item: Medium — SWIM fast-restart rejoin.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[EnRaiha]

Addressed the two critical review items in 1876085:

1. register_default_subsystems now propagates catalog read errors (?) instead of unwrap_or(None) — bootstrap fails on IO/corruption instead of silently restarting at incarnation 0.
2. load_swim_incarnation now returns an error (with key + length) when stored metadata length != 8 — corrupted catalogs surface early.

Thanks for the review!

[farhan-syah]

The diagnosis was right, and the fix is actually stronger than the description claims. ProbeScheduler::reshuffle builds the probe queue from snapshot().alive() only, so a node marked Dead is never probed again — peers holding Dead(A, N) send A nothing at all. Once the rumour ages out of piggyback buffers there is no path left for A to learn it is considered dead, so this is not just a probabilistic stick, it is a permanent one-way split.

F1 and F2 together are the part I would highlight: A pings with its stale Alive, the peer refutes and echoes its stored Dead record on the reply, A self-refutes above it, converged. That is a deterministic recovery loop and it works without persistence at all — F4 just makes the first announcement win outright. I checked whether F2 lets a stale claim bypass incarnation ordering and it does not: the sender's claim goes through apply_and_notify like any other update.

Landed in 47f890d, with you as co-author.

Four corrections went in on top, as 40ea609 and one hunk in the first commit.

1. save_swim_incarnation was a blind overwrite. Bumps are persisted off the probe path, so two of them can reach the catalog out of order and leave the lower value on disk. The node then restarts at an incarnation it has already advertised and loses to a peer still holding a Dead record at that number — the exact stick the key exists to break. The write now reads inside its own transaction and discards anything at or below what is stored. swim_incarnation_never_moves_backwards covers it.

2. The persistence ran in tokio::spawn, but save is a synchronous redb commit — a blocking call on a reactor worker. Now spawn_blocking. Note these two interlock: taking the write off the probe path is what makes bumps land out of order, so it is only safe because the store is monotonic now.

3. spawn_with_subscribers gained a parameter but nodedb-cluster-tests/tests/swim_routing_invalidation.rs was left on the old arity, so the branch does not compile against the integration suite.

4. CatalogIncarnationStore held Arc<ClusterCatalog>. The failure detector outlives the shutdown sequence that closes the catalog, so the redb file lock survived it and the next open failed with "Database already open. Cannot acquire lock." A node could not restart against its own data directory — four restart tests fail on it. It now holds a Weak, matching what LeaseRenewalLoop already does with Weak<SharedState> for the same reason.

Items 3 and 4 both surface immediately in the second test stage (-p nodedb-cluster-tests), which the verification note does not mention — swim 129/129 and cluster lib 1023 are both first-stage. Worth running both stages before posting results; item 4 in particular is a real availability bug, not a test artifact.