Skip to content

migration: add range version RPC handlers - #1088

Open
bootjp wants to merge 52 commits into
mainfrom
design/hotspot-split-m2-cross-group
Open

migration: add range version RPC handlers#1088
bootjp wants to merge 52 commits into
mainfrom
design/hotspot-split-m2-cross-group

Conversation

@bootjp

@bootjp bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • add Internal ExportRangeVersions and ImportRangeVersions handlers backed by the local group store
  • wire per-group stores into Internal server registration
  • preserve staged migration route metadata in the serving route engine snapshot

Tests

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./distribution ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestEngineApplySnapshot|TestShardStore|TestLeaderRoutedStore' -count=1 -timeout=180s\n- GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution ./kv ./store --timeout=5m\n\nAuthor: bootjp

Summary by CodeRabbit

  • 新機能

    • データ移行用のエクスポート、インポート、ステージングデータの昇格に対応しました。
    • 指定バージョンのルート所有者や、範囲に交差するルートを取得できるようになりました。
    • 移行処理の進捗保存と再開に対応しました。
  • 改善

    • 移行中の書き込み保護、重複実行防止、ルーティング精度を強化しました。
    • スナップショットの保存・復元時に移行状態を適切に引き継げるようになりました。
    • 大容量データの分割処理、カーソル管理、入力検証を改善しました。
    • 予約済み制御キーへの誤操作を防止しました。

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

Next included review available in 21 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d4950b6a-df07-43d1-9a2b-125191e14c3e

📥 Commits

Reviewing files that changed from the base of the PR and between ace20ae and 8fc01b3.

📒 Files selected for processing (10)
  • adapter/distribution_server.go
  • adapter/distribution_server_test.go
  • adapter/internal.go
  • adapter/internal_migration_test.go
  • kv/fsm.go
  • kv/fsm_migration_fence_test.go
  • kv/shard_key.go
  • kv/shard_key_test.go
  • kv/shard_store.go
  • kv/shard_store_test.go
📝 Walkthrough

Walkthrough

移行データのエクスポート、インポート、ステージ済みデータの昇格を追加しました。ルート履歴参照、書き込みフェンス、ステージ可視性、S3補助キーのルーティング、MVCCスナップショットの移行状態保存も更新しました。

Changes

移行データ経路とルーティング

Layer / File(s) Summary
API契約とルート履歴
proto/internal.proto, store/store.go, distribution/migrator.go, adapter/distribution_server.go
ルート所有権と交差ルートの参照APIを追加しました。移行キー形式、予約キー判定、インポート・昇格のストア契約を追加しました。
移行RPCとRaft適用
adapter/internal.go, kv/fsm_migration_import.go, kv/fsm_migration_promote.go, store/migration_versions.go, store/migration_promote.go
エクスポート、インポート、ステージ済みデータ昇格を実装しました。リーダー検証、カーソル検証、サイズ上限、opcodeゲート、HLC更新、Raft適用結果を処理します。
ルーティングと書き込み制御
kv/sharded_coordinator.go, kv/fsm.go, kv/shard_router.go, kv/migrator_filter.go
ステージ可視性とS3補助ルートを使う所有グループ解決を追加しました。予約制御キー、書き込みフェンス、移行タイムスタンプ下限をraw、トランザクション、DEL_PREFIXに適用しました。
スナップショットと永続化
store/mvcc_store.go, store/lsm_store.go, store/migration_promote.go, store/lsm_migration.go
複数プレフィックス削除、適用インデックスの同一バッチ保存、昇格状態の保存・復元、V1/V2/V3スナップショットを実装しました。
実行時配線とRedis補助処理
main.go, main_bootstrap_e2e_test.go, adapter/redis_lua_context.go, adapter/redis_retry_test.go, adapter/s3_test.go
内部サーバーへストア、移行提案者、SQSパーティションリゾルバを渡す配線を追加しました。Redis Luaの開始時raw型キャッシュと再試行時の重複排除検証を更新しました。

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk: 🟠 High · up to ace20

This PR exposes migration read and write operations without the existing privileged authentication and can bypass write-fence checks when S3 auxiliary keys are present, allowing unauthorized data access or writes during migration. These high-impact current-head risks should be fixed before merge.

Sequence Diagram(s)

sequenceDiagram
  participant MigrationClient
  participant Internal
  participant RaftProposer
  participant kvFSM
  participant MVCCStore
  MigrationClient->>Internal: ExportRangeVersions
  Internal->>MVCCStore: ExportVersions
  MVCCStore-->>Internal: ページとカーソル
  Internal-->>MigrationClient: エクスポートページ
  MigrationClient->>Internal: ImportRangeVersions または PromoteStagedVersions
  Internal->>RaftProposer: 移行コマンドを提案
  RaftProposer->>kvFSM: 移行opcodeを適用
  kvFSM->>MVCCStore: ImportVersionsRaft または PromoteVersions
  MVCCStore-->>kvFSM: 適用結果
  kvFSM-->>Internal: 結果
  Internal-->>MigrationClient: ACKまたは昇格結果
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.40% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 348 functions across 52 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed タイトルは、PRの主要な変更であるマイグレーション用の範囲バージョンRPCハンドラーの追加を正確に要約しています。
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (64dc6f4):

  • distribution/engine.go
  • distribution/engine_test.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for range migrations by implementing ExportRangeVersions and ImportRangeVersions on the internal gRPC server, alongside updating the distribution Route metadata to track migration states. Feedback focuses on adding a defensive check in ExportRangeVersions to prevent infinite loops when the cursor does not progress, and removing an unreachable nil guard on i.leader in verifyInternalLeader to comply with guidelines against dead defensive code, which also requires updating the corresponding unit tests to use a mock leader.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread adapter/internal.go
Comment thread adapter/internal.go
Comment thread adapter/internal_migration_test.go
Comment thread adapter/internal_migration_test.go Outdated
Comment thread adapter/internal_migration_test.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64dc6f4239

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread adapter/internal.go Outdated
Comment thread adapter/internal.go
Comment thread adapter/internal.go
Comment thread adapter/internal.go
Comment thread adapter/internal.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (59a1672):

  • distribution/engine.go
  • distribution/engine_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Updated in 59a1672:

  • Route ImportRangeVersions through the target-group Raft/FSM path before ACK.
  • Observe the imported HLC floor before acknowledging non-empty imports.
  • Carry export key_family and bracket exclusions through the internal RPC.
  • Reject unbounded max_commit_ts exports and non-progressing export cursors.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./distribution -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution ./kv ./store --timeout=5m
  • git diff --check

Note: a broader go test ./adapter ./distribution ./kv ./store -count=1 -timeout=240s timed out in the adapter integration suite; the targeted touched-path tests above passed.

@bootjp
bootjp force-pushed the design/hotspot-split-m2-cross-group branch from 59a1672 to fbd7f56 Compare July 13, 2026 14:32
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (fbd7f56):

  • distribution/engine.go
  • distribution/engine_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Rebased onto the latest design/hotspot-split-m2-fence-drain head in fbd7f56 after #1087 advanced. The migration import/export fixes remain intact and now include the parent broad DEL_PREFIX fence fix.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore|TestRoutePrefixRange|TestFSMRejects.*WriteFenced|TestShardedCoordinatorRejects.*WriteFenced' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./distribution -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution ./kv ./store --timeout=5m
  • git diff --check

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Updated in 0fe341d:

  • Serve GetRouteOwnership from the exact RouteHistorySnapshot requested by catalog_version.
  • Serve GetIntersectingRoutes from the same versioned snapshot, including open-ended ranges.
  • Return NotFound for catalog versions outside the retained route-history window and FailedPrecondition when the distribution engine is absent.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter -run 'TestDistributionServer(GetRouteOwnership|GetIntersectingRoutes|OwnershipRPCs)' -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./distribution -run 'TestDistributionServer|TestEngineSnapshotAt|TestEngineGetIntersectingRoutes|TestRouteHistorySnapshot' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore|TestDistributionServer(GetRouteOwnership|GetIntersectingRoutes|OwnershipRPCs)' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution --timeout=5m
  • git diff --check

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (0fe341d):

  • distribution/engine.go
  • distribution/engine_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (7d3b01c):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Updated in 7d3b01c:

  • ImportRangeVersions now stages imported MVCC versions under the target-local !dist|migstage|<job_id>|<raw_key> shadow keyspace instead of writing raw user keys directly.
  • Added staged-data key helpers for job-scoped scans and future promotion/read-merge code to recover the original raw key.
  • Updated RPC/FSM tests to assert staged writes, raw-key absence before promotion, HLC floor preservation, and key/value clone safety.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./distribution -run 'TestMigrationStagedDataKeyRoundTrip|TestPlanMigrationBrackets|TestValidateMigrationRouteRange' -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./kv ./store -run 'TestInternal|TestMigration|TestImportVersions|TestExportVersions|TestApplyMigration|TestDistributionServer(GetRouteOwnership|GetIntersectingRoutes|OwnershipRPCs)' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./distribution ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore|TestDistributionServer(GetRouteOwnership|GetIntersectingRoutes|OwnershipRPCs)' -count=1 -timeout=240s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution ./kv --timeout=5m
  • git diff --check

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (e7f69ef):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Updated in e7f69ef:

  • Route reads with staged_visibility_active now merge live raw keys with target-local staged keys for GetAt, ScanAt, ReverseScanAt, and LatestCommitTS.
  • The merge chooses the newest MVCC version across live/staged candidates, preserves tombstone and TTL winner semantics, and maps staged keys back to their original raw keys.
  • Physical-limit scans on staged-visible routes now fail closed because the staged/live export merge cannot preserve the physical scan budget exactly.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -run 'TestShardStore(GetAt_MergesStagedVisibility|ScanAndLatestCommitTS_MergeStagedVisibility)' -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -run 'TestShardStore|TestMergeAndTrim|TestScanLockBounds' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -count=1 -timeout=240s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./store -run 'TestInternal|TestMigration|TestImportVersions|TestExportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore|TestGRPCServer_Raw' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./kv --timeout=5m
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./distribution ./kv ./store -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestShardStore|TestLeaderRoutedStore|TestDistributionServer(GetRouteOwnership|GetIntersectingRoutes|OwnershipRPCs)' -count=1 -timeout=240s
  • git diff --check

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7f69efd9e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/shard_store.go Outdated
func latestMVCCVersionAt(ctx context.Context, st store.MVCCStore, key []byte, ts uint64) (store.MVCCVersion, bool, error) {
result, err := st.ExportVersions(ctx, store.ExportVersionsOptions{
StartKey: key,
EndKey: nextScanCursor(key),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use exact-key export bounds for staged point reads

When the backing store is Pebble, ExportVersions converts EndKey to encodeKey(EndKey, MaxUint64) (store/lsm_migration.go), and nextScanCursor(key) (key + 0x00) sorts before normal encoded versions of key because the timestamp suffix begins with ^commitTS. On a staged-visible route backed by Pebble, both the live and staged point exports therefore return no candidates for ordinary keys, so GetAt can incorrectly return ErrKeyNotFound even though the key exists. Use a true single-key upper bound/exact-key filter instead of the scan cursor here.

Useful? React with 👍 / 👎.

Comment thread kv/shard_store.go Outdated
ts uint64,
reverse bool,
) ([]*store.KVPair, error) {
live, err := collectLatestLogicalVersions(ctx, g.Store, start, end, start, end, ts, liveLogicalVersionKey)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve scan limits while merging staged visibility

For staged-visible routes, this call starts an export loop that runs until result.Done, and the caller only applies limit after collecting both the full live range and the full staged prefix into maps. A user scan such as ScanAt(..., limit=100) over a migrated production range can therefore read and retain every version in the route before returning one page, which is a large latency/memory regression versus the previous bounded store scan. Please stop the merge once enough ordered candidates are available or otherwise page the merge by the requested limit.

Useful? React with 👍 / 👎.

Comment thread kv/shard_store.go
Comment thread kv/shard_store.go Outdated
Comment thread adapter/internal.go Outdated
Comment thread distribution/engine.go
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (ca1a050):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

Updated in ca1a050:

  • Bound staged visibility point reads to the exact key and reject compacted staged reads before merging.
  • Preserve staged scan limits with bounded candidate windows, and fail closed for physical-limit scans when staged visibility is active.
  • Route S3 bucket meta/generation exports by decoded bucket name so route filters match logical ownership.
  • Enforce migration write timestamp floors at coordinator, FSM, and ShardStore write paths.

Validation:

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -run 'TestFSMRejects.*MigrationTimestampFloor|TestFSMRejectsCommitButNotPrepareAtMigrationTimestampFloor|TestShardStore(GetAt_MergesStagedVisibilityPebbleExactKey|StagedVisibilityReadTSCompacted|PhysicalLimitFailsClosedBeforeStagedVisibilityFallback|RejectsWritesAtMigrationTimestampFloor)|TestShardStoreScanAndLatestCommitTS_MergeStagedVisibility|TestShardedCoordinatorRejects(PointWrite|DelPrefix)AtMigrationTimestampFloor|TestShardedCoordinatorDispatchTxn_RejectsMigrationTimestampFloor' -count=1 -timeout=180s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter -run 'TestInternalExportRangeVersions(UsesDecodedS3BucketRouteFilter|UsesStoreAndRouteFilter|RejectsUnboundedExport)|TestInternalImportRangeVersionsAppliesStoreBatch' -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./internal/s3keys -run 'TestBucketGenerationKey|TestParseBucketGenerationKey|TestBucketMetaKey' -count=1 -timeout=120s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -count=1 -timeout=240s
  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter ./distribution ./store ./internal/s3keys -run 'TestInternal|TestMigration|TestExportVersions|TestImportVersions|TestApply|TestRouteKeyFilter|TestLeaderRoutedStore|TestDistributionServer|TestRouteDescriptor|TestEngine|TestBucket' -count=1 -timeout=240s
  • GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./adapter ./distribution ./kv ./store ./internal/s3keys --timeout=5m
  • git diff --check

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca1a05070c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/shard_store.go
Comment thread kv/shard_store.go Outdated
Comment on lines +717 to +719
nextWindow := nextStagedVisibilityCandidateWindow(window)
if nextWindow == window {
return out, nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Continue staged scans after capped windows

For staged-visibility routes with more than 8192 visible logical keys, any caller asking for a limit above stagedVisibilityMaxCandidateWindow can get a truncated result. Once window reaches 8192, a full underlying scan with len(out) < limit falls into nextWindow == window and returns immediately, even though there may be more keys after the current candidate window. This makes large ScanAt/ReverseScanAt calls silently miss keys during staged cutover.

Useful? React with 👍 / 👎.

Comment thread adapter/internal.go Outdated
if routeStart != nil && bytes.Compare(routeKey, routeStart) < 0 {
return false
}
return routeEnd == nil || bytes.Compare(routeKey, routeEnd) < 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat empty S3 route ends as unbounded

This decoded S3 predicate only treats nil as an unbounded route_end, unlike RouteKeyFilter and the route descriptor convention where nil or empty means +infinity. If an unbounded last route is passed as a non-nil empty byte slice, every decoded bucket route key compares greater than []byte{}, so bucket meta/generation exports for that route return no versions. Check len(routeEnd) == 0 here instead of only routeEnd == nil.

Useful? React with 👍 / 👎.

Comment thread kv/fsm_migration_import.go Outdated
if err := proto.Unmarshal(data, req); err != nil {
return errors.WithStack(err)
}
result, err := f.store.ImportVersions(ctx, store.ImportVersionsOptions{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep migration imports off the direct write gate

In encrypted clusters where storage-envelope cutover is active but a replica's writer registration is not yet committed, this FSM-apply path calls store.ImportVersions, whose Pebble implementation uses the direct write/encryption gate (encryptForKey(..., true)). That can return ErrWriterNotRegistered while replaying the Raft entry and fail the import on that replica; migration imports need a raft-apply variant that uses the non-gated FSM path instead of the direct ImportVersions path.

Useful? React with 👍 / 👎.

Comment thread kv/sharded_coordinator.go Outdated
Comment thread kv/shard_store.go Outdated

if engineForGroup(g) == nil {
return s.localGetAt(ctx, g, key, ts)
return s.localGetAt(ctx, g, distribution.Route{}, key, ts)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve staged visibility on explicit group reads

When callers use the explicit-group path (for example RawGet/RawScanAt with group_id for SQS partitioned ownership), this passes a zero route into the local read path, so routeHasStagedVisibility is always false and imported staged versions are ignored after cutover. Those reads can return not found or stale live data even though normal GetAt/ScanAt for the same logical route would merge staged data; the explicit group path needs the staged route metadata or must fail closed while staged visibility is active.

Useful? React with 👍 / 👎.

Comment thread adapter/internal.go
@bootjp

bootjp commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

Addressed the staged-visibility migration review items.

Changes:

  • Merge staged commit timestamps into OCC validation/read-key checks for both ShardStore and ShardedCoordinator paths.
  • Keep migration imports on the raft-apply store path so encryption writer-registration gates do not block replay.
  • Preserve staged visibility metadata for explicit-group RawGet/RawScan paths, failing closed if the route cannot be resolved while staged visibility is active.
  • Continue staged scans past capped candidate windows.
  • Enforce migration write timestamp floors after TSO raw timestamp stamping.
  • Harden S3 decoded route export filters and reject fully unbounded migration export requests.

Validation:

  • go test -count=1 ./kv -run 'TestShardStore(ExplicitGroupReads_MergeStagedVisibility|ScanAt_ContinuesStagedVisibilityAfterCandidateWindow|ApplyMutations_ValidatesStagedReadKeys|PhysicalLimitFailsClosedBeforeStagedVisibilityFallback)|TestShardedCoordinator(ValidateReadKeysOnShard_UsesStagedVisibility|DispatchTxn_AddsStagedReadKeyAlias|RejectsTSORawPointWriteAfterStamping|DispatchTxn_RejectsMigrationTimestampFloor)'\n- go test -count=1 ./adapter -run 'TestGRPCServer_Raw(Get|ScanAt)UsesExplicitGroup|TestInternalExportRangeVersions(RejectsUnboundedExport|UsesDecodedS3BucketRouteFilter|DecodedS3EmptyRouteEndIsUnbounded|AppliesStoreBatch)'\n- go test -count=1 ./store -run 'TestRegistrationGate(DirectPathFailsClosedBeforeRegistration|FSMApplyPathNeverGated)'\n- golangci-lint --config=.golangci.yaml run ./kv ./store ./adapter --timeout=5m\n- git diff --check\n\nBroader go test -count=1 ./kv ./store ./adapter passed ./kv and ./store, then ./adapter hit the package-level 10m timeout; the focused adapter coverage above passed.\n\n@codex review

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (d33bf35):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d33bf3589a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/shard_store.go Outdated
if route, ok := s.engine.GetRoute(routeKey(key)); ok && route.GroupID == groupID {
return route, nil
}
if s.groupHasStagedVisibility(groupID) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail closed when the resolved staged route moved groups

When an explicit-group RawGet arrives with a stale group_id after cutover (for example, the caller still targets the old source group while engine.GetRoute(key) now returns the target route with StagedVisibilityActive), this checks staged visibility on the requested group instead of the resolved route. If the old group has no staged route, the function falls through to fallback and reads the old group's live MVCC without the staged merge, serving stale pre-migration data instead of failing closed or rerouting.

Useful? React with 👍 / 👎.

Comment thread kv/shard_store.go Outdated
if len(matched) > 0 {
return matched, true, nil
}
if s.groupHasStagedVisibility(groupID) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject stale explicit scans over staged target routes

For explicit-group RawScan, a stale proxy can still send the old source group_id after the current intersecting route has moved to the target with staged visibility. In that case matched is empty, this check only inspects the requested group, and the fallback scans the old group's local store, so range reads can miss staged target data instead of failing closed; fail when any intersecting current route is staged but not matched to the requested group.

Useful? React with 👍 / 👎.

Comment thread kv/shard_store.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b9e199fd0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/sharded_coordinator.go Outdated
Comment on lines +1272 to +1274
for _, route := range c.engine.GetIntersectingRoutes(start, end) {
if route.State == distribution.RouteStateWriteFenced {
return errors.Wrapf(ErrRouteWriteFenced, "key %q route range [%q,%q)", key, start, end)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Check only the selected S3 auxiliary owner route

When a bucket's !s3route|... interval has been split into multiple catalog routes and a non-owner slice is write-fenced, this loop rejects bucket metadata and generation writes even though s3BucketAuxiliaryOwnerRouteFromRange routes them through only the first intersecting, still-active route. The matching FSM WriteFencedIntersects check and timestamp-floor loops have the same issue, so migrating an unrelated object slice can block all auxiliary writes for that bucket; resolve the selected auxiliary owner route first and enforce only its fence and floor.

Useful? React with 👍 / 👎.

@bootjp

bootjp commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Addressed the latest S3 bucket auxiliary owner-route write-fence/floor findings and the Lua raw type cache bound finding.\n\nValidation:\n- go test ./kv -run 'TestFSM.*S3BucketAuxiliary|TestShardedCoordinator.*S3BucketAuxiliary' -count=1 -timeout=180s\n- go test ./adapter -run 'TestLua.*Type.*Cache|TestLuaNegativeTypeCache|TestLuaRawTypeAtStartCache' -count=1 -timeout=180s\n- go test ./kv -count=1 -timeout=240s\n- go test ./adapter -run 'TestLua|TestRedisLua' -count=1 -timeout=240s\n- go test ./... -run '^$' -count=1 -timeout=300s\n- golangci-lint run ./kv ./adapter --timeout=5m\n- git diff --check\n\n@codex review

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (b16dd62):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

elastickv/kv/fsm.go

Lines 1136 to 1137 in b16dd62

observedSnap, ok := f.routes.SnapshotAt(observedVer)
if !ok {

P1 Badge Reject raw writes applied on a former owner

When a stale coordinator sends a raw PUT or DEL to the source after cutover with ObservedRouteVersion == 0 (the normal raw path does not auto-pin a version), this branch skips the historical check, and the current-snapshot check only tests whether the route is WriteFenced; once the target route is Active, it never compares that route's GroupID with f.shardGroupID. The former source therefore accepts a new MVCC version that current readers route past, silently losing the acknowledged write. Raw applies need a current-owner check or a durable source-side cutover fence even for zero-version requests.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/sharded_coordinator.go
Comment thread adapter/internal.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (ace20ae):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

Inline comments:
In `@kv/fsm.go`:
- Around line 1204-1206: Update the mutation loop around
verifyS3BucketAuxiliaryWriteFenceFromSnapshot so a checked auxiliary key with
nil error skips only that mutation and continues validating subsequent
mutations; return immediately only when the helper reports a non-nil error,
preserving WriteFence validation for later keys.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 65afc31a-804e-42da-9f21-0404286ca866

📥 Commits

Reviewing files that changed from the base of the PR and between 0f499f9 and ace20ae.

📒 Files selected for processing (12)
  • adapter/internal.go
  • adapter/internal_migration_test.go
  • adapter/redis_lua_context.go
  • adapter/redis_lua_negative_type_cache_test.go
  • kv/fsm.go
  • kv/fsm_abort_test.go
  • kv/fsm_migration_fence_test.go
  • kv/fsm_onephase_dedup_test.go
  • kv/fsm_reserved_control_test.go
  • kv/fsm_txn_test.go
  • kv/sharded_coordinator.go
  • kv/sharded_coordinator_del_prefix_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment thread kv/fsm.go Outdated
@bootjp

bootjp commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Addressed the filesystem migration export gap in 986823c.

Changes:

  • derive physical export scan bounds for filesystem chunk rows when the request carries virtual route bounds
  • derive physical export scan bounds for routed filesystem usage counters
  • keep the raw bracket filter on the same derived physical range so route-filtered auxiliary rows are not dropped before export

Validation:

  • go test ./adapter -run 'TestInternalExportRangeVersions(DerivesFilesystem|UsesPartitionResolverGroup|UsesDecodedS3BucketRouteFilter)' -count=1 -timeout=180s
  • go test ./adapter -run 'TestInternalExportRangeVersions|TestExportRangeVersionsOptions' -count=1 -timeout=180s
  • go test ./distribution -run 'TestPlanMigrationBracketsCoversFilesystem|TestMigrationBracketContainsRoutedKeyUsesFilesystem|TestEveryFamilyBracketPrefixIsExcludedFromUserBracket' -count=1 -timeout=180s
  • golangci-lint run ./adapter ./distribution --timeout=5m
  • git diff --check

Note: go test ./adapter ./distribution -count=1 -timeout=240s timed out in the broader adapter raft/GRPC test set after the targeted export tests had passed.

@codex review

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (986823c):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Addressed the write-fence validation continuation issue in da19644.

Changes:

  • keep scanning the remaining mutations after an S3 bucket auxiliary key is accepted by the owner-route write-fence check
  • added a PREPARE regression where an S3 auxiliary mutation is followed by a write-fenced user mutation

Caller audit:

  • verifyWriteFenceFromSnapshot is only reached from verifyWriteFence for the observed and current route snapshots; verifyWriteFence is called by raw and txn apply paths.

Validation:

  • go test ./kv -run 'TestFSM(ContinuesWriteFenceValidationAfterS3BucketAuxiliaryWrite|RejectsCurrentWriteFencedS3BucketAuxiliaryPointWrite|IgnoresRawRouteFenceForS3BucketAuxiliaryWrite|IgnoresNonOwnerS3BucketAuxiliaryFenceForPointWrite|RejectsCurrentWriteFenced.|RejectsObservedWriteFenced.)' -count=1 -timeout=180s
  • go test ./kv -run 'TestFSM.*WriteFence|TestFSM.*S3BucketAuxiliary|TestFSM.*MigrationTimestampFloor' -count=1 -timeout=180s
  • golangci-lint run ./kv --timeout=5m
  • git diff --check

@codex review

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (da19644):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 986823cb59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: da19644a47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (30be14d):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Handled the explicit-group reverse scan finding at head 30be14d. Reverse ScanAtWithReadFence with GroupID now uses the same explicit-group route resolver even when route bounds are absent, and ReverseScanGroupAt uses that path too, so staged visibility is preserved in both reverse entry points.

Validation:

  • go test ./kv -run 'TestShardStoreExplicitGroupReads_MergeStagedVisibility|TestShardStoreScanAtWithReadFence_AllowsExplicitGroupRouteBoundReverse|TestShardStoreExplicitGroupRead_FailsClosedAfterPromotionClearsStaging' -count=1 -timeout=180s\n- go test ./adapter -run 'TestGRPCServer_RawScanAt_(ReadFenceAwareStoreFencesExplicitGroupReverse|GroupedReverseGoesThroughReadFenceStore|AllowsRouteBoundGroupedReverseWithShardStore)' -count=1 -timeout=180s\n- go test ./kv -run 'TestLeaderRoutedStore.*ScanAtWithReadFence|TestShardStoreTxnLock.*ScanAtWithReadFence|TestShardStore.ScanAtWithReadFence' -count=1 -timeout=180s\n- golangci-lint run ./kv ./adapter --timeout=5m\n- git diff --check\n\nCaller audit: rg -n "ReverseScanGroupAt\(|ScanAtWithReadFence\(" kv adapter internal -g'.go'\n\n@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30be14dd2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kv/shard_store.go
@@ -570,7 +797,7 @@ func (s *ShardStore) ScanKeysAtWithReadFence(ctx context.Context, start []byte,
return keysFromKVs(kvs), nil
}
if groupID != 0 {
return s.scanKeyRouteAtWithReadFence(ctx, distribution.Route{GroupID: groupID}, start, end, limit, ts, readRouteVersion)
return s.scanKeyRouteAtWithReadFence(ctx, distribution.Route{GroupID: groupID}, start, end, limit, ts, true, readRouteVersion)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve staged routes for grouped key scans

When a forward RawScanAt sets both keys_only and group_id without route bounds, adapter/grpc.go selects this method, but this branch constructs a route containing only the group ID. It therefore lacks the migration job and staged-visibility metadata, causing staged-only keys to be omitted and stale source-group keys to remain visible after ownership moves. Resolve the catalog route through the same explicit-group path used by value and reverse scans before scanning keys.

Useful? React with 👍 / 👎.

Comment on lines +371 to +372
states[jobID] = clonePromotionState(state)
return errors.WithStack(batch.Set(migrationPromoteMetaKeyBytes, encodeMigrationPromotionStates(states), nil))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire persisted promotion state with the migration

After a terminal split job is archived, both RetireMigration implementations remove its import acknowledgements and HLC floor but never remove the promotion-state entry stored here. On clusters performing repeated autosplits, the in-memory snapshot metadata and Pebble _migpromote map therefore grow monotonically; moreover, every later promotion chunk reads and re-encodes the entire accumulated Pebble map. Delete the selected job's promotion state as part of RetireMigration so completed jobs do not make subsequent migrations progressively more expensive.

Useful? React with 👍 / 👎.

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (8fc01b3):

  • distribution/engine.go
  • distribution/engine_test.go
  • distribution/migrator.go
  • distribution/migrator_export_plan_test.go
  • kv/fsm.go
  • kv/sharded_coordinator.go
  • store/mvcc_store.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@bootjp

bootjp commented Aug 31, 2026

Copy link
Copy Markdown
Owner Author

Fixed the bucket auxiliary route ownership and selected-owner export issues at 8fc01b3.

  • GetRoute/GetRouteOwnership now normalize S3 bucket meta/generation keys to the bucket route owner key.
  • ExportRangeVersions now emits S3 bucket auxiliary rows only for the selected owner route, while preserving raw-route fallback behavior.
  • Caller audit: rg -n "GetRouteOwnership(|GetRoute(|RouteOwnershipKey(" . -g'*.go'

Validation:

  • go test ./adapter -run 'TestDistributionServerGetRoute(_Ownership)?_Normalizes(S3BucketAuxiliary|FilesystemChunk)Keys|TestInternalExportRangeVersions(UsesDecodedS3BucketRouteFilter|DecodedS3EmptyRouteEndIsUnbounded|SkipsS3BucketAuxiliaryForNonOwnerRouteSlice|PreservesS3BucketRawRouteMatches|UsesPartitionResolverGroup)' -count=1 -timeout=180s
  • go test ./kv -run 'TestRoute(Key_NormalizesS3|OwnershipKey_NormalizesS3BucketAuxiliaryKeys)|Test(GroupReadKeysByShardID_RoutesS3BucketAuxiliaryTo(Staged|Promoted)Owner|ShardedCoordinatorRoutesS3BucketAuxiliaryWriteTo(Staged|Promoted)Owner)|TestShardStore(GetAt_RoutesS3BucketAuxiliaryToPromotedOwner|S3BucketAuxiliaryScanUsesPromotedOwner)' -count=1 -timeout=180s
  • golangci-lint run ./adapter ./kv --timeout=5m
  • git diff --check

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fc01b3c00

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread adapter/internal.go
if !ok {
return false
}
if allowRawRouteMatch && rawRouteFilter(rawKey) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve raw rows when a mixed S3 route moves

When a previously split source route starts under !s3route| and extends through the physical !s3| namespace (for example, [!s3route|bucket-z, +∞)), a later right-hand split disables allowRawRouteMatch even though the moved interval owns the raw bucket key. A legacy auxiliary row for an earlier bucket whose virtual owner is outside this source is then rejected by both predicates: the raw match is suppressed and its selected owner is outside the moving slice. After cutover the raw-key fallback resolves to the target, but the row remains stranded on the old source, making that legacy metadata unavailable; retain raw matches whenever the physical key itself lies in the migrated interval, including mixed virtual/raw ranges.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant