Ledger engine: crash-safe, resumable drill_events migration — no Redis - #7

Open

ar2rsawseen wants to merge 77 commits into

mainfrom

poc/ledger-no-redis

Commits on Aug 17, 2026

feat: ledger engine — chunk checklist + per-chunk staging, no Redis (A/B vs classic)

ar2rsawseen
and
claude
committed
4e58e48
bench: kill-drill supports both engines via DRILL_ENGINE

ar2rsawseen
and
claude
committed
c0b71cf
feat: Countly-branded live dashboard for the ledger engine (/viz)

ar2rsawseen
and
claude
committed
d2ae949
feat: complete the ledger engine — DLQ, bisection, breaker, backpressure, monitor, dry-run, report

ar2rsawseen
and
claude
committed
7f85b6c
refactor: single-architecture service — remove the legacy engine entirely

ar2rsawseen
and
claude
committed
7845383
fix: honest completion + operator recovery (found by live failure drill)

ar2rsawseen
and
claude
committed
1505849
feat: ingestion-matching transform — adopt shared spec + differential harness (D4, #6)

ar2rsawseen
and
claude
committed
0d23752
fix: per-process dedup-canary table name (multi-pod create/drop race)

ar2rsawseen
and
claude
committed
691609a
docs: operational runbook (flow, incident responses, in-place variant)

ar2rsawseen
and
claude
committed
fa3d1a2
feat: poison-pill quarantine — auto-split chunks that crash the process

ar2rsawseen
and
claude
committed
ed9ef40
feat: DLQ waive — the terminal operator decision for unmigratable docs

ar2rsawseen
and
claude
committed
d0ac2c8
feat(viz): full operator console — controls, DLQ panel, coercion report

ar2rsawseen
and
claude
committed
57eb793
feat(viz): self-service migration console — guided runbook, preflight, verification

ar2rsawseen
and
claude
committed
4d6f710
feat: pod scaling UX, real preflight actions, chaos-hardening

ar2rsawseen
and
claude
committed
63413e9

Commits on Aug 18, 2026

feat(viz): configuration card — current knobs, defaults, when-to-change, state location

ar2rsawseen
and
claude
committed
58baa64
fix: null-cd sweep ordering — close a silent data-miss window (found by review)

ar2rsawseen
and
claude
committed
87c5478
docs+setup: fresh-operator path — compose/.env rewritten, README split, null-cd preflight count

ar2rsawseen
and
claude
committed
c9d5620
feat(viz): serve the console at / — /viz kept as alias

ar2rsawseen
and
claude
committed
b4da411
fix(engine): a startup failure no longer kills the console

ar2rsawseen
and
claude
committed
7ebcd8c
feat: ledger rebuild from data + per-collection scoping of all live cd-window queries

ar2rsawseen
and
claude
committed
9cf0073
feat(k8s): ready-to-apply Kubernetes manifests

ar2rsawseen
and
claude
committed
d461d94
docs: decouple the transform spec from the unmerged platform PR

ar2rsawseen
and
claude
committed
0e2c34c
docs: reference countly-platform#1105 (surgical cd passthrough)

ar2rsawseen
and
claude
committed
82d4783
feat(verify): attribute duplicate ids to live ingestion vs migration

ar2rsawseen
and
claude
committed
900e016
feat: migrated provenance flag — 100% exact migrated/live distinction

ar2rsawseen
and
claude
committed
68cfe26
refactor: provenance without schema changes — (_id, cd) pairs replace the migrated column

ar2rsawseen
and
claude
committed
622dcf3
ci: run the full integration suite on every PR (MongoDB 7 + ClickHouse 26.4)

ar2rsawseen
and
claude
committed
081afc4
fix(viz): honest counters + DLQ fix-location guidance

ar2rsawseen
and
claude
committed
2243004
feat: scale-proof UI + self-driving preflight

ar2rsawseen
and
claude
committed
2575422
feat: billion-scale engine paths — async verify, partitioned dup scan, full DLQ drain

ar2rsawseen
and
claude
committed
06e4782
config: default SERVICE_NAME to drill-migrator — two required vars, not three

ar2rsawseen
and
claude
committed
43fa12d
test: migration under concurrent live ingestion

ar2rsawseen
and
claude
committed
930f9aa
feat: mass-DLQ safety — global pause guard, background replay, redo-then-replay dedup

ar2rsawseen
and
claude
committed
2f6d87d
feat(dlq): mass guard and panel report the disk cost

ar2rsawseen
and
claude
committed
c801daa
fix(viz): replay toast says started, not finished

ar2rsawseen
and
claude
committed
d7dbef0
feat: optional dashboard auth (DASHBOARD_PASSWORD)

ar2rsawseen
and
claude
committed
f2759d0
Revert dashboard auth — not needed

ar2rsawseen
and
claude
committed
c78e2df
docs: Docker multi-container scaling recipe

ar2rsawseen
and
claude
committed
8fa6c2a

Commits on Aug 20, 2026

feat: post-migration audits — source recount + sampled doc-per-doc comparison

ar2rsawseen
and
claude
committed
9652804
feat: per-commit source-count guard + cd-pruned id lookups

ar2rsawseen
and
claude
committed
363cc63
fix(audit): waived/pending DLQ docs explain their window's shortfall

ar2rsawseen
and
claude
committed
5e71394
test: de-flake live-parallel writer assertion

ar2rsawseen
and
claude
committed
a78df1a

Commits on Aug 22, 2026

fix: index builds survive any duration and crash-resume correctly

ar2rsawseen
and
claude
committed
83b09ee
feat(transform): stringify integer doubles in the Int64/UInt64 overflow window

ar2rsawseen
and
claude
committed
f3b87ad
test: pin platform query compatibility of overflow-window strings

ar2rsawseen
and
claude
committed
50e4095

Commits on Aug 23, 2026

fix: dry-run DLQ replay must never write the live table

ar2rsawseen
and
claude
committed
5647a5a
feat: global chunk scheduling — pods parallelize across collections

ar2rsawseen
and
claude
committed
af1b267
feat: top-up passes — delta chunks for data arriving after mapping

ar2rsawseen
and
claude
committed
2115e6c
fix(rebuild): unscopable windows in mixed-shape runs are pending, not misclassified

ar2rsawseen
and
claude
committed
abca6a4
test: pin both source schemas — hashed-collection (old) and base drill_events (new format)

ar2rsawseen
and
claude
committed
7b1de12
fix: multi-pod correctness — claim fencing, append reservation, sweep barrier-lite

ar2rsawseen
and
claude
committed
974fee5
fix: initChunks gets the same reservation as appendChunks + race test sweep

ar2rsawseen
and
claude
committed
e482b80
fix: single-winner recovery + attach self-healing — found by new pod-chaos harness

ar2rsawseen
and
claude
committed
f4926ef
test(chaos): full field scenario — live source growth + live CH ingestion under random pod kills

ar2rsawseen
and
claude
committed
6560b1c
fix(ui): Docs/Second shows the CLUSTER rate, not this pod's — field-reported

ar2rsawseen
and
claude
committed
09c0b61
fix(guards): lease-aware active-pod checks for rebuild and both audits

ar2rsawseen
and
claude
committed
09b68e2
fix: raise max_partitions_per_insert_block for migration inserts (field failure)

ar2rsawseen
and
claude
committed
492408a
fix: type the partition-block setting as string; bound it at the provable ceiling

ar2rsawseen
and
claude
committed
2d3f5c7
test: attach a 120-partition chunk end-to-end (retry path of field chunk #110)

ar2rsawseen
and
claude
committed
af1421f
feat: outage self-healing — classified breaker pause + auto-resume probe

ar2rsawseen
and
claude
committed
58a3e5d
fix: recovery must never kill the pod + outage suite proves the breaker loop live

ar2rsawseen
and
claude
committed
c2ee314

Commits on Aug 24, 2026

fix: retention-TTL shrinkage is not a defect + dashboard truth at 73k-chunk scale

ar2rsawseen
and
claude
committed
ce865af
feat: mirror-first playbook — live change-stream mirror + checkpoint-bounded migration

ar2rsawseen
and
claude
committed
6871100
refactor: tee-mirror cutover — drop the change-stream mirror, keep the bound

ar2rsawseen
and
claude
committed
7f343ed
feat: tee-boundary auto-detection + tee-sync parity monitor

ar2rsawseen
and
claude
committed
da3cd70
docs(ui): the cd bound is opt-in — spell out both modes

ar2rsawseen
and
claude
committed
83cdc4f
feat: one-click apply of the detected bound — multi-pod, no restarts

ar2rsawseen
and
claude
committed
a5b5e78
docs(ui): scenario picker — all four cutover topologies, spelled out

ar2rsawseen
and
claude
committed
75381ff
perf: prune pair checks to the primary key — fixes the tiny-chunk tail collapse

ar2rsawseen
and
claude
committed
b0e705a

Commits on Aug 25, 2026

fix: structured skips are a data condition — not fuel for the fail-rate breaker

ar2rsawseen
and
claude
committed
792f544
feat(ui): field feedback round — realtime rate, pause visibility, run timeline

ar2rsawseen
and
claude
committed
2d968af
feat(ui): chunk map is a row-aligned window over the claim-order tape

ar2rsawseen
and
claude
committed
eedcf70
feat(ui): progress bar — documents are the work, chunks are the bookkeeping

ar2rsawseen
and
claude
committed
889c787
feat(ui): one Pause/Resume toggle button

ar2rsawseen
and
claude
committed
472663d

Commits on Aug 26, 2026

fix(ui): progress gets a FROZEN denominator — "to go" is a countdown now

ar2rsawseen
and
claude
committed
b137b8b
feat(audit): cd-checksum fingerprint — catches the right COUNT of the WRONG docs

ar2rsawseen
and
claude
committed
b0d9d3d
feat: SSH-first monitoring — /status.txt, progress heartbeat, curl cookbook

ar2rsawseen
and
claude
committed
fa65d28