Governance review Phase 0 — mechanize what is decidable (closes C-98) - #280
Merged
Conversation
…list drifting Phase 0 of the governance consistency review. The documents cite each other about 3,300 times — 1,165 ADR references, 1,829 register concerns, 125 disagreements, 179 source paths. Every one is a claim that something exists. None was checked, and the only reason none had rotted is that nobody had renamed a file yet. `scripts/check_doc_refs.py` resolves each against its source of truth and runs in the `docs` CI job beside `check_arch_tree.py`. Offline by design: issue references need the network, and a doc check that goes red when GitHub does is a doc check people learn to ignore. `--check-issues` covers them manually. **The allowlist is the load-bearing part, and it is guarded in both directions.** Seventeen references are correct despite not resolving locally — a sibling repo's ADR, a concern in another register, an id this register skipped on purpose. A checker that flags those gets switched off within a week. So each is listed with its reason, AND the script fails if an id it allowlists is not also named in the register's Register Conventions. Neither half can quietly diverge from the other. Writing that guard immediately found that the register documented **8 of 17** non-local ids; nine were cited in the corpus and explained nowhere. That is the same drift the script exists to catch, in the convention that governs it. The register now carries the full list. **Two findings the first run produced, both about the checker rather than the docs.** `perspectives/` and `critiqus/` are written from OTHER repositories' points of view — a datafactory reviewer citing datafactory's ADRs, concerns and source tree. Resolving those against this register flagged 36 pieces of correct prose. References there are the author's namespace and are not resolved; paths under `src/views_frames*` still are, because those are unambiguously about us. `critiqus/` — 4 files, 1,194 lines — was missing from the corpus survey this review was scoped on. The real corpus is 100 markdown files, not 82. Mutation-tested six ways, all firing: a dangling ADR; a dangling concern; a dangling path under our own packages; removing a foreign id from the allowlist; undocumenting one in the register; and a missing register. The last of those found a defect in the script itself. A moved register produced an unhandled FileNotFoundError traceback rather than a named failure — the same self-disabling shape C-89 records, in a script written to prevent that class. It now says which file it could not find and exits 1. Two of the six mutations initially "passed" because I aimed them at a file that did not contain the reference. A no-op mutation proves nothing; they were redone against CLAUDE.md and docs/CICs/SpatioTemporalIndex.md, which do. What it does not catch, stated in the script: that a reference points at the RIGHT thing. ADR-018 resolving proves the file exists, not that it says what the citing sentence claims. That is a comparison problem, and comparison is what the read in Phase 1 is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he filesystem (closes C-98)
Phase 0 of the governance consistency review, second half. Three checks added to
`docs/validate_docs.sh`, and the four stale counts they immediately caught.
**Check 11 — counts stated in prose.** Three documents stated numbers derived from the
document set, and they were wrong DIFFERENTLY from each other:
CLAUDE.md "CICs ... (7 active)" 9 exist
CLAUDE.md "project ADRs 011-026" 011-029
docs/ADRs/README.md "Project-Specific ADRs (011-027)" 011-029 (x2)
Two documents counting the same range, disagreeing with each other AND with the
filesystem, is worse than either being wrong alone — a reader who checks one against
the other finds a contradiction with no way to tell which is right.
A third defect surfaced while fixing them. CLAUDE.md said "Constitutional ADRs
000-010", folding ADR-010 into the constitutional range. The ADR index has three
sections, not two: constitutional 000-009, a Governance ADRs section holding only
ADR-010, then project 011-029. The index's structure is the authority.
Check 11 pins the CIC count, every project-ADR range against the highest ADR on disk,
and the register's header against its own body. It validated the C-98 entry this
commit adds, seconds after being written.
**Check 12 — CIC Test Alignment references.** Each CIC's section 10 cites the tests
holding its guarantees; that citation is the only link between a contract and its
evidence, and nothing stopped a rename stranding one. All 19 references resolve, and
the check verifies both the file and, where named, the function inside it.
**Check 13 — amendments are two-sided.** When ADR-A carries "Amendment (date,
ADR-B)", ADR-B must mention ADR-A. A one-sided amendment is how a reader arrives at
the amended document and never learns it was amended. The ADR-018 / ADR-025 / ADR-028
triangle was correct by hand; it is now checked.
Mutation-tested, and the harness matters here: **two of my mutations initially
"passed" because I aimed them at files that did not contain the string** — the same
no-op mistake made earlier in this review. `test_frame_parity.py` lives in
TargetFrame.md, not PredictionFrame.md. The rerun asserts the file actually changed
before reading any result, and only then are the results meaningful: a drifted CIC
count, a new ADR with ranges unupdated, a register header that stops matching its
body, a renamed test file, and a vanished test function all error.
What check 11 does not catch, stated in its comment: a count stated in a phrasing the
pattern does not match. It targets three known claim shapes. A general numeric auditor
would flag every number in the corpus and be switched off within a week.
Register: 95 entries, 10 open, 85 resolved. Still 0 actionable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of the governance consistency review: check what a program can check, so the human read in Phase 1 is spent on what it cannot.
Two new checkers, ~3,300 + 4 claims now held
scripts/check_doc_refs.pyresolves every cross-reference in the corpus — 1,165 ADR citations, 1,829 register concerns, 125 disagreements, 179 source paths. None was checked before; the only reason none had rotted is that nobody had renamed a file yet. Offline by design (issue refs need the network;--check-issuesfor manual sweeps).docs/validate_docs.shchecks 11–13 — prose counts against the filesystem, CIC §10 test references, amendment symmetry.What they caught immediately
CLAUDE.mdCLAUDE.mddocs/ADRs/README.md×2Two documents counting the same range, wrong differently from each other. Plus a third:
CLAUDE.mdfolded ADR-010 into the constitutional range, but the index has three sections — constitutional 000–009, a Governance ADRs section holding only ADR-010, then project 011–029.And the register documented 8 of 17 non-local ids; nine were cited across the corpus and explained nowhere.
Two findings about the corpus itself
critiqus/— 4 files, 1,194 lines — was missing from the survey this review was scoped on. The corpus is 100 markdown files, not 82.perspectives/andcritiqus/use a different reference namespace. They are written from other repositories' points of view — a datafactory reviewer citing datafactory's ADRs, concerns and source tree. My first run flagged 36 pieces of correct prose. Those documents cannot be checked against our register; paths undersrc/views_frames*still are.The allowlist is guarded both ways
17 references are correct despite not resolving locally. Each is listed with its reason, and the script fails if an id it allowlists is not also named in the register's conventions — neither half can quietly diverge.
Mutation-tested, including two of my own mistakes
Nine mutations across both checkers, all firing. Two of them initially "passed" because I aimed them at files that did not contain the string — a no-op mutation proves nothing. The rerun asserts the file actually changed before reading the result.
One mutation found a defect in my own script: a moved register produced an unhandled traceback rather than a named failure — the same self-disabling shape C-89 records, in a script written to prevent that class.
Register: 95 entries, 10 open, 85 resolved, 0 actionable.
🤖 Generated with Claude Code