From 8a001bef171d7bba00c838375bb3914d4356128c Mon Sep 17 00:00:00 2001 From: Claude Code Date: Wed, 2 Sep 2026 16:06:37 +0300 Subject: [PATCH] docs(analytics): restore explicit Analytical Judge gate after findings (#357) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a compact post-findings orchestration checkpoint — findings -> Analytical Judge -> pass / revise / blocked -> final findings -> memo — that challenges the analytical conclusion before narrative packaging. It reuses the existing controls and introduces no second QA framework, taxonomy, method, or intent. - ANALYTICAL_REASONING_STANDARD.md: new §8 "Analytical Judge gate" (seven semantic checks mapped to TASK_PROFILE / METHOD_PLAN / CLAIM_EVIDENCE_REGISTRY / explanation challenge / FINAL_EVIDENCE_SUFFICIENCY / material synthesis; ANALYTICAL_JUDGE output record; bounded revise/rerun; quick-mode collapse). §8->§9 (Runtime collapse), §9->§10 (Stop/escalation) renumbered; full-path block updated. - ANALYTICS_WORKFLOW.md: canonical workflow + Step 9 make the gate explicit before memo/report. - QA_CHECKLIST.md: post-findings Analytical Judge gate block (orchestration, not a second framework). - ACCEPTANCE_CRITERIA.md: criterion 12, analytical_judge_status field, blocked status entry. - GOVERNANCE_AND_ANTI_PATTERNS.md: gate principle, blocker, and anti-pattern rows (no autonomous retry loop, no second framework, blocked != executed). - SMOKE_QA_FOR_ANALYTICS.md: case forcing an over-strong "root cause" claim to revise; quick-mode collapse case. - CHANGELOG.md entry; regenerated ANALYTICS_01/03/05 bundles + provenance. Enforced invariants unchanged: claim strength <= final evidence sufficiency; driver != root cause; correlation != causation; blocked != executed; manual_review_required governed by the reasoning standard; AES canonical. Checks: 6 canonical + provenance + pytest (231) pass. Co-Authored-By: Claude Sonnet 5 --- .../Knowledge/ACCEPTANCE_CRITERIA.md | 13 ++++ .../ANALYTICAL_REASONING_STANDARD.md | 74 ++++++++++++++++++- .../Knowledge/ANALYTICS_WORKFLOW.md | 7 +- ChatGPT/[Analytics]/Knowledge/CHANGELOG.md | 33 +++++++++ .../Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md | 8 +- ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md | 18 +++++ .../Knowledge/SMOKE_QA_FOR_ANALYTICS.md | 38 ++++++++++ .../ANALYTICS_01_CORE_WORKFLOW.md | 8 +- .../ANALYTICS_03_TECHNIQUES_AND_CHARTS.md | 59 ++++++++++++++- .../ANALYTICS_05_QA_GOVERNANCE_ROUTING.md | 67 ++++++++++++++++- docs/knowledge_bundle_provenance_audit.json | 12 +-- docs/knowledge_bundle_provenance_audit.md | 12 +-- 12 files changed, 324 insertions(+), 25 deletions(-) diff --git a/ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md b/ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md index 43dd0d3..f30c707 100644 --- a/ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md +++ b/ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md @@ -15,6 +15,15 @@ A result is accepted when: 11. Before publication, every flagship metric in a quantitative report passes `QUANTITATIVE_SANITY_GATE.md`; otherwise the result is `revise` or `blocked` and is not published as a final quantitative conclusion. +12. For `analytical_depth = material / decision_critical`, the Analytical Judge + gate (`ANALYTICAL_REASONING_STANDARD.md` §8) ran after findings and before + memo / report generation, and an `ANALYTICAL_JUDGE` result is recorded with + `status: pass` (or a `revise` resolved by one bounded correction and a + passing re-check). A `blocked` Judge status means the final management + conclusion is not published. `maximum_claim_strength` does not exceed + `FINAL_EVIDENCE_SUFFICIENCY`; `driver != root cause` and + `correlation != causation` hold. Routine / no-trigger cases satisfy this + through the compact QA note without a full Judge record. ## Main file acceptance @@ -31,6 +40,7 @@ slices_from_mart_main_full: pass/fail/blocked/not_applicable accepted: yes/no qa_status: pass/fail/blocked quantitative_sanity_gate_status: pass/revise/blocked/not_applicable +analytical_judge_status: pass/revise/blocked/not_applicable confidence: high/medium/low residual_risks: known_limitations: @@ -103,6 +113,9 @@ Use `blocked` when: - compact-only input is insufficient for requested conclusion; - implementation is required before result can be produced. - a required flagship metric has a blocked quantitative sanity gate. +- the Analytical Judge gate returns `blocked` (required prerequisite, + reconciliation, grain, validation path, or discriminating evidence + unavailable). ## Not production-ready rule diff --git a/ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md b/ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md index 3606e8d..802f3b1 100644 --- a/ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md +++ b/ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md @@ -393,7 +393,74 @@ claim `formula_or_method` remains a human-readable description and does not replace `method_execution_id`. A claim that requires an actual method result cannot use an execution with status `blocked`, `planned`, or `not_needed` as supporting evidence. -## 8. Runtime collapse +## 8. Analytical Judge gate + +An explicit, bounded orchestration checkpoint that runs **after deterministic findings and before narrative packaging** (memo / report). It challenges the analytical conclusion itself. It does not add a QA framework, taxonomy, method, intent, or execution state: it reads the controls already defined in §4–§7 and the existing Analytics QA, and returns one visible result. + +```text +deterministic calculation +→ findings +→ ANALYTICAL JUDGE +→ pass / revise / blocked +→ revise or rerun when required +→ final findings +→ memo / report +→ existing Memo QA / Judge +→ acceptance +``` + +### Seven checks (semantic, over existing controls — no new definitions) + +| # | Judge check | Reads | +|---|---|---| +| 1 | Question fit — did the analysis answer the declared business question and scope? | `TASK_PROFILE.question`, active intent classification (§1–§2), `expected_output` | +| 2 | Method adequacy — were the selected methods sufficient, and was every supporting method actually executed with prerequisites met? | `METHOD_PLAN` (`status`, `prerequisites_met`, `execution_status`), the registry and triggers in `ANALYTICAL_TECHNIQUES.md`, `blocked != executed` (§4) | +| 3 | Evidence lineage — can every headline conclusion be traced executed method → source mart / table / slice → metric → period → grain → filter / baseline → evidence? | the lineage in §7 and `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md` | +| 4 | Alternative explanations — is a materially plausible alternative untested or not kept visible? | `COMPETING_EXPLANATIONS[]`, `DISCRIMINATING_EVIDENCE`, `FALSIFICATION_TEST` (§6) | +| 5 | Contradicting evidence — does evidence or method disagreement materially weaken the conclusion? | `CONTRADICTING_EVIDENCE`, `RESIDUAL_UNCERTAINTY`, the method-disagreement rule (§6) | +| 6 | Claim calibration — is any conclusion stronger than `FINAL_EVIDENCE_SUFFICIENCY` (driver → root cause, association → causation, single-period → systemic)? | claim ladder, `claim_support`, `causal_status`, `FINAL_EVIDENCE_SUFFICIENCY`, `claim strength <= final evidence sufficiency` (§7); `VARIANCE_DIAGNOSTIC_CONTRACT.md` for material Plan/Fact | +| 7 | Decision proportionality — does any recommendation, risk statement, or management implication exceed the verified evidence? | the material management synthesis rule (`ANALYTICS_WORKFLOW.md` Step 11) and its acceptance criteria | + +The Judge introduces no competing definition for any concept it reads. It reuses `PRELIMINARY_EVIDENCE_CHECK`, the explanation-challenge record, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md`, Analysis QA, and the material variance diagnostic QA where applicable. + +### Output + +```text +ANALYTICAL_JUDGE +status: pass / revise / blocked +failed_checks: +material_findings: +required_action: +rerun_required: yes / no +manual_review_required: yes / no +maximum_claim_strength: +``` + +- **pass** — every material conclusion is supported at its stated claim strength; no material unresolved contradiction or missing analytical check remains. +- **revise** — the conclusion or its wording can be corrected from existing evidence, or a bounded additional deterministic check / rerun is available. +- **blocked** — a required prerequisite, reconciliation, grain, validation path, or discriminating evidence is unavailable; the final management conclusion must not be published. + +`maximum_claim_strength` cannot exceed `FINAL_EVIDENCE_SUFFICIENCY.maximum_claim_strength`. `manual_review_required` is the value set by §1; the Judge surfaces it and does not redefine it. + +### Bounded revise / rerun + +Allowed: + +```text +Judge finding +→ one explicit bounded correction or deterministic rerun +→ Judge re-check +``` + +Forbidden: silent self-retry; unrestricted iterative analysis; adding a method without registry / trigger support; reasoning around a missing deterministic prerequisite; treating a `blocked` method as evidence. Correction count, stop conditions, and rollback remain those of the canonical `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` and the `[Codex]` one-fix limit where applicable; this gate does not widen them. + +### Runtime collapse + +`routine` + low uncertainty + no material trigger: collapse the Judge to the existing compact QA note on the compact path (Runtime collapse, §9). Do not instantiate the seven-question `ANALYTICAL_JUDGE` record. + +`material` / `decision_critical`: the explicit gate is mandatory. A recorded `ANALYTICAL_JUDGE` with `status: pass` — or a `revise` resolved by one bounded correction and a passing re-check — is required before final findings are handed to memo / report generation. A `blocked` status stops publication of the final management conclusion. + +## 9. Runtime collapse P0 is one workflow with conditional depth, not two independent workflows. @@ -419,12 +486,13 @@ TASK PROFILE → EXPLANATION CHALLENGE → CLAIM CALIBRATION → FINAL EVIDENCE SUFFICIENCY +→ ANALYTICAL JUDGE (pass / revise / blocked) → FINAL FINDINGS → FULL QA ``` -Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, or full evidence-sufficiency record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default. +Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, full evidence-sufficiency record, or the seven-question `ANALYTICAL_JUDGE` record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default. -## 9. Stop and escalation rules +## 10. Stop and escalation rules Stop analytical deepening when: diff --git a/ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md b/ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md index 9a3b3ba..9769150 100644 --- a/ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md +++ b/ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md @@ -47,10 +47,13 @@ question / scope → deterministic calculation → findings → evidence challenge / calibration as required +→ Analytical Judge (pass / revise / blocked) +→ revise or rerun when required +→ final findings → management synthesis when material and management-facing → LLM context package → memo / report -→ judge / QA +→ memo QA / judge → revise or rerun → acceptance → next run trigger @@ -188,6 +191,8 @@ State method, metric, period, grain, data source and limitation. After deterministic findings, apply the preliminary evidence check, explanation challenge, claim calibration, and final evidence sufficiency from `ANALYTICAL_REASONING_STANDARD.md` only to the depth required by the case. Preserve `driver != root cause` and do not silently reconcile material method disagreement. +Then run the **Analytical Judge gate** (`ANALYTICAL_REASONING_STANDARD.md` §8): a compact post-findings orchestration checkpoint over the controls above that returns `pass / revise / blocked` before the findings become narrative. For `analytical_depth = material / decision_critical` the explicit gate is mandatory and a recorded `ANALYTICAL_JUDGE` result (`pass`, or a `revise` resolved by one bounded correction and a passing re-check) is required before memo / report generation; `blocked` stops publication. For routine / low-uncertainty cases with no material trigger it collapses to the existing compact QA note. It adds no method, intent, taxonomy, or QA framework. + ## Step 10 — Charts Charts must be sourced from `mart_main_full` or a documented slice derived from it. diff --git a/ChatGPT/[Analytics]/Knowledge/CHANGELOG.md b/ChatGPT/[Analytics]/Knowledge/CHANGELOG.md index e343dca..8155e06 100644 --- a/ChatGPT/[Analytics]/Knowledge/CHANGELOG.md +++ b/ChatGPT/[Analytics]/Knowledge/CHANGELOG.md @@ -1,5 +1,38 @@ # Changelog +## 2026-09-02 — analytical-judge-gate (issue #357) + +Added: + +- Explicit `Analytical Judge` gate as `ANALYTICAL_REASONING_STANDARD.md` §8: a + compact post-findings orchestration checkpoint (`findings → Analytical Judge + → pass / revise / blocked → final findings → memo`). Seven semantic checks + mapped to existing controls; compact `ANALYTICAL_JUDGE` output record; + bounded revise/rerun rule; `quick` runtime collapse. No new QA framework, + taxonomy, method, or intent; §8→§9, §9→§10 renumbered. + +Updated: + +- `ANALYTICS_WORKFLOW.md` canonical workflow and Step 9 to make the gate + explicit before memo/report. +- `QA_CHECKLIST.md` with a post-findings Analytical Judge gate block + (orchestration, not a second framework). +- `ACCEPTANCE_CRITERIA.md` with criterion 12, `analytical_judge_status`, and a + blocked-status entry. +- `GOVERNANCE_AND_ANTI_PATTERNS.md` with the gate principle, a blocker entry, + and anti-pattern rows (no autonomous retry loop, no second framework, + `blocked != executed`). +- `SMOKE_QA_FOR_ANALYTICS.md` with a case forcing a plausible over-strong + "root cause" claim to `revise`, plus a `quick`-mode collapse case. + +Status: + +```text +production_ready: not claimed +pilot_case_required: yes +smoke_qa_status: pass +``` + ## 2026-05-25 — analytics-project-settings-minor-fix Added: diff --git a/ChatGPT/[Analytics]/Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md b/ChatGPT/[Analytics]/Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md index b31e55a..4f176cf 100644 --- a/ChatGPT/[Analytics]/Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md +++ b/ChatGPT/[Analytics]/Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md @@ -11,6 +11,7 @@ - Method eligibility and triggers follow `ANALYTICAL_TECHNIQUES.md`; the LLM cannot silently override the registry. - Reasoning cannot substitute for deterministic execution or missing prerequisites. - Claim strength cannot exceed final evidence sufficiency. +- Every material / decision-critical analytical conclusion passes an explicit post-findings Analytical Judge gate (`ANALYTICAL_REASONING_STANDARD.md` §8) before narrative packaging; the gate orchestrates existing controls and adds no second QA framework. - `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` remains canonical execution governance; `ANALYTICS_EXTENSION.md` supplies domain-specific constraints without creating a second execution framework. ## Evidence labels @@ -39,7 +40,8 @@ Do not publish final management conclusion when: - unsupported cause; - risk without basis; - action without owner/due date; -- no main mart for a mart-based conclusion. +- no main mart for a mart-based conclusion; +- the Analytical Judge gate returns `blocked`. ## Anti-patterns @@ -89,6 +91,10 @@ Apply these controls through `ANALYTICAL_REASONING_STANDARD.md` and the existing | Method catalog inflated by converting controls into methods | Require a distinct question, execution procedure, and material analytical effect before adding a method. | | Decision methods leak from `[Thinking]` into `[Analytics]` | Keep trade-offs, reversibility, premortem, risk appetite, choice, and decision in `[Thinking]`. | | Reasoning-control loop mistaken for an autonomous execution loop | Keep reasoning/method selection inside the AES-governed scope, checks, bounded correction, stop, rollback, acceptance, and authority boundaries. | +| Analytical conclusion reaches memo without an explicit post-findings challenge | Run the Analytical Judge gate after findings; record `ANALYTICAL_JUDGE` `pass / revise / blocked` for material / decision-critical cases. | +| Analytical Judge treated as an autonomous retry loop | Allow only `Judge finding → one bounded correction or deterministic rerun → Judge re-check`; no silent self-retry, no unrestricted iteration; AES limits and the `[Codex]` one-fix limit are unchanged. | +| Analytical Judge used as a second QA framework or a new taxonomy | The gate only orchestrates `PRELIMINARY_EVIDENCE_CHECK`, explanation challenge, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY`, Analysis QA, and variance diagnostic QA; it defines no new field. | +| `blocked` method converted into evidence at the Judge step | Enforce `blocked != executed`; a `blocked` prerequisite forces Judge `blocked`, not a weaker `pass`. | ## Metric / artifact explosion diff --git a/ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md b/ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md index d5a614e..9b60804 100644 --- a/ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md +++ b/ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md @@ -116,6 +116,24 @@ the record internal or in evidence/appendix so `quick` output remains compact. Use `ANALYTICAL_REASONING_STANDARD.md` for field semantics. These checks extend the existing Analysis QA; they do not create a separate QA framework. +### Analytical Judge gate (post-findings) + +Explicit checkpoint after findings and before memo / report — an orchestration +pass over the controls above, not a second QA framework. For +`analytical_depth = material / decision_critical`, record an `ANALYTICAL_JUDGE` +result; routine / no-trigger cases collapse to the compact QA note. See +`ANALYTICAL_REASONING_STANDARD.md` §8. + +- [ ] `question_fit?` — analysis answered the declared business question and scope. +- [ ] `method_adequacy?` — selected methods sufficient; every supporting method `execution_status: executed` with `prerequisites_met`. +- [ ] `evidence_lineage_complete?` — each headline conclusion traces executed method → source mart/table/slice → metric/period/grain/filter/baseline → evidence. +- [ ] `alternative_explanation_tested_or_visible?` +- [ ] `contradicting_evidence_or_method_disagreement_not_silently_passed?` +- [ ] `claim_strength <= final_evidence_sufficiency?` — including `driver != root cause`, `correlation != causation`, single-period != systemic. +- [ ] `recommendation_risk_implication_within_verified_evidence?` +- [ ] `ANALYTICAL_JUDGE` status recorded: `pass` / `revise` / `blocked`; `revise` resolved by one bounded correction + passing re-check; `blocked` stops publication. +- [ ] `judge_did_not_become_autonomous_retry_loop?` — no silent self-retry; no method added without registry/trigger support; `blocked != executed`. + ### Material variance diagnostic QA Apply `VARIANCE_DIAGNOSTIC_CONTRACT.md` only to material/decision-critical Plan/Fact cases or a material variance risk: diff --git a/ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md b/ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md index 32bc3c1..7f3d276 100644 --- a/ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md +++ b/ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md @@ -233,6 +233,44 @@ Pass condition: - sets the metric and aggregate gate to `pass`; - does not add unnecessary executive-facing gate tables. +## 11. Analytical Judge gate + +Question: + +```text +Deterministic calculation shows category X contributed 70% of a monthly +variance. No causal test, timing validation, or alternative explanation test +was executed. The draft conclusion says: "Category X is the root cause of the +deterioration." Каков результат Analytical Judge? +``` + +Pass condition: + +- `ANALYTICAL_JUDGE status: revise` (not `pass`); +- reason: a 70% contribution supports a driver candidate / calculated effect + within the observed period, not a root cause; +- `maximum_claim_strength`: "Category X is the main quantified contributor + within the observed period"; +- `required_action`: weaken the claim, or execute the discriminating tests + (timing validation, alternative explanation, causal test) if prerequisites + exist; +- no silent self-retry; no method added without registry/trigger support; +- `driver != root cause` and `claim strength <= final evidence sufficiency` + are cited. + +Question: + +```text +Routine quick task, low uncertainty, no material trigger. Нужен полный +семиквестионный ANALYTICAL_JUDGE record? +``` + +Pass condition: + +- collapses to the compact QA note; +- does not instantiate the full seven-question `ANALYTICAL_JUDGE` record; +- `quick` output stays compact. + ## Smoke QA output ```text diff --git a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_01_CORE_WORKFLOW.md b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_01_CORE_WORKFLOW.md index 69fd227..5ccd797 100644 --- a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_01_CORE_WORKFLOW.md +++ b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_01_CORE_WORKFLOW.md @@ -21,7 +21,7 @@ ChatGPT Project Sources / Knowledge for `[Analytics]`. - production_promotion: no, unless explicitly accepted elsewhere - bundle_type: generated compact upload artifact - source_of_truth: declared granular source files -- source_fingerprint: sha256:fcc7572736f26e5950c33f53cf8e31ddb34e160679cb1fe2a7dd0a805d4e0336 +- source_fingerprint: sha256:60241aaddf4fce9f54d1857f6b33f521830b909cbdd39e97de8a6d212c3045f6 - generator: scripts/build_knowledge_bundles.py --- @@ -123,10 +123,13 @@ question / scope → deterministic calculation → findings → evidence challenge / calibration as required +→ Analytical Judge (pass / revise / blocked) +→ revise or rerun when required +→ final findings → management synthesis when material and management-facing → LLM context package → memo / report -→ judge / QA +→ memo QA / judge → revise or rerun → acceptance → next run trigger @@ -221,6 +224,7 @@ mart_main_full Select the deterministic-first minimum sufficient method set through the registry and intent mapping in `ANALYTICAL_TECHNIQUES.md`. Apply the prerequisite gate before execution. A blocked method is not an executed method and is not supporting evidence. State method, metric, period, grain, data source and limitation. After deterministic findings, apply the preliminary evidence check, explanation challenge, claim calibration, and final evidence sufficiency from `ANALYTICAL_REASONING_STANDARD.md` only to the depth required by the case. Preserve `driver != root cause` and do not silently reconcile material method disagreement. +Then run the **Analytical Judge gate** (`ANALYTICAL_REASONING_STANDARD.md` §8): a compact post-findings orchestration checkpoint over the controls above that returns `pass / revise / blocked` before the findings become narrative. For `analytical_depth = material / decision_critical` the explicit gate is mandatory and a recorded `ANALYTICAL_JUDGE` result (`pass`, or a `revise` resolved by one bounded correction and a passing re-check) is required before memo / report generation; `blocked` stops publication. For routine / low-uncertainty cases with no material trigger it collapses to the existing compact QA note. It adds no method, intent, taxonomy, or QA framework. ## Step 10 — Charts Charts must be sourced from `mart_main_full` or a documented slice derived from it. ## Step 11 — Memo diff --git a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_03_TECHNIQUES_AND_CHARTS.md b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_03_TECHNIQUES_AND_CHARTS.md index cd55057..84de0eb 100644 --- a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_03_TECHNIQUES_AND_CHARTS.md +++ b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_03_TECHNIQUES_AND_CHARTS.md @@ -21,7 +21,7 @@ ChatGPT Project Sources / Knowledge for `[Analytics]`. - production_promotion: no, unless explicitly accepted elsewhere - bundle_type: generated compact upload artifact - source_of_truth: declared granular source files -- source_fingerprint: sha256:7eb89642dffbb0ee2feaa8ee5a157d5d12052515abf06f2b07676618d0262612 +- source_fingerprint: sha256:8f8dab33afe5178d7985bfacf67f65bdf4e562ec25f50114b63e33426dd11fac - generator: scripts/build_knowledge_bundles.py --- @@ -478,7 +478,57 @@ claim → evidence ``` `formula_or_method` remains a human-readable description and does not replace `method_execution_id`. A claim that requires an actual method result cannot use an execution with status `blocked`, `planned`, or `not_needed` as supporting evidence. -## 8. Runtime collapse +## 8. Analytical Judge gate +An explicit, bounded orchestration checkpoint that runs **after deterministic findings and before narrative packaging** (memo / report). It challenges the analytical conclusion itself. It does not add a QA framework, taxonomy, method, intent, or execution state: it reads the controls already defined in §4–§7 and the existing Analytics QA, and returns one visible result. +```text +deterministic calculation +→ findings +→ ANALYTICAL JUDGE +→ pass / revise / blocked +→ revise or rerun when required +→ final findings +→ memo / report +→ existing Memo QA / Judge +→ acceptance +``` +### Seven checks (semantic, over existing controls — no new definitions) +| # | Judge check | Reads | +|---|---|---| +| 1 | Question fit — did the analysis answer the declared business question and scope? | `TASK_PROFILE.question`, active intent classification (§1–§2), `expected_output` | +| 2 | Method adequacy — were the selected methods sufficient, and was every supporting method actually executed with prerequisites met? | `METHOD_PLAN` (`status`, `prerequisites_met`, `execution_status`), the registry and triggers in `ANALYTICAL_TECHNIQUES.md`, `blocked != executed` (§4) | +| 3 | Evidence lineage — can every headline conclusion be traced executed method → source mart / table / slice → metric → period → grain → filter / baseline → evidence? | the lineage in §7 and `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md` | +| 4 | Alternative explanations — is a materially plausible alternative untested or not kept visible? | `COMPETING_EXPLANATIONS[]`, `DISCRIMINATING_EVIDENCE`, `FALSIFICATION_TEST` (§6) | +| 5 | Contradicting evidence — does evidence or method disagreement materially weaken the conclusion? | `CONTRADICTING_EVIDENCE`, `RESIDUAL_UNCERTAINTY`, the method-disagreement rule (§6) | +| 6 | Claim calibration — is any conclusion stronger than `FINAL_EVIDENCE_SUFFICIENCY` (driver → root cause, association → causation, single-period → systemic)? | claim ladder, `claim_support`, `causal_status`, `FINAL_EVIDENCE_SUFFICIENCY`, `claim strength <= final evidence sufficiency` (§7); `VARIANCE_DIAGNOSTIC_CONTRACT.md` for material Plan/Fact | +| 7 | Decision proportionality — does any recommendation, risk statement, or management implication exceed the verified evidence? | the material management synthesis rule (`ANALYTICS_WORKFLOW.md` Step 11) and its acceptance criteria | +The Judge introduces no competing definition for any concept it reads. It reuses `PRELIMINARY_EVIDENCE_CHECK`, the explanation-challenge record, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md`, Analysis QA, and the material variance diagnostic QA where applicable. +### Output +```text +ANALYTICAL_JUDGE +status: pass / revise / blocked +failed_checks: +material_findings: +required_action: +rerun_required: yes / no +manual_review_required: yes / no +maximum_claim_strength: +``` +- **pass** — every material conclusion is supported at its stated claim strength; no material unresolved contradiction or missing analytical check remains. +- **revise** — the conclusion or its wording can be corrected from existing evidence, or a bounded additional deterministic check / rerun is available. +- **blocked** — a required prerequisite, reconciliation, grain, validation path, or discriminating evidence is unavailable; the final management conclusion must not be published. +`maximum_claim_strength` cannot exceed `FINAL_EVIDENCE_SUFFICIENCY.maximum_claim_strength`. `manual_review_required` is the value set by §1; the Judge surfaces it and does not redefine it. +### Bounded revise / rerun +Allowed: +```text +Judge finding +→ one explicit bounded correction or deterministic rerun +→ Judge re-check +``` +Forbidden: silent self-retry; unrestricted iterative analysis; adding a method without registry / trigger support; reasoning around a missing deterministic prerequisite; treating a `blocked` method as evidence. Correction count, stop conditions, and rollback remain those of the canonical `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` and the `[Codex]` one-fix limit where applicable; this gate does not widen them. +### Runtime collapse +`routine` + low uncertainty + no material trigger: collapse the Judge to the existing compact QA note on the compact path (Runtime collapse, §9). Do not instantiate the seven-question `ANALYTICAL_JUDGE` record. +`material` / `decision_critical`: the explicit gate is mandatory. A recorded `ANALYTICAL_JUDGE` with `status: pass` — or a `revise` resolved by one bounded correction and a passing re-check — is required before final findings are handed to memo / report generation. A `blocked` status stops publication of the final management conclusion. +## 9. Runtime collapse P0 is one workflow with conditional depth, not two independent workflows. Eligible compact path: ```text @@ -499,10 +549,11 @@ TASK PROFILE → EXPLANATION CHALLENGE → CLAIM CALIBRATION → FINAL EVIDENCE SUFFICIENCY +→ ANALYTICAL JUDGE (pass / revise / blocked) → FINAL FINDINGS → FULL QA ``` -Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, or full evidence-sufficiency record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default. -## 9. Stop and escalation rules +Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, full evidence-sufficiency record, or the seven-question `ANALYTICAL_JUDGE` record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default. +## 10. Stop and escalation rules Stop analytical deepening when: - another method is unlikely to materially change finding, confidence, risk, recommendation, limitation, or evidence assurance; - required discriminating evidence is unavailable; diff --git a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_05_QA_GOVERNANCE_ROUTING.md b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_05_QA_GOVERNANCE_ROUTING.md index 696616d..7ffe488 100644 --- a/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_05_QA_GOVERNANCE_ROUTING.md +++ b/ChatGPT/[Analytics]/Knowledge_Bundles/ANALYTICS_05_QA_GOVERNANCE_ROUTING.md @@ -24,7 +24,7 @@ ChatGPT Project Sources / Knowledge for `[Analytics]`. - production_promotion: no, unless explicitly accepted elsewhere - bundle_type: generated compact upload artifact - source_of_truth: declared granular source files -- source_fingerprint: sha256:d345e39209df2643666d8910d767fd2f93afd2b16f8954710f84ef3d4a08353d +- source_fingerprint: sha256:ed25446081dce3ae01ffc6c79a4617105593f6f3293313b0f8b06cb5ae508503 - generator: scripts/build_knowledge_bundles.py --- @@ -137,6 +137,21 @@ the record internal or in evidence/appendix so `quick` output remains compact. - [ ] `analytics_extension_applied_without_duplication?` - [ ] `reasoning_control_not_treated_as_autonomous_execution_loop?` Use `ANALYTICAL_REASONING_STANDARD.md` for field semantics. These checks extend the existing Analysis QA; they do not create a separate QA framework. +### Analytical Judge gate (post-findings) +Explicit checkpoint after findings and before memo / report — an orchestration +pass over the controls above, not a second QA framework. For +`analytical_depth = material / decision_critical`, record an `ANALYTICAL_JUDGE` +result; routine / no-trigger cases collapse to the compact QA note. See +`ANALYTICAL_REASONING_STANDARD.md` §8. +- [ ] `question_fit?` — analysis answered the declared business question and scope. +- [ ] `method_adequacy?` — selected methods sufficient; every supporting method `execution_status: executed` with `prerequisites_met`. +- [ ] `evidence_lineage_complete?` — each headline conclusion traces executed method → source mart/table/slice → metric/period/grain/filter/baseline → evidence. +- [ ] `alternative_explanation_tested_or_visible?` +- [ ] `contradicting_evidence_or_method_disagreement_not_silently_passed?` +- [ ] `claim_strength <= final_evidence_sufficiency?` — including `driver != root cause`, `correlation != causation`, single-period != systemic. +- [ ] `recommendation_risk_implication_within_verified_evidence?` +- [ ] `ANALYTICAL_JUDGE` status recorded: `pass` / `revise` / `blocked`; `revise` resolved by one bounded correction + passing re-check; `blocked` stops publication. +- [ ] `judge_did_not_become_autonomous_retry_loop?` — no silent self-retry; no method added without registry/trigger support; `blocked != executed`. ### Material variance diagnostic QA Apply `VARIANCE_DIAGNOSTIC_CONTRACT.md` only to material/decision-critical Plan/Fact cases or a material variance risk: - [ ] Source/raw formula and sign convention remain distinct from normalized management direction; unresolved KPI direction blocks normalization. @@ -310,6 +325,15 @@ A result is accepted when: 11. Before publication, every flagship metric in a quantitative report passes `QUANTITATIVE_SANITY_GATE.md`; otherwise the result is `revise` or `blocked` and is not published as a final quantitative conclusion. +12. For `analytical_depth = material / decision_critical`, the Analytical Judge + gate (`ANALYTICAL_REASONING_STANDARD.md` §8) ran after findings and before + memo / report generation, and an `ANALYTICAL_JUDGE` result is recorded with + `status: pass` (or a `revise` resolved by one bounded correction and a + passing re-check). A `blocked` Judge status means the final management + conclusion is not published. `maximum_claim_strength` does not exceed + `FINAL_EVIDENCE_SUFFICIENCY`; `driver != root cause` and + `correlation != causation` hold. Routine / no-trigger cases satisfy this + through the compact QA note without a full Judge record. ## Main file acceptance ```text stage_main_full: pass/fail/blocked/not_applicable @@ -322,6 +346,7 @@ slices_from_mart_main_full: pass/fail/blocked/not_applicable accepted: yes/no qa_status: pass/fail/blocked quantitative_sanity_gate_status: pass/revise/blocked/not_applicable +analytical_judge_status: pass/revise/blocked/not_applicable confidence: high/medium/low residual_risks: known_limitations: @@ -376,6 +401,9 @@ Use `blocked` when: - compact-only input is insufficient for requested conclusion; - implementation is required before result can be produced. - a required flagship metric has a blocked quantitative sanity gate. +- the Analytical Judge gate returns `blocked` (required prerequisite, + reconciliation, grain, validation path, or discriminating evidence + unavailable). ## Not production-ready rule Smoke QA or a good memo does not equal production readiness. Production readiness requires implementation evidence, tests, acceptance and rollback/release notes where relevant. @@ -490,6 +518,7 @@ AI OS gives evidence and patterns. `[Analytics]` applies them only when they aff - Method eligibility and triggers follow `ANALYTICAL_TECHNIQUES.md`; the LLM cannot silently override the registry. - Reasoning cannot substitute for deterministic execution or missing prerequisites. - Claim strength cannot exceed final evidence sufficiency. +- Every material / decision-critical analytical conclusion passes an explicit post-findings Analytical Judge gate (`ANALYTICAL_REASONING_STANDARD.md` §8) before narrative packaging; the gate orchestrates existing controls and adds no second QA framework. - `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` remains canonical execution governance; `ANALYTICS_EXTENSION.md` supplies domain-specific constraints without creating a second execution framework. ## Evidence labels Use: @@ -512,7 +541,8 @@ Do not publish final management conclusion when: - unsupported cause; - risk without basis; - action without owner/due date; -- no main mart for a mart-based conclusion. +- no main mart for a mart-based conclusion; +- the Analytical Judge gate returns `blocked`. ## Anti-patterns | Anti-pattern | Why bad | Correct action | |---|---|---| @@ -557,6 +587,10 @@ Apply these controls through `ANALYTICAL_REASONING_STANDARD.md` and the existing | Method catalog inflated by converting controls into methods | Require a distinct question, execution procedure, and material analytical effect before adding a method. | | Decision methods leak from `[Thinking]` into `[Analytics]` | Keep trade-offs, reversibility, premortem, risk appetite, choice, and decision in `[Thinking]`. | | Reasoning-control loop mistaken for an autonomous execution loop | Keep reasoning/method selection inside the AES-governed scope, checks, bounded correction, stop, rollback, acceptance, and authority boundaries. | +| Analytical conclusion reaches memo without an explicit post-findings challenge | Run the Analytical Judge gate after findings; record `ANALYTICAL_JUDGE` `pass / revise / blocked` for material / decision-critical cases. | +| Analytical Judge treated as an autonomous retry loop | Allow only `Judge finding → one bounded correction or deterministic rerun → Judge re-check`; no silent self-retry, no unrestricted iteration; AES limits and the `[Codex]` one-fix limit are unchanged. | +| Analytical Judge used as a second QA framework or a new taxonomy | The gate only orchestrates `PRELIMINARY_EVIDENCE_CHECK`, explanation challenge, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY`, Analysis QA, and variance diagnostic QA; it defines no new field. | +| `blocked` method converted into evidence at the Judge step | Enforce `blocked != executed`; a `blocked` prerequisite forces Judge `blocked`, not a weaker `pass`. | ## Metric / artifact explosion Anti-pattern: A short analytical request produces a large workbook, many sheets, or hundreds of columns without explicit need. @@ -749,6 +783,35 @@ Pass condition: - records all required evidence, including both locators and tolerance; - sets the metric and aggregate gate to `pass`; - does not add unnecessary executive-facing gate tables. +## 11. Analytical Judge gate +Question: +```text +Deterministic calculation shows category X contributed 70% of a monthly +variance. No causal test, timing validation, or alternative explanation test +was executed. The draft conclusion says: "Category X is the root cause of the +deterioration." Каков результат Analytical Judge? +``` +Pass condition: +- `ANALYTICAL_JUDGE status: revise` (not `pass`); +- reason: a 70% contribution supports a driver candidate / calculated effect + within the observed period, not a root cause; +- `maximum_claim_strength`: "Category X is the main quantified contributor + within the observed period"; +- `required_action`: weaken the claim, or execute the discriminating tests + (timing validation, alternative explanation, causal test) if prerequisites + exist; +- no silent self-retry; no method added without registry/trigger support; +- `driver != root cause` and `claim strength <= final evidence sufficiency` + are cited. +Question: +```text +Routine quick task, low uncertainty, no material trigger. Нужен полный +семиквестионный ANALYTICAL_JUDGE record? +``` +Pass condition: +- collapses to the compact QA note; +- does not instantiate the full seven-question `ANALYTICAL_JUDGE` record; +- `quick` output stays compact. ## Smoke QA output ```text smoke_qa_status: pass/fail/blocked diff --git a/docs/knowledge_bundle_provenance_audit.json b/docs/knowledge_bundle_provenance_audit.json index 0d30bc2..1e4b2b4 100644 --- a/docs/knowledge_bundle_provenance_audit.json +++ b/docs/knowledge_bundle_provenance_audit.json @@ -227,8 +227,8 @@ "ChatGPT/[Analytics]/Knowledge/MAIN_FILES_STANDARD.md", "ChatGPT/[Analytics]/Knowledge/ANALYTICS_01_CORE_WORKFLOW_BUNDLE_SEMANTICS.md" ], - "source_bytes": 26755, - "bundle_bytes": 27812, + "source_bytes": 27535, + "bundle_bytes": 28591, "classification": "equivalent", "bundle_only_excerpt_or_ref": [], "source_only_excerpt_or_ref": [], @@ -265,8 +265,8 @@ "ChatGPT/[Analytics]/Knowledge/CHART_SELECTION_STANDARD.md", "ChatGPT/[Analytics]/Knowledge/ANALYTICS_03_TECHNIQUES_AND_CHARTS_BUNDLE_SEMANTICS.md" ], - "source_bytes": 68947, - "bundle_bytes": 69967, + "source_bytes": 73824, + "bundle_bytes": 74827, "classification": "equivalent", "bundle_only_excerpt_or_ref": [], "source_only_excerpt_or_ref": [], @@ -309,8 +309,8 @@ "ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md", "ChatGPT/[Analytics]/Knowledge/ANALYTICS_05_QA_GOVERNANCE_ROUTING_BUNDLE_SEMANTICS.md" ], - "source_bytes": 40326, - "bundle_bytes": 41724, + "source_bytes": 45150, + "bundle_bytes": 46536, "classification": "equivalent", "bundle_only_excerpt_or_ref": [], "source_only_excerpt_or_ref": [], diff --git a/docs/knowledge_bundle_provenance_audit.md b/docs/knowledge_bundle_provenance_audit.md index 5d9bc93..f943382 100644 --- a/docs/knowledge_bundle_provenance_audit.md +++ b/docs/knowledge_bundle_provenance_audit.md @@ -185,8 +185,8 @@ Blocking record count: **0** - Project: [Analytics] - Source paths: `ChatGPT/[Analytics]/Knowledge/ANALYTICS_PROJECT_FILES_INDEX.md`, `ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md`, `ChatGPT/[Analytics]/Knowledge/IN_PROJECT_ANALYSIS_MODE.md`, `ChatGPT/[Analytics]/Knowledge/MAIN_FILES_STANDARD.md`, `ChatGPT/[Analytics]/Knowledge/ANALYTICS_01_CORE_WORKFLOW_BUNDLE_SEMANTICS.md` -- Source bytes: 26755 -- Bundle bytes: 27812 +- Source bytes: 27535 +- Bundle bytes: 28591 - Classification: equivalent - Mapping status: mapped - Resolution status: not_applicable @@ -213,8 +213,8 @@ Blocking record count: **0** - Project: [Analytics] - Source paths: `ChatGPT/[Analytics]/Knowledge/ANALYTICAL_TECHNIQUES.md`, `ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md`, `ChatGPT/[Analytics]/Knowledge/VARIANCE_DIAGNOSTIC_CONTRACT.md`, `ChatGPT/[Analytics]/Knowledge/CHART_SELECTION_STANDARD.md`, `ChatGPT/[Analytics]/Knowledge/ANALYTICS_03_TECHNIQUES_AND_CHARTS_BUNDLE_SEMANTICS.md` -- Source bytes: 68947 -- Bundle bytes: 69967 +- Source bytes: 73824 +- Bundle bytes: 74827 - Classification: equivalent - Mapping status: mapped - Resolution status: not_applicable @@ -241,8 +241,8 @@ Blocking record count: **0** - Project: [Analytics] - Source paths: `ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md`, `ChatGPT/[Analytics]/Knowledge/QUANTITATIVE_SANITY_GATE.md`, `ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md`, `ChatGPT/[Analytics]/Knowledge/ROUTING_AND_HANDOFF.md`, `ChatGPT/[Analytics]/Knowledge/AI_OS_REFERENCE.md`, `ChatGPT/[Analytics]/Knowledge/GOVERNANCE_AND_ANTI_PATTERNS.md`, `ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md`, `ChatGPT/[Analytics]/Knowledge/ANALYTICS_05_QA_GOVERNANCE_ROUTING_BUNDLE_SEMANTICS.md` -- Source bytes: 40326 -- Bundle bytes: 41724 +- Source bytes: 45150 +- Bundle bytes: 46536 - Classification: equivalent - Mapping status: mapped - Resolution status: not_applicable