Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions ChatGPT/[Analytics]/Knowledge/ACCEPTANCE_CRITERIA.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,15 @@ A result is accepted when:
11. Before publication, every flagship metric in a quantitative report passes
`QUANTITATIVE_SANITY_GATE.md`; otherwise the result is `revise` or
`blocked` and is not published as a final quantitative conclusion.
12. For `analytical_depth = material / decision_critical`, the Analytical Judge
gate (`ANALYTICAL_REASONING_STANDARD.md` §8) ran after findings and before
memo / report generation, and an `ANALYTICAL_JUDGE` result is recorded with
`status: pass` (or a `revise` resolved by one bounded correction and a
passing re-check). A `blocked` Judge status means the final management
conclusion is not published. `maximum_claim_strength` does not exceed
`FINAL_EVIDENCE_SUFFICIENCY`; `driver != root cause` and
`correlation != causation` hold. Routine / no-trigger cases satisfy this
through the compact QA note without a full Judge record.

## Main file acceptance

Expand All @@ -31,6 +40,7 @@ slices_from_mart_main_full: pass/fail/blocked/not_applicable
accepted: yes/no
qa_status: pass/fail/blocked
quantitative_sanity_gate_status: pass/revise/blocked/not_applicable
analytical_judge_status: pass/revise/blocked/not_applicable
confidence: high/medium/low
residual_risks:
known_limitations:
Expand Down Expand Up @@ -103,6 +113,9 @@ Use `blocked` when:
- compact-only input is insufficient for requested conclusion;
- implementation is required before result can be produced.
- a required flagship metric has a blocked quantitative sanity gate.
- the Analytical Judge gate returns `blocked` (required prerequisite,
reconciliation, grain, validation path, or discriminating evidence
unavailable).

## Not production-ready rule

Expand Down
74 changes: 71 additions & 3 deletions ChatGPT/[Analytics]/Knowledge/ANALYTICAL_REASONING_STANDARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,74 @@ claim

`formula_or_method` remains a human-readable description and does not replace `method_execution_id`. A claim that requires an actual method result cannot use an execution with status `blocked`, `planned`, or `not_needed` as supporting evidence.

## 8. Runtime collapse
## 8. Analytical Judge gate

An explicit, bounded orchestration checkpoint that runs **after deterministic findings and before narrative packaging** (memo / report). It challenges the analytical conclusion itself. It does not add a QA framework, taxonomy, method, intent, or execution state: it reads the controls already defined in §4–§7 and the existing Analytics QA, and returns one visible result.

```text
deterministic calculation
→ findings
→ ANALYTICAL JUDGE
→ pass / revise / blocked
→ revise or rerun when required
→ final findings
→ memo / report
→ existing Memo QA / Judge
→ acceptance
```

### Seven checks (semantic, over existing controls — no new definitions)

| # | Judge check | Reads |
|---|---|---|
| 1 | Question fit — did the analysis answer the declared business question and scope? | `TASK_PROFILE.question`, active intent classification (§1–§2), `expected_output` |
| 2 | Method adequacy — were the selected methods sufficient, and was every supporting method actually executed with prerequisites met? | `METHOD_PLAN` (`status`, `prerequisites_met`, `execution_status`), the registry and triggers in `ANALYTICAL_TECHNIQUES.md`, `blocked != executed` (§4) |
| 3 | Evidence lineage — can every headline conclusion be traced executed method → source mart / table / slice → metric → period → grain → filter / baseline → evidence? | the lineage in §7 and `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md` |
| 4 | Alternative explanations — is a materially plausible alternative untested or not kept visible? | `COMPETING_EXPLANATIONS[]`, `DISCRIMINATING_EVIDENCE`, `FALSIFICATION_TEST` (§6) |
| 5 | Contradicting evidence — does evidence or method disagreement materially weaken the conclusion? | `CONTRADICTING_EVIDENCE`, `RESIDUAL_UNCERTAINTY`, the method-disagreement rule (§6) |
| 6 | Claim calibration — is any conclusion stronger than `FINAL_EVIDENCE_SUFFICIENCY` (driver → root cause, association → causation, single-period → systemic)? | claim ladder, `claim_support`, `causal_status`, `FINAL_EVIDENCE_SUFFICIENCY`, `claim strength <= final evidence sufficiency` (§7); `VARIANCE_DIAGNOSTIC_CONTRACT.md` for material Plan/Fact |
| 7 | Decision proportionality — does any recommendation, risk statement, or management implication exceed the verified evidence? | the material management synthesis rule (`ANALYTICS_WORKFLOW.md` Step 11) and its acceptance criteria |

The Judge introduces no competing definition for any concept it reads. It reuses `PRELIMINARY_EVIDENCE_CHECK`, the explanation-challenge record, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY_TEMPLATE.md`, Analysis QA, and the material variance diagnostic QA where applicable.

### Output

```text
ANALYTICAL_JUDGE
status: pass / revise / blocked
failed_checks:
material_findings:
required_action:
rerun_required: yes / no
manual_review_required: yes / no
maximum_claim_strength:
```

- **pass** — every material conclusion is supported at its stated claim strength; no material unresolved contradiction or missing analytical check remains.
- **revise** — the conclusion or its wording can be corrected from existing evidence, or a bounded additional deterministic check / rerun is available.
- **blocked** — a required prerequisite, reconciliation, grain, validation path, or discriminating evidence is unavailable; the final management conclusion must not be published.

`maximum_claim_strength` cannot exceed `FINAL_EVIDENCE_SUFFICIENCY.maximum_claim_strength`. `manual_review_required` is the value set by §1; the Judge surfaces it and does not redefine it.

### Bounded revise / rerun

Allowed:

```text
Judge finding
→ one explicit bounded correction or deterministic rerun
→ Judge re-check
```

Forbidden: silent self-retry; unrestricted iterative analysis; adding a method without registry / trigger support; reasoning around a missing deterministic prerequisite; treating a `blocked` method as evidence. Correction count, stop conditions, and rollback remain those of the canonical `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` and the `[Codex]` one-fix limit where applicable; this gate does not widen them.

### Runtime collapse

`routine` + low uncertainty + no material trigger: collapse the Judge to the existing compact QA note on the compact path (Runtime collapse, §9). Do not instantiate the seven-question `ANALYTICAL_JUDGE` record.

`material` / `decision_critical`: the explicit gate is mandatory. A recorded `ANALYTICAL_JUDGE` with `status: pass` — or a `revise` resolved by one bounded correction and a passing re-check — is required before final findings are handed to memo / report generation. A `blocked` status stops publication of the final management conclusion.

## 9. Runtime collapse

P0 is one workflow with conditional depth, not two independent workflows.

Expand All @@ -419,12 +486,13 @@ TASK PROFILE
→ EXPLANATION CHALLENGE
→ CLAIM CALIBRATION
→ FINAL EVIDENCE SUFFICIENCY
→ ANALYTICAL JUDGE (pass / revise / blocked) → FINAL FINDINGS
→ FULL QA
```

Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, or full evidence-sufficiency record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default.
Without a material trigger, do not instantiate an unnecessary full Explanation Challenge, competing-explanations structure, falsification structure, expanded claim-registry state, full evidence-sufficiency record, or the seven-question `ANALYTICAL_JUDGE` record. `quick` retains the existing quick-mode artifact and context budget and does not become a full governance artifact by default.

## 9. Stop and escalation rules
## 10. Stop and escalation rules

Stop analytical deepening when:

Expand Down
7 changes: 6 additions & 1 deletion ChatGPT/[Analytics]/Knowledge/ANALYTICS_WORKFLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,13 @@ question / scope
→ deterministic calculation
→ findings
→ evidence challenge / calibration as required
→ Analytical Judge (pass / revise / blocked)
→ revise or rerun when required
→ final findings
→ management synthesis when material and management-facing
→ LLM context package
→ memo / report
judge / QA
memo QA / judge
→ revise or rerun
→ acceptance
→ next run trigger
Expand Down Expand Up @@ -188,6 +191,8 @@ State method, metric, period, grain, data source and limitation.

After deterministic findings, apply the preliminary evidence check, explanation challenge, claim calibration, and final evidence sufficiency from `ANALYTICAL_REASONING_STANDARD.md` only to the depth required by the case. Preserve `driver != root cause` and do not silently reconcile material method disagreement.

Then run the **Analytical Judge gate** (`ANALYTICAL_REASONING_STANDARD.md` §8): a compact post-findings orchestration checkpoint over the controls above that returns `pass / revise / blocked` before the findings become narrative. For `analytical_depth = material / decision_critical` the explicit gate is mandatory and a recorded `ANALYTICAL_JUDGE` result (`pass`, or a `revise` resolved by one bounded correction and a passing re-check) is required before memo / report generation; `blocked` stops publication. For routine / low-uncertainty cases with no material trigger it collapses to the existing compact QA note. It adds no method, intent, taxonomy, or QA framework.

## Step 10 — Charts

Charts must be sourced from `mart_main_full` or a documented slice derived from it.
Expand Down
33 changes: 33 additions & 0 deletions ChatGPT/[Analytics]/Knowledge/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# Changelog

## 2026-09-02 — analytical-judge-gate (issue #357)

Added:

- Explicit `Analytical Judge` gate as `ANALYTICAL_REASONING_STANDARD.md` §8: a
compact post-findings orchestration checkpoint (`findings → Analytical Judge
→ pass / revise / blocked → final findings → memo`). Seven semantic checks
mapped to existing controls; compact `ANALYTICAL_JUDGE` output record;
bounded revise/rerun rule; `quick` runtime collapse. No new QA framework,
taxonomy, method, or intent; §8→§9, §9→§10 renumbered.

Updated:

- `ANALYTICS_WORKFLOW.md` canonical workflow and Step 9 to make the gate
explicit before memo/report.
- `QA_CHECKLIST.md` with a post-findings Analytical Judge gate block
(orchestration, not a second framework).
- `ACCEPTANCE_CRITERIA.md` with criterion 12, `analytical_judge_status`, and a
blocked-status entry.
- `GOVERNANCE_AND_ANTI_PATTERNS.md` with the gate principle, a blocker entry,
and anti-pattern rows (no autonomous retry loop, no second framework,
`blocked != executed`).
- `SMOKE_QA_FOR_ANALYTICS.md` with a case forcing a plausible over-strong
"root cause" claim to `revise`, plus a `quick`-mode collapse case.

Status:

```text
production_ready: not claimed
pilot_case_required: yes
smoke_qa_status: pass
```

## 2026-05-25 — analytics-project-settings-minor-fix

Added:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
- Method eligibility and triggers follow `ANALYTICAL_TECHNIQUES.md`; the LLM cannot silently override the registry.
- Reasoning cannot substitute for deterministic execution or missing prerequisites.
- Claim strength cannot exceed final evidence sufficiency.
- Every material / decision-critical analytical conclusion passes an explicit post-findings Analytical Judge gate (`ANALYTICAL_REASONING_STANDARD.md` §8) before narrative packaging; the gate orchestrates existing controls and adds no second QA framework.
- `docs/standards/AUTONOMOUS_EXECUTION_STANDARD.md` remains canonical execution governance; `ANALYTICS_EXTENSION.md` supplies domain-specific constraints without creating a second execution framework.

## Evidence labels
Expand Down Expand Up @@ -39,7 +40,8 @@ Do not publish final management conclusion when:
- unsupported cause;
- risk without basis;
- action without owner/due date;
- no main mart for a mart-based conclusion.
- no main mart for a mart-based conclusion;
- the Analytical Judge gate returns `blocked`.

## Anti-patterns

Expand Down Expand Up @@ -89,6 +91,10 @@ Apply these controls through `ANALYTICAL_REASONING_STANDARD.md` and the existing
| Method catalog inflated by converting controls into methods | Require a distinct question, execution procedure, and material analytical effect before adding a method. |
| Decision methods leak from `[Thinking]` into `[Analytics]` | Keep trade-offs, reversibility, premortem, risk appetite, choice, and decision in `[Thinking]`. |
| Reasoning-control loop mistaken for an autonomous execution loop | Keep reasoning/method selection inside the AES-governed scope, checks, bounded correction, stop, rollback, acceptance, and authority boundaries. |
| Analytical conclusion reaches memo without an explicit post-findings challenge | Run the Analytical Judge gate after findings; record `ANALYTICAL_JUDGE` `pass / revise / blocked` for material / decision-critical cases. |
| Analytical Judge treated as an autonomous retry loop | Allow only `Judge finding → one bounded correction or deterministic rerun → Judge re-check`; no silent self-retry, no unrestricted iteration; AES limits and the `[Codex]` one-fix limit are unchanged. |
| Analytical Judge used as a second QA framework or a new taxonomy | The gate only orchestrates `PRELIMINARY_EVIDENCE_CHECK`, explanation challenge, `FINAL_EVIDENCE_SUFFICIENCY`, `CLAIM_EVIDENCE_REGISTRY`, Analysis QA, and variance diagnostic QA; it defines no new field. |
| `blocked` method converted into evidence at the Judge step | Enforce `blocked != executed`; a `blocked` prerequisite forces Judge `blocked`, not a weaker `pass`. |

## Metric / artifact explosion

Expand Down
18 changes: 18 additions & 0 deletions ChatGPT/[Analytics]/Knowledge/QA_CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,24 @@ the record internal or in evidence/appendix so `quick` output remains compact.

Use `ANALYTICAL_REASONING_STANDARD.md` for field semantics. These checks extend the existing Analysis QA; they do not create a separate QA framework.

### Analytical Judge gate (post-findings)

Explicit checkpoint after findings and before memo / report — an orchestration
pass over the controls above, not a second QA framework. For
`analytical_depth = material / decision_critical`, record an `ANALYTICAL_JUDGE`
result; routine / no-trigger cases collapse to the compact QA note. See
`ANALYTICAL_REASONING_STANDARD.md` §8.

- [ ] `question_fit?` — analysis answered the declared business question and scope.
- [ ] `method_adequacy?` — selected methods sufficient; every supporting method `execution_status: executed` with `prerequisites_met`.
- [ ] `evidence_lineage_complete?` — each headline conclusion traces executed method → source mart/table/slice → metric/period/grain/filter/baseline → evidence.
- [ ] `alternative_explanation_tested_or_visible?`
- [ ] `contradicting_evidence_or_method_disagreement_not_silently_passed?`
- [ ] `claim_strength <= final_evidence_sufficiency?` — including `driver != root cause`, `correlation != causation`, single-period != systemic.
- [ ] `recommendation_risk_implication_within_verified_evidence?`
- [ ] `ANALYTICAL_JUDGE` status recorded: `pass` / `revise` / `blocked`; `revise` resolved by one bounded correction + passing re-check; `blocked` stops publication.
- [ ] `judge_did_not_become_autonomous_retry_loop?` — no silent self-retry; no method added without registry/trigger support; `blocked != executed`.

### Material variance diagnostic QA

Apply `VARIANCE_DIAGNOSTIC_CONTRACT.md` only to material/decision-critical Plan/Fact cases or a material variance risk:
Expand Down
38 changes: 38 additions & 0 deletions ChatGPT/[Analytics]/Knowledge/SMOKE_QA_FOR_ANALYTICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,44 @@ Pass condition:
- sets the metric and aggregate gate to `pass`;
- does not add unnecessary executive-facing gate tables.

## 11. Analytical Judge gate

Question:

```text
Deterministic calculation shows category X contributed 70% of a monthly
variance. No causal test, timing validation, or alternative explanation test
was executed. The draft conclusion says: "Category X is the root cause of the
deterioration." Каков результат Analytical Judge?
```

Pass condition:

- `ANALYTICAL_JUDGE status: revise` (not `pass`);
- reason: a 70% contribution supports a driver candidate / calculated effect
within the observed period, not a root cause;
- `maximum_claim_strength`: "Category X is the main quantified contributor
within the observed period";
- `required_action`: weaken the claim, or execute the discriminating tests
(timing validation, alternative explanation, causal test) if prerequisites
exist;
- no silent self-retry; no method added without registry/trigger support;
- `driver != root cause` and `claim strength <= final evidence sufficiency`
are cited.

Question:

```text
Routine quick task, low uncertainty, no material trigger. Нужен полный
семиквестионный ANALYTICAL_JUDGE record?
```

Pass condition:

- collapses to the compact QA note;
- does not instantiate the full seven-question `ANALYTICAL_JUDGE` record;
- `quick` output stays compact.

## Smoke QA output

```text
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ChatGPT Project Sources / Knowledge for `[Analytics]`.
- production_promotion: no, unless explicitly accepted elsewhere
- bundle_type: generated compact upload artifact
- source_of_truth: declared granular source files
- source_fingerprint: sha256:fcc7572736f26e5950c33f53cf8e31ddb34e160679cb1fe2a7dd0a805d4e0336
- source_fingerprint: sha256:60241aaddf4fce9f54d1857f6b33f521830b909cbdd39e97de8a6d212c3045f6
- generator: scripts/build_knowledge_bundles.py

---
Expand Down Expand Up @@ -123,10 +123,13 @@ question / scope
→ deterministic calculation
→ findings
→ evidence challenge / calibration as required
→ Analytical Judge (pass / revise / blocked)
→ revise or rerun when required
→ final findings
→ management synthesis when material and management-facing
→ LLM context package
→ memo / report
judge / QA
memo QA / judge
→ revise or rerun
→ acceptance
→ next run trigger
Expand Down Expand Up @@ -221,6 +224,7 @@ mart_main_full
Select the deterministic-first minimum sufficient method set through the registry and intent mapping in `ANALYTICAL_TECHNIQUES.md`. Apply the prerequisite gate before execution. A blocked method is not an executed method and is not supporting evidence.
State method, metric, period, grain, data source and limitation.
After deterministic findings, apply the preliminary evidence check, explanation challenge, claim calibration, and final evidence sufficiency from `ANALYTICAL_REASONING_STANDARD.md` only to the depth required by the case. Preserve `driver != root cause` and do not silently reconcile material method disagreement.
Then run the **Analytical Judge gate** (`ANALYTICAL_REASONING_STANDARD.md` §8): a compact post-findings orchestration checkpoint over the controls above that returns `pass / revise / blocked` before the findings become narrative. For `analytical_depth = material / decision_critical` the explicit gate is mandatory and a recorded `ANALYTICAL_JUDGE` result (`pass`, or a `revise` resolved by one bounded correction and a passing re-check) is required before memo / report generation; `blocked` stops publication. For routine / low-uncertainty cases with no material trigger it collapses to the existing compact QA note. It adds no method, intent, taxonomy, or QA framework.
## Step 10 — Charts
Charts must be sourced from `mart_main_full` or a documented slice derived from it.
## Step 11 — Memo
Expand Down
Loading
Loading