Skip to content

Freeze the V3 targeted confirmation protocol - #3527

Merged
joshlf merged 1 commit into
mainfrom
G5k3oylk4nmllritz5hdpiksvqbp24ffs
Sep 3, 2026
Merged

Freeze the V3 targeted confirmation protocol#3527
joshlf merged 1 commit into
mainfrom
G5k3oylk4nmllritz5hdpiksvqbp24ffs

Conversation

@joshlf

@joshlf joshlf commented Aug 3, 2026

Copy link
Copy Markdown
Member

Freeze a blind 80-report evaluation of V3 against V2 after independent review
and preregistration refinement: eight modes, two conditions, and five fresh
replicates per cell. Seal condition maps, prompts, fixtures, frozen skill
packages, rubrics, allowed authority, expected atoms, and report and scoring
schemas before generation begins.

Exercise symbolic release domains, nonlinear policy composition,
configuration products, existential unsoundness certificates,
whole-execution behavioral claims, positive multi-version proofs,
abstraction-design firewalls, and regression breadth. Require V3 to pass every
required atom in every replicate with zero hard errors, authority defects,
proposal laundering, semantic noncompletion, and scope or budget failures.

Add an append-only event ledger and explicit attempt lifecycle so generation,
validation, retries, blind scoring, adjudication, and unblinding remain
auditable. At freeze time the report count is zero, so later results cannot
have influenced the protocol or success criteria.

Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf


This PR is on branch codex/unsafe-rust-stack.

Latest Update: v14 — Compare vs v13

📚 Full Patch History

Links show the diff between the row version and the column version.

Version v13 v12 v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v14 v13 v12 v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v13 v12 v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v12 v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v8 v7 v6 v5 v4 v3 v2 v1 Base
v7 v6 v5 v4 v3 v2 v1 Base
v6 v5 v4 v3 v2 v1 Base
v5 v4 v3 v2 v1 Base
v4 v3 v2 v1 Base
v3 v2 v1 Base
v2 v1 Base
v1 Base
⬇️ Download this PR

Branch

git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git checkout -b pr-G5k3oylk4nmllritz5hdpiksvqbp24ffs FETCH_HEAD

Checkout

git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git checkout FETCH_HEAD

Cherry Pick

git fetch origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs && git cherry-pick FETCH_HEAD

Pull

git pull origin refs/heads/G5k3oylk4nmllritz5hdpiksvqbp24ffs

Stacked PRs enabled by GHerrit.

@codecov-commenter

codecov-commenter commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.85%. Comparing base (e89f90c) to head (4e58a75).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3527   +/-   ##
=======================================
  Coverage   91.85%   91.85%           
=======================================
  Files          20       20           
  Lines        6093     6093           
=======================================
  Hits         5597     5597           
  Misses        496      496           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6880d2b469

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/unsafe-rust/runs/2026-07-31-v3-targeted/prepare.py
Comment thread evals/unsafe-rust/runs/2026-07-31-v3-targeted/protocol.py
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 6880d2b to a16f872 Compare August 29, 2026 10:42
@joshlf
joshlf force-pushed the Gd5c25nyjtgu5tav2bs53xvoz525b6kpe branch from 34363db to 1cfd581 Compare August 29, 2026 10:42
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from a16f872 to 92001a9 Compare August 29, 2026 11:11
@joshlf
joshlf changed the base branch from Gd5c25nyjtgu5tav2bs53xvoz525b6kpe to Gcbutfa3xssppp2tjpxo2fjo6kguufo6t August 29, 2026 11:11
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from b87ebc8 to 9401243 Compare September 1, 2026 22:15
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 92001a9 to 66480de Compare September 1, 2026 22:15
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from 9401243 to 5d3022d Compare September 2, 2026 20:45
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 66480de to 09052a6 Compare September 2, 2026 20:45
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from 5d3022d to edf9e1a Compare September 2, 2026 21:51
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 09052a6 to 2d99774 Compare September 2, 2026 21:51
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from edf9e1a to 96e9d25 Compare September 2, 2026 22:28
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 2d99774 to 06a5a3e Compare September 2, 2026 22:28
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from 96e9d25 to 36a04b4 Compare September 2, 2026 23:04
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from 06a5a3e to 0faa5b5 Compare September 2, 2026 23:04
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from 36a04b4 to 53958dd Compare September 3, 2026 00:07
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch 2 times, most recently from 28657b2 to d5bc5c7 Compare September 3, 2026 01:00
@joshlf
joshlf force-pushed the Gcbutfa3xssppp2tjpxo2fjo6kguufo6t branch from 53958dd to c37eb34 Compare September 3, 2026 01:00
Base automatically changed from Gcbutfa3xssppp2tjpxo2fjo6kguufo6t to main September 3, 2026 01:32
Freeze a blind 80-report evaluation of V3 against V2 after independent review
and preregistration refinement: eight modes, two conditions, and five fresh
replicates per cell. Seal condition maps, prompts, fixtures, frozen skill
packages, rubrics, allowed authority, expected atoms, and report and scoring
schemas before generation begins.

Exercise symbolic release domains, nonlinear policy composition,
configuration products, existential unsoundness certificates,
whole-execution behavioral claims, positive multi-version proofs,
abstraction-design firewalls, and regression breadth. Require V3 to pass every
required atom in every replicate with zero hard errors, authority defects,
proposal laundering, semantic noncompletion, and scope or budget failures.

Add an append-only event ledger and explicit attempt lifecycle so generation,
validation, retries, blind scoring, adjudication, and unblinding remain
auditable. At freeze time the report count is zero, so later results cannot
have influenced the protocol or success criteria.

gherrit-pr-id: G5k3oylk4nmllritz5hdpiksvqbp24ffs
Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf
@joshlf
joshlf force-pushed the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch from d5bc5c7 to 4e58a75 Compare September 3, 2026 01:33
@joshlf
joshlf enabled auto-merge September 3, 2026 01:34
@joshlf
joshlf added this pull request to the merge queue Sep 3, 2026
Merged via the queue into main with commit e90aed1 Sep 3, 2026
105 checks passed
@joshlf
joshlf deleted the G5k3oylk4nmllritz5hdpiksvqbp24ffs branch September 3, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants