Skip to content

Evaluate the V1 abstraction-design workflow - #3521

Merged
joshlf merged 1 commit into
mainfrom
G3y45zv35fuuyeejc26bftqdd33lqz2oh
Sep 2, 2026
Merged

Evaluate the V1 abstraction-design workflow#3521
joshlf merged 1 commit into
mainfrom
G3y45zv35fuuyeejc26bftqdd33lqz2oh

Conversation

@joshlf

@joshlf joshlf commented Aug 3, 2026

Copy link
Copy Markdown
Member

Run a 54-report treatment/core-ablation study over nine abstraction-design
modes. Preserve the fixtures, frozen packages, manifests, raw reports, blind
scores, adjudications, and limitations needed to reproduce the comparison.

No treatment report certifies an unimplemented proposal, while 16 of 27
core-ablation reports do. Treatment matches or exceeds every adjudicated mode
and produces parsimonious designs such as checked construction, real sealing,
safe slice splitting, and receiver-bound lifetimes.

The preregistered gates nevertheless fail. Four treatment reports use
executions containing UB as behavioral counterexamples, and two incorrectly
prove a Rust-1.70 empty-slice pointer loop by promoting producer facts into a
universal invariant. These failures motivate V2's whole-execution verdict,
exact-domain, boundary-case, and producer-quantifier rules.

The study remains exploratory: isolation is procedural, model identity and
sampling seed are unavailable, modes are heterogeneous, and one replicate per
cell cannot establish release readiness.

Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf


This PR is on branch codex/unsafe-rust-stack.

Latest Update: v10 — Compare vs v9

📚 Full Patch History

Links show the diff between the row version and the column version.

Version v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v8 v7 v6 v5 v4 v3 v2 v1 Base
v7 v6 v5 v4 v3 v2 v1 Base
v6 v5 v4 v3 v2 v1 Base
v5 v4 v3 v2 v1 Base
v4 v3 v2 v1 Base
v3 v2 v1 Base
v2 v1 Base
v1 Base
⬇️ Download this PR

Branch

git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git checkout -b pr-G3y45zv35fuuyeejc26bftqdd33lqz2oh FETCH_HEAD

Checkout

git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git checkout FETCH_HEAD

Cherry Pick

git fetch origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh && git cherry-pick FETCH_HEAD

Pull

git pull origin refs/heads/G3y45zv35fuuyeejc26bftqdd33lqz2oh

Stacked PRs enabled by GHerrit.

@codecov-commenter

codecov-commenter commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (Gup6qexxvahgx6uken5vngpapqsaio22h@b565a8b). Learn more about missing BASE report.

Additional details and impacted files
@@                         Coverage Diff                          @@
##             Gup6qexxvahgx6uken5vngpapqsaio22h    #3521   +/-   ##
====================================================================
  Coverage                                     ?   91.85%           
====================================================================
  Files                                        ?       20           
  Lines                                        ?     6093           
  Branches                                     ?        0           
====================================================================
  Hits                                         ?     5597           
  Misses                                       ?      496           
  Partials                                     ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9da92525b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/unsafe-rust/runs/2026-07-31-abstraction-design-v1/manifest.md
Comment thread evals/unsafe-rust/runs/2026-07-31-abstraction-design-v1/manifest.md
@joshlf
joshlf force-pushed the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch from 9da9252 to 5fa7e9e Compare August 29, 2026 10:42
@joshlf
joshlf force-pushed the G6f4nmapfoz5ur4camwh2yhyazexc2lct branch 2 times, most recently from 6dfbeb8 to 788854e Compare August 29, 2026 11:11
@joshlf
joshlf force-pushed the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch from 5fa7e9e to 65a892d Compare August 29, 2026 11:11
@joshlf joshlf changed the title Evaluate V1 legacy behavior and abstraction-design workflow Evaluate the V1 abstraction-design workflow Aug 29, 2026
@joshlf
joshlf changed the base branch from G6f4nmapfoz5ur4camwh2yhyazexc2lct to Gup6qexxvahgx6uken5vngpapqsaio22h August 29, 2026 11:11
@joshlf
joshlf force-pushed the Gup6qexxvahgx6uken5vngpapqsaio22h branch from 6829891 to ba00cb2 Compare September 1, 2026 22:15
@joshlf
joshlf force-pushed the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch from 65a892d to 792ff41 Compare September 1, 2026 22:15
@joshlf
joshlf force-pushed the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch from 792ff41 to 405a8cb Compare September 2, 2026 20:45
Run a 54-report treatment/core-ablation study over nine abstraction-design
modes. Preserve the fixtures, frozen packages, manifests, raw reports, blind
scores, adjudications, and limitations needed to reproduce the comparison.

No treatment report certifies an unimplemented proposal, while 16 of 27
core-ablation reports do. Treatment matches or exceeds every adjudicated mode
and produces parsimonious designs such as checked construction, real sealing,
safe slice splitting, and receiver-bound lifetimes.

The preregistered gates nevertheless fail. Four treatment reports use
executions containing UB as behavioral counterexamples, and two incorrectly
prove a Rust-1.70 empty-slice pointer loop by promoting producer facts into a
universal invariant. These failures motivate V2's whole-execution verdict,
exact-domain, boundary-case, and producer-quantifier rules.

The study remains exploratory: isolation is procedural, model identity and
sampling seed are unavailable, modes are heterogeneous, and one replicate per
cell cannot establish release readiness.

gherrit-pr-id: G3y45zv35fuuyeejc26bftqdd33lqz2oh
Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf
@joshlf
joshlf force-pushed the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch from c1b65d0 to 6c7a55f Compare September 2, 2026 22:28
@joshlf
joshlf enabled auto-merge September 2, 2026 22:29
@joshlf
joshlf added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit 1bfbbdf Sep 2, 2026
105 checks passed
@joshlf
joshlf deleted the G3y45zv35fuuyeejc26bftqdd33lqz2oh branch September 2, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants