Skip to content

Add Kernel-backed Harbor control arm with agent stop propagation and numeric verifier metrics - #1

Open
rgarcia wants to merge 6 commits into
mainfrom
hypeship/harbor-kernel-control
Open

Add Kernel-backed Harbor control arm with agent stop propagation and numeric verifier metrics#1
rgarcia wants to merge 6 commits into
mainfrom
hypeship/harbor-kernel-control

Conversation

@rgarcia

@rgarcia rgarcia commented Aug 22, 2026

Copy link
Copy Markdown

Summary

Upstream TIGER-AI-Lab/ClawBench now ships the native Kernel browser runtime (TIGER-AI-Lab#311), so this PR is reduced to what upstream does not have: the Harbor integration. The branch is rebased on the current upstream main; the two runtime commits that upstream accepted are dropped as duplicates.

What remains:

  • add a Kernel-backed mode to clawbench-harbor-adapt (--browser-runtime kernel) that attaches ClawBench instrumentation to one pre-created Kernel browser per task, with headful stealth creation and no provider-default viewport override
  • configure stock Harbor agents with pinned Playwright MCP (@playwright/mcp@0.0.79) through the credential-free CDP bridge; the Kernel API key never reaches the agent
  • stop stock Claude Code and Codex cleanly when ClawBench records the final request, and persist stop timing
  • intercept the Red Cross ZIP lookup request rather than blocking the chapter-finder landing page
  • emit numeric verifier metrics (reward, intercepted, judge_match) and timestamped interception evidence
  • finalize the replay, download recording.mp4, delete the browser idempotently (including failure paths via a setup trap) and verify deletion through the provider API
  • docs: new Harbor control-arm section plus changelog entry; tests for adapter wiring, lifecycle scripts, stop wrapper, and provider session checks

Validation

  • uv run pytest -q — 219 passed
  • uv run ruff check src/clawbench tests — clean
  • uv run ruff format --check src/clawbench tests — clean
  • uv run --frozen pyright src/clawbench tests — 0 errors

End-to-end Harbor runs recorded during development of these commits (patches unchanged by the rebase): 3/3 tasks with intercepted=1, judge_match=1, reward=1 on stealth Kernel browsers; median agent-start-to-interception 40.8s; median stop propagation 47ms; all replays retained and all browser deletions verified.

@rgarcia
rgarcia force-pushed the hypeship/harbor-kernel-control branch from 6cf9dc5 to 2a1b2ac Compare August 23, 2026 23:13
@rgarcia rgarcia changed the title Add Kernel-backed Harbor control arm Add Kernel-backed Harbor control arm with agent stop propagation and numeric verifier metrics Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant