Skip to content

Add Harbor and ClawBench MCP benchmarks - #162

Open
rgarcia wants to merge 18 commits into
mainfrom
hypeship/harbor-mcp-benchmark
Open

Add Harbor and ClawBench MCP benchmarks#162
rgarcia wants to merge 18 commits into
mainfrom
hypeship/harbor-mcp-benchmark

Conversation

@rgarcia

@rgarcia rgarcia commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a source-pinned Harbor smoke benchmark for stock Claude Code and Codex agents in Hypeman
  • verify native MCP calls through paired ATIF observations without a telemetry proxy
  • add an opt-in MCP toolset allowlist for controlled benchmark deployments
  • add a ClawBench Kernel MCP arm that reuses one pre-created, instrumented stealth browser and disables lifecycle and managed-auth toolsets
  • verify clean stop propagation while accepting the expected interrupted terminal tool observation
  • pin Harbor 0.21.0 with harbor-hypeman 0.1.1 for reproducible runs

Validation

  • bun test (249 passed)
  • production image build
  • Harbor smoke verifier tests (4 passed)
  • ClawBench control-arm tests (6 passed)
  • Kernel MCP arm on stealth Kernel browsers: 3/3 intercepted=1, judge_match=1, reward=1, infra_ok=1
  • median agent-start-to-interception: 74.0s (51.7–89.9s)
  • median first-browser-tool-to-interception: 59.6s (38.3–65.4s)
  • median stop propagation: 56ms (53–100ms)
  • median Kernel MCP calls: 9 (6–10)
  • exact session reuse, project scope, source identity, replay retention, and browser deletion verified in every attempt

@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mcp Ready Ready Preview Aug 22, 2026 9:50pm

Comment thread benchmarks/harbor/run-smoke.sh
Comment thread benchmarks/harbor/clawbench/prepare-control.py
Comment thread benchmarks/harbor/clawbench/prepare-control.py Outdated
@rgarcia rgarcia changed the title Add Harbor MCP benchmark harness Add Harbor and ClawBench MCP benchmarks Aug 22, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7475b8a. Configure here.

Comment thread benchmarks/harbor/clawbench/verify-control.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant