feat(evals): Add --claude-judge to run the judge on the Agent SDK - #1299
Open
jirispilka wants to merge 4 commits into
Open
feat(evals): Add --claude-judge to run the judge on the Agent SDK#1299jirispilka wants to merge 4 commits into
jirispilka wants to merge 4 commits into
Conversation
jirispilka
marked this pull request as draft
August 21, 2026 15:13
The judge previously required OPENROUTER_API_KEY. --claude-judge backs it with a short-lived Claude Code subprocess (no tools, one turn) on the same local login --subscription uses, so a run needs only APIFY_TOKEN and the Langfuse keys. --judge-model then takes an Anthropic model ID (default claude-sonnet-5); run metadata records judgeProvider. The SDK exposes no structured-output parameter, so the verdict schema is enforced by instruction plus JSON extraction. parseJudgeResponse also gains a prose-verdict fallback: some OpenRouter providers answer "FAIL. The agent ..." despite the JSON schema, and the fallback recovers the verdict without spending the retry call (which remains the net for verdict-less answers).
- Share the Langfuse usageDetails mapping via toUsageDetails() instead of duplicating the spread in both LLM clients - Compute prompt tokens once and derive totalTokens from it - Extract the JSON object in callLlm; drop the extractJson flag threaded through sendRequest - Reuse JudgeResponseValidator for the prose verdict instead of a cast - Drop a stray eslint-disable on the node:os import
Also mention the --claude-judge path in the config docstring.
jirispilka
force-pushed
the
feat/evals-claude-judge
branch
from
August 22, 2026 20:41
bce9bc6 to
b68ef77
Compare
jirispilka
marked this pull request as ready for review
August 22, 2026 20:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1294. Lets the whole eval harness run on a Claude Code subscription login with zero API keys:
--subscription(from #1294) covers the agent,--claude-judge(this PR) covers the judge.The main motivation is to run judge from claude.ai/code so that it can run over-night.
I would like to run the judge also from claude.ai
Used to run the web-fetch/web-selection eval ladders (suites in the follow-up PR) both in a remote Claude Code sandbox (no keys available) and locally.