feat(evals): Add task-tool workflow eval suites - #1294
Open
jirispilka wants to merge 6 commits into
Open
Conversation
github-actions
Bot
added
t-ai
jirispilka
force-pushed
the
feat/tasks-workflow-evals
branch
from
August 20, 2026 21:09
f02eb04 to
2cd3a7e
Compare
jirispilka
added a commit
that referenced
this pull request
Aug 21, 2026
…1293) Task tools now return the stored task `input` verbatim, under the same key the API, CLI, and apify-client use. Everything in this PR came out of the task-tool workflow evals (#1294): agents could not answer "what input is my task configured with?", could not verify their own updates, and hallucinated values from the field-name list. Not obvious from the code: - **Returning the input is safe.** The platform encrypts input fields declared `isSecret: true` server-side on save. Verified empirically: a task created via the raw API with a plaintext secret returns `ENCRYPTED_VALUE:...` from both `GET actor-task` and `GET actor-task/input`; the plaintext never comes back. Non-secret values are already returned by every other Apify surface. - **`seoDescription` is required to publish**, which no docs state — discovered via API probing after eval agents got stuck. Descriptions now list the full requirements, and the schema declares the SEO length limits (60/160) the API enforces. - **Contract change:** the task tools' `structuredContent` replaces `inputFields` (names only) with `input`. - Tool errors append the machine-readable API error type, e.g. `(API error type: actor-task-name-not-unique)`.
Two Langfuse suites for the Actor task tools, split by doctrine: tasks-evals (7 proper cases) and tasks-evals-errors (3 error-recovery cases that provoke failures on purpose). The run gate now requires zero failed tool calls by default; every item carries a tool_errors score with the failing calls in the comment, and error suites run with --allow-tool-errors. Failures injected via failTools are exempt. Add --subscription (the agent runs on the local Claude Code login instead of ANTHROPIC_API_KEY), a fixtures seed/cleanup script for the eval-* tasks, per-dataset snapshot export, and snapshots for both suites.
A transient SDK connection error dropped a whole item from the run; the agent conversation now retries once, and a persistent failure still throws. Some OpenRouter providers occasionally answer the judge prompt in plain text despite the JSON schema, which discarded a scored item; the judge call now retries once before failing. The update-medium case now pins the sequence (create with default run options, then change them), so setting options at creation can no longer bypass the update path the case exists to cover.
The gate filtered tool invocations on failure alone, and tool names are prefix-stripped before it sees them, so a failed Claude Code built-in (Bash, WebFetch) failed an eval item on a judge PASS. Carry `isMcpTool` from the adapter, computed on the unstripped name, and count only the server's own tools. Also in the harness: - Truncate tool errors to the first line at capture; the full text already sits on the tool span, so it was uploaded three times. - Extract `judgeScore` and `formatToolErrors`, each written twice. - Name the judge parse-attempt count and keep the raw response in the failure, which the retry had dropped. - One snapshot file per dataset, dropping the default-name special case. - Paginate the fixture cleanup and reuse `findMissingEnvVars`.
jirispilka
force-pushed
the
feat/tasks-workflow-evals
branch
from
August 21, 2026 09:57
7cdfaf8 to
abd92cc
Compare
This was referenced Aug 21, 2026
The query asks the agent to "confirm it's actually live", but the reference never said what counts as confirmation, so the judge invented a requirement for a separate lookup and failed runs that had already published successfully. Say it explicitly: a successful publish-actor-task is the confirmation, since its response carries publishedAt. Also record two things probed against the API: publishing requires all three of inputSchemaFields, datasetView and seoDescription, and the API reports missing ones non-exhaustively, which is what task-publish-discovery spends its turns on. And note task-chain-hard-1's ~6/8 rate on Haiku with the guessed-slug residual, so a single red run is not read as a break.
Sonnet passes task-chain-hard-1 3/3, Haiku about 5 in 8, and every Haiku failure is the same constructed Actor name. Records that hardening fetch-actor-details' description was measured at 5/8 against ~7/10 without it, so the next person does not spend another round rewording descriptions that already say it twice.
jirispilka
commented
Aug 21, 2026
| // The Agent SDK's Claude Code subprocess falls back to the local login only when no | ||
| // API key is in its environment, which it inherits from this process. | ||
| if (argv.subscription) { | ||
| delete process.env.ANTHROPIC_API_KEY; |
Collaborator
Author
There was a problem hiding this comment.
This is not a good solution but I guess we can live it for now 😊
jirispilka
marked this pull request as draft
August 21, 2026 15:12
jirispilka
commented
Aug 22, 2026
| // far more expensive agent conversation; a second malformed answer still throws. | ||
| let lastError: unknown; | ||
| let lastRawResponse = ''; | ||
| for (let attempt = 1; attempt <= JUDGE_PARSE_ATTEMPTS; attempt++) { |
Collaborator
Author
There was a problem hiding this comment.
I've added attempts, sometimes the judge failed on a slow interner
jirispilka
marked this pull request as ready for review
August 22, 2026 20:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workflow eval suites for the Actor task tools. Closes #1288.
I created a new dataset specifically for this, we'll merge it later.
I did not want to mixed it up with existing evals as they are not "clean" (=there are errors)
We will improve the test cases later. I think they are overly verbose, no one gonna prompt in that way.
I'm sorry, we'll need to clean this up afterwards but I don't have time for it now 😊
tasks-evals(7 proper cases) andtasks-evals-errors(3 cases that provoke errors: name collision, not-found lookup, publish-requirement discovery). Error-provoking cases inside a proper suite mask real failures, so they are quarantined and run with--allow-tool-errors.--subscriptionruns the agent on the local Claude Code login instead ofANTHROPIC_API_KEY.