Skip to content

feat(studio): add guided scenario evaluation release gate - #963

Open
shenchengtsi wants to merge 2 commits into
volcengine:mainfrom
shenchengtsi:feat/scenario-evaluation-release-quality-check
Open

feat(studio): add guided scenario evaluation release gate#963
shenchengtsi wants to merge 2 commits into
volcengine:mainfrom
shenchengtsi:feat/scenario-evaluation-release-quality-check

Conversation

@shenchengtsi

@shenchengtsi shenchengtsi commented Aug 21, 2026

Copy link
Copy Markdown

Summary

  • add persisted scenario, dataset, evaluator, policy, evaluated-version, formal-run, badcase, and publish-audit lifecycle
  • guide Studio users through a visible step-by-step evaluation flow with explicit publish feedback and evaluator calibration
  • automatically pass immutable scene criteria, hard-failure conditions, and published case criteria into evaluator judgment without exposing expected answers to the Agent under test
  • support deterministic required/forbidden regular-expression checks with bounded validation and execution timeout; keep LLM rubric text as optional supplemental guidance
  • freeze evaluated code and deployment profiles, then enforce owner-scoped, audited normal, skip, and risk release gates for direct and GitHub delivery

Safety and compatibility

  • require Agent management access for scenario data and Admin access for formal evaluation
  • protect Agent assets with verified owner ACL, admin handoff, source-fingerprint gates, and server-checked Runtime identity
  • hydrate previously persisted evaluator versions from their immutable scene version before formal evaluation
  • require every structured LLM judgment to explicitly classify hard-failure status and preserve dynamic hard failures through calibration and release decisions
  • limit regular expressions to 512 characters and 20 ms execution; return controlled 422 responses for invalid patterns
  • execute only server-generated, safety-checked, signed project snapshots in formal evaluation; support shared rotatable attestation keys
  • keep legacy deployment available when scenario storage is not configured
  • mark GitHub source synchronization as submitted until Runtime delivery is actually Ready

Verification

  • uv run --group dev pre-commit run --all-files
  • uv run --frozen pytest -q — 2718 passed, 5 skipped, 36 warnings, 6 subtests passed
  • scenario-evaluation focused tests — 135 passed
  • npm test — 835 passed
  • npm run build
  • final blocking review — no Critical or Important findings

Validation boundary

  • live Ark planner and real Pi runtime smoke tests remain opt-in and were skipped
  • real cloud Runtime, TOS, and AgentKit acceptance still requires target Agent, cases, credentials, and environment data

@shenchengtsi
shenchengtsi force-pushed the feat/scenario-evaluation-release-quality-check branch from 009f76b to ea0c72d Compare August 23, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant