Skip to content

test(eval): add SimilarWeb budget stop-loss cases - #19

Open
eddiearc wants to merge 1 commit into
feat/resource-plan-modefrom
codex/aisa-pricing-contract-docs
Open

test(eval): add SimilarWeb budget stop-loss cases#19
eddiearc wants to merge 1 commit into
feat/resource-plan-modefrom
codex/aisa-pricing-contract-docs

Conversation

@eddiearc

@eddiearc eddiearc commented Sep 1, 2026

Copy link
Copy Markdown

Summary

  • add a deterministic high-cost SimilarSites hard-budget regression: the plan must remain over_budget rather than become runnable
  • add the matching natural-language user request to the evaluation suite
  • document that the scenarios are sanitized feedback-derived regressions; no customer identifiers or raw prompts are stored

Validation

  • npm run build
  • npm test (169 tests)
  • npm run eval (24/24 scripted scenarios)

The harness remains local-only and poisons AISA_API_KEY, so it cannot make a paid AIsa gateway request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant