Skip to content

feat(evals): add Tone Appropriateness onto the ela-writing taxonomy - #183

Open
adnanrhussain wants to merge 3 commits into
mainfrom
feedback-tone-appropriateness
Open

feat(evals): add Tone Appropriateness onto the ela-writing taxonomy#183
adnanrhussain wants to merge 3 commits into
mainfrom
feedback-tone-appropriateness

Conversation

@adnanrhussain

Copy link
Copy Markdown
Collaborator

Migrates Tone Appropriateness onto the evals/feedback/ela-writing/ taxonomy as Tone Appropriateness, at evals/feedback/ela-writing/tone-appropriateness/. Same pattern as the student-facing-text family (#159, #161, #163, #173#177). The evaluator already had a complete contract; this moves it, renames it, and closes the gaps that family standardised on.

Evaluator

  • evaluator.id: feedback.ela_writing.tone_appropriateness
  • stable_id: e6d674bd-2c0c-4f8d-9582-e9924c39445f, id_history: ["feedback.productive_coaching_writing_feedback.is_tone_appropriate"]
  • evaluator.name: Tone Appropriateness Evaluator — drops the Feedback-Quality suffix so it matches the <X> Evaluator form used across student-facing-text
  • Step id evaluate_is_tone_appropriateevaluate_tone_appropriateness
  • Single LLM step on gpt-5.4-2026-03-05, temperature 1, unchanged

quality_score

The output score field was named after the evaluator (tone_appropriateness_score). Every evaluator in this family had a differently-named field for the same binary judgement, so a consumer had to know which evaluator ran before it could read the result.

Student-facing-text names this after the measured dimensioncomplexity_score, identical across all 8 of its evaluators. This adopts the same convention: quality_score, shared across all 7 feedback evaluators. Values are unchanged (integer, 0/1).

Renamed in output_schema.json (property, required[], description), fixtures.json (expected), and the notebook. Nothing consumes these configs — no TypeScript or Python SDK reads them — so there is no downstream breakage.

Also fixed

The notebook carried two copy-paste references to evals/prompts/purpose, a leftover from the Purpose evaluator it was templated from. Both now point at this directory.

Not applicable

The grade_level string standardisation doesn't apply — this evaluator takes student_text and feedback_text, no grade input. supported_grades: ["8", "9"] is metadata only.

Verification

scripts/check.py all pass.

Migrates the tone-appropriateness feedback evaluator to
evals/feedback/ela-writing/tone-appropriateness/. Adds stable_id/id_history, and
renames the output score field to quality_score so the whole feedback
family shares one name for the measured dimension, mirroring
complexity_score in student-facing-text.
Copilot AI lite review requested due to automatic review settings August 21, 2026 06:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the Tone Appropriateness feedback evaluator into the evals/feedback/ela-writing/ taxonomy and aligns it with the shared config-driven evaluator contract used across related evaluators.

Changes:

  • Introduces canonical prompt assets (system.txt, user.txt) plus a new input_schema.json for the evaluator.
  • Standardizes the binary output field name to quality_score across schema, fixtures, and the example notebook.
  • Updates evaluator identity metadata (evaluator.id, stable_id, id_history) and renames the step id to evaluate_tone_appropriateness.

Reviewed changes

Copilot reviewed 4 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
evals/feedback/ela-writing/tone-appropriateness/user.txt Adds the user prompt template with {student_text} / {feedback_text} placeholders.
evals/feedback/ela-writing/tone-appropriateness/system.txt Adds the system prompt rubric/instructions for tone appropriateness.
evals/feedback/ela-writing/tone-appropriateness/output_schema.json Renames the output score field to quality_score and updates schema description.
evals/feedback/ela-writing/tone-appropriateness/input_schema.json Adds an input JSON schema for student_text and feedback_text.
evals/feedback/ela-writing/tone-appropriateness/fixtures.json Updates fixtures to expect quality_score.
evals/feedback/ela-writing/tone-appropriateness/example_notebook.ipynb Updates notebook narrative + helper function + fixture evaluation to use quality_score and new naming.
evals/feedback/ela-writing/tone-appropriateness/config.json Updates evaluator identifiers/metadata and step id; pins prompt hashes and schema refs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

scripts/checks/eval_config.py hashes raw bytes deliberately -- text-mode
reads translate newlines, so a CRLF prompt file (or a non-UTF-8 default
locale) would make the notebook's computed hash disagree with the
sha256 the config pins, failing the drift assertion on a file CI
considers fine. The notebook now matches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants