feat(evals): add Tone Appropriateness onto the ela-writing taxonomy - #183
Open
adnanrhussain wants to merge 3 commits into
Open
feat(evals): add Tone Appropriateness onto the ela-writing taxonomy#183adnanrhussain wants to merge 3 commits into
adnanrhussain wants to merge 3 commits into
Conversation
Migrates the tone-appropriateness feedback evaluator to evals/feedback/ela-writing/tone-appropriateness/. Adds stable_id/id_history, and renames the output score field to quality_score so the whole feedback family shares one name for the measured dimension, mirroring complexity_score in student-facing-text.
There was a problem hiding this comment.
Pull request overview
Adds the Tone Appropriateness feedback evaluator into the evals/feedback/ela-writing/ taxonomy and aligns it with the shared config-driven evaluator contract used across related evaluators.
Changes:
- Introduces canonical prompt assets (
system.txt,user.txt) plus a newinput_schema.jsonfor the evaluator. - Standardizes the binary output field name to
quality_scoreacross schema, fixtures, and the example notebook. - Updates evaluator identity metadata (
evaluator.id,stable_id,id_history) and renames the step id toevaluate_tone_appropriateness.
Reviewed changes
Copilot reviewed 4 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| evals/feedback/ela-writing/tone-appropriateness/user.txt | Adds the user prompt template with {student_text} / {feedback_text} placeholders. |
| evals/feedback/ela-writing/tone-appropriateness/system.txt | Adds the system prompt rubric/instructions for tone appropriateness. |
| evals/feedback/ela-writing/tone-appropriateness/output_schema.json | Renames the output score field to quality_score and updates schema description. |
| evals/feedback/ela-writing/tone-appropriateness/input_schema.json | Adds an input JSON schema for student_text and feedback_text. |
| evals/feedback/ela-writing/tone-appropriateness/fixtures.json | Updates fixtures to expect quality_score. |
| evals/feedback/ela-writing/tone-appropriateness/example_notebook.ipynb | Updates notebook narrative + helper function + fixture evaluation to use quality_score and new naming. |
| evals/feedback/ela-writing/tone-appropriateness/config.json | Updates evaluator identifiers/metadata and step id; pins prompt hashes and schema refs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
scripts/checks/eval_config.py hashes raw bytes deliberately -- text-mode reads translate newlines, so a CRLF prompt file (or a non-UTF-8 default locale) would make the notebook's computed hash disagree with the sha256 the config pins, failing the drift assertion on a file CI considers fine. The notebook now matches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates Tone Appropriateness onto the
evals/feedback/ela-writing/taxonomy as Tone Appropriateness, atevals/feedback/ela-writing/tone-appropriateness/. Same pattern as the student-facing-text family (#159, #161, #163, #173–#177). The evaluator already had a complete contract; this moves it, renames it, and closes the gaps that family standardised on.Evaluator
evaluator.id:feedback.ela_writing.tone_appropriatenessstable_id:e6d674bd-2c0c-4f8d-9582-e9924c39445f,id_history:["feedback.productive_coaching_writing_feedback.is_tone_appropriate"]evaluator.name:Tone Appropriateness Evaluator— drops theFeedback-Qualitysuffix so it matches the<X> Evaluatorform used across student-facing-textevaluate_is_tone_appropriate→evaluate_tone_appropriatenessgpt-5.4-2026-03-05, temperature 1, unchangedquality_scoreThe output score field was named after the evaluator (
tone_appropriateness_score). Every evaluator in this family had a differently-named field for the same binary judgement, so a consumer had to know which evaluator ran before it could read the result.Student-facing-text names this after the measured dimension —
complexity_score, identical across all 8 of its evaluators. This adopts the same convention:quality_score, shared across all 7 feedback evaluators. Values are unchanged (integer,0/1).Renamed in
output_schema.json(property,required[], description),fixtures.json(expected), and the notebook. Nothing consumes these configs — no TypeScript or Python SDK reads them — so there is no downstream breakage.Also fixed
The notebook carried two copy-paste references to
evals/prompts/purpose, a leftover from the Purpose evaluator it was templated from. Both now point at this directory.Not applicable
The
grade_levelstring standardisation doesn't apply — this evaluator takesstudent_textandfeedback_text, no grade input.supported_grades: ["8", "9"]is metadata only.Verification
scripts/check.pyall pass.