Add one-step distilled specialist reference - #19
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ba583990ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| student, calibration = calibrate_one_step_student( | ||
| train_one_step_student(dataset, distilled=True), | ||
| _study_holdout_rows()[:4], | ||
| ) |
There was a problem hiding this comment.
The first four rows labeled as held out are used—with their expected labels—to tune both students' confidence and distance thresholds, and then those same rows are included in _evaluate_reference as four of the six held-out examples. Because these thresholds determine abstention, the reported correctness, coverage, abstention, and disagreement measurements are test-set-contaminated; use a distinct calibration split or exclude these rows from the held-out metrics.
Useful? React with 👍 / 👎.
| if ( | ||
| not isinstance(architecture, dict) | ||
| or architecture.get("kind") != "tiny-affine-tanh" | ||
| or architecture.get("learned_forward_passes") != 1 |
There was a problem hiding this comment.
When an artifact is accepted through OneStepStudentModel.load, only kind and learned_forward_passes are checked in the nested architecture object. An artifact can therefore declare input_dimensions: 999, an incompatible hidden/output width, altered context semantics, or extra architecture fields and still load after its outer artifact digest is recomputed; the later model-identity check does not catch this because to_dict() reconstructs and thereby discards the supplied architecture metadata. Validate the complete architecture shape declared by the strict schema before accepting the artifact.
Useful? React with 👍 / 👎.
1 participant
Summary
forge.evidence-relevancemncs-language, compiler, syntax, profile, backend, or stdlib changesValidation
PYTHONPATH=src pytest -q --ignore=tests/test_mncs_reconstruction.py --ignore=tests/test_mncs_training.py— 110 passedPYTHONPATH=src ruff check src/mnel/one_step_specialist.py tests/test_one_step_specialist.py src/mnel/__init__.py— passedok: trueNotes
mncsreconstruction subprocess hanging before collecting tests; those language/compiler tests were intentionally excluded per scope