I review how AI benchmarks are graded, and I build the measurement tools underneath that work.
Three published reviews, four evaluation tools and datasets. Every number recomputes from published data.
A task-level audit of all 220 GDPval tasks and all 10,453 rubric criteria. Maximum attainable score ranges from 22 to 234 points, a 10.6-fold spread across 95 distinct values, with no normalization anywhere in the published dataset. One rubric line that no grader can mark true or false appears in 138 of 220 tasks. Four tasks can score arbitrarily negative. Every figure recomputes from the published parquet in about two seconds. Read the review.
Does a machine-text detector get worse as the generator gets newer? Everyone assumes so. Across eighteen generators released between 2019 and 2025, it does not: within a fixed corpus the correlation runs the other way. What predicts detectability is instruction tuning, worth 26 to 29 points on average across the n-gram detectors and +17.8 to +38.4 on the individual same-family, same-date model pairs. Read the review.
A literature review of AI-text detection benchmarks, dated by when their text was generated instead of when the paper appeared. The first draft claimed every benchmark was stale. An adversarial check found eight published since September 2025 running on current models, so the thesis was false and the piece was rewritten around the gap that actually exists, between what has been benchmarked and what the field cites.
A quality gate for prose. Eighteen gates, every threshold set by measuring the statistic across 2,602 human-written documents so the false-positive rate is measured at 5%. Fails 46.9% of machine-written documents and 6.3% of human ones. Nine gates don't discriminate on the available corpus; their numbers are in the README.
The evaluation harness underneath it. Structural cadence carries nearly all the signal; the corpus turned out to have a detokenization artifact separating the classes on 73% of documents; and the fitted model transfers below chance to a new domain. The failure is in the README next to the in-domain score.
38 measured features across 5,402 documents on Hugging Face, licensed CC-BY-SA-4.0 to match its sources. It exists so the bluepencil thresholds can be recomputed independently. It also contains a finding that undercuts bluepencil.
Pipeline for a paired human/machine corpus with licence filtering per row and a date cutoff on the human half, so the output is redistributable and the "human" class isn't silently contaminated by post-2023 AI-assisted writing.
Three live business systems: a lead-scoring CRM with auto-routing, milestone invoicing synced to QuickBooks Online, and a legacy data migration that turned a 113-row export into 89 clean typed records while preserving 21 that naive deduplication would have dropped. Screenshots and a reconciled invoice, not mockups.
The generative video system behind a running short-form series. Remotion, Replicate, ElevenLabs, Python, Airtable for production tracking. Sanitized by design: architecture and interfaces public, production tree not.
Also six Python and reportlab applications producing regulatory radon inspection reports for three companies, with per-client threshold logic, in production use.
I write the reviews up at wolfvswhale.github.io, each one linking the code and data behind it.
Four years as a digital court reporter before any of this, producing transcripts that were checked against the recording. I also edit books.
Python, JavaScript, Node. Claude Code and agentic tooling daily. n8n and Airtable for automation. Remotion, Replicate, ElevenLabs for generative media. React Native and EAS for the iOS side.