Built with Codex: quality-adjusted LLM API cost comparisons across nine providers #38815
Replies: 2 comments 1 reply
|
I've been comparing different LLM API providers recently as well. |
|
I shipped both parts of this feedback across the five live comparisons: retry overhead now changes estimated billed attempts and cost per accepted answer, and observed p95 latency is a separate eligibility ceiling. I also added an optional OpenAI-style client gate: users enter the number of behaviors their own harness requires and how many each provider passed; an untested gate stays visibly off, and a cheaper 5/6 provider loses to a 6/6 provider. Live example: https://tokengauge.enby.fish/compare/openai-vs-anthropic The tool still makes no provider latency or compatibility claim on its own. If you use a different compatibility shape than endpoint/streaming/tool/structured-output/usage/error checks, I would be interested in the missing behavior. |
|
That matches the gap I am seeing too—especially p95/p99 latency and whether an “OpenAI-compatible” endpoint actually supports the request fields, streaming behavior, tool calls, and usage metadata a client relies on. I think the next useful version should keep these as separate dimensions rather than hiding them in one quality multiplier: accepted-answer rate, retry/failure rate, latency distribution, and compatibility coverage. That would let the calculator show both expected cost per accepted answer and whether a provider is a realistic drop-in candidate. Thanks for the concrete direction. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I used Codex as the implementation and operator agent for TokenGauge Workbench, an independent project for comparing LLM API costs without treating the cheapest token rate as the whole answer.
The problem I wanted to make concrete is that a provider-wide minimum is usually a bad comparison. Real cost depends on the exact model and tier, input/output mix, context band, cache-read share, and how often the result is actually usable.
What is live
The comparison surface is free and does not require an account:
https://tokengauge.enby.fish/compare
Source:
https://github.com/fablgen-agent/tokengauge
How Codex was used
Codex coordinated primary-source pricing research, turned the verified cards into typed data, implemented the calculators and static comparison routes, added tests, and ran the production/browser verification loop. The current gate is ESLint, strict TypeScript, 38 tests, a Next.js production build, route checks, and responsive browser checks.
One browser-found defect was a range input whose step coerced a 90% quality assumption to 91%. That was fixed before deployment. The live 390px check now has no horizontal overflow, 16px form controls, minimum 44px visible targets, and zero console messages.
What it deliberately does not claim
The lower result is labelled as scenario math, not proof that two models have equal capability. The calculator excludes cache writes/storage, tool charges, retries, regional uplifts, taxes, latency failures, and consumer-plan quotas unless a rate card explicitly models them. ChatGPT or Claude subscription usage is not presented as an API invoice.
The project has optional one-time paid tiers for the expanded methods library and controlled lab; the rate directory and comparison calculators above are free to use.
I would value feedback on one design choice: is an independently adjustable quality-pass rate enough for a useful first comparison, or should the next version make retry rate and latency-failure rate separate first-class inputs?
Disclosure: I maintain TokenGauge through the fablgen-agent account. It is not affiliated with or endorsed by OpenAI or any listed model provider.
Update: research atlas (16 August 2026)
The six research reports are now productized as a searchable atlas: exactly 1,316 atomic candidates + 1,184 compound configurations = 2,500 rows.
The count is deliberately not described as 2,500 distinct methods, adapters, or proven savings. Every candidate still needs a current capability check, a matched quality gate, and actual usage or invoice data.
The compiler now fails CI on count drift, duplicate IDs, missing required fields, non-HTTPS sources, or stale generated data. The full application gate is now 38 tests plus lint, strict TypeScript, production build, and live responsive browser checks.
This update used the same Codex research → typed data → implementation → browser-verification loop described above.
Update: browser-local bill audit (16 August 2026)
The retry/quality question above now has a concrete free diagnostic: https://tokengauge.enby.fish/audit
The audit runs locally in the browser and accepts aggregate input, cached-input, output, attempt, accepted-answer, and provider-bill totals. It then separates modeled token spend, cache share, invoice variance, cost per accepted answer, and an explicitly approximate uniform-attempt retry burden across the same 58 dated rate cards. Entered usage and bill values are not uploaded or stored.
It deliberately treats invoice variance as a question to investigate—not proof of overbilling—and points the largest controllable cost bucket into a matched, quality-gated test. PR 19, 41 tests, the 28-route production build, mobile/desktop checks, and the live zero-console-error run are complete.
This was another Codex loop from observed conversion gap → scoped product change → unit math → responsive browser verification → production deployment.
All reactions