Most teams route every task to one frontier model at maximum effort and hope the bill is worth it. A stack benchmark replaces that guess with a measurement, and the measurement starts with a custom eval suite modeled on the work your team actually does. Mostly Python in mid-sized codebases? The suite is Python in mid-sized codebases. TypeScript across a large codebase? Same. The suite runs across the model-and-effort matrix in the sandboxed harness that grades every public report, and distills into a routing plan: which model, at which reasoning effort, for which class of task.
Model routing is sending each class of engineering task to the model and reasoning-effort level that handles it best, instead of sending everything to one frontier model at maximum effort. Done well, it holds accuracy while cutting cost and latency, because most day-to-day tasks don't need the most expensive configuration. A routing decision is only as good as the measurement behind it: that measurement is what this engagement produces.
The public reports keep converging on the same finding from different directions. Frontier accuracy has largely converged while cost still spreads by an order of magnitude (Report 03). Reasoning effort pays for one model and backfires for another on the identical tasks (Report 05). And when tasks are run repeatedly, the cheapest model can be the most dependable one (Report 06).
Put those together and "which model is best" stops being a useful question. The useful question is which model, at which effort, for which class of task, and a leaderboard built on someone else's suite cannot answer it, because your stack and your task mix aren't generic. The answer has to be measured on your work.
Task class by task class: which model, at which effort, with escalation rules for when the cheap route misses. Not a leaderboard, a decision your platform team can implement.
Your results in the same journal format as the public reports: per-task tables, costs, traces, and replay transcripts. Private by default, yours to share internally.
Your task volume priced through the routing table: projected spend against your current setup, break-even points included, re-runnable when the next model ships.
We map what your team actually ships: languages, frameworks and versions, the size and shape of the codebases you work in, and the task mix, from bug fixes and feature work to refactors, test writing, and migrations, plus volume and latency constraints. This works per team, too: a platform team on Go and a product team on TypeScript don't have to share a suite, or a routing table.
Evals are built from your own merged pull requests where possible, or curated equivalents matched to your work profile where not: same language, same scale of codebase, same kind of change. Every task gets hidden tests validated to fail before the fix and pass after it, the same fail→pass standard as the public suite. Your code stays private and never becomes public eval material.
Models × effort levels × repeats, in the network-isolated Docker sandbox with deterministic grading, no judges. Per task class we record pass@1, pass^k reliability, tokens, cost, and wall-clock: the same five-way view the public reports use.
The matrix collapses into a routing table with escalation rules and the cost model behind it, delivered with the full report and a walkthrough with your team. When a new model ships, the suite already exists: re-running the matrix is the cheap part.
| Task class | Route to | Escalate on miss | Why |
|---|---|---|---|
| Routine bug fixes | Mid-tier · low | Frontier · low | Accuracy parity with frontier at a fraction of the cost |
| Test writing | Small · low | Mid-tier · low | Highest-volume class; pass^k is the bar and the small model clears it |
| Feature work | Mid-tier · medium | Frontier · medium | Effort pays here, but only up to medium |
| Refactors & migrations | Frontier · low | Frontier · high | Breadth wins; extra effort didn't move the score |
| Novel, cross-cutting work | Frontier · high | Human review | The only route that moves these tasks at all |
Illustrative, not a recommendation: the entire point of a stack benchmark is that your table comes out of your measurement, not ours. On your stack the rows, the routes, and the escalation rules can all land differently.
A stack benchmark is your code, your suite, your numbers: tasks built from your repositories never become public evals and never appear on the leaderboard. And no model vendor pays for placement in a routing recommendation. The routing table is bought by you, graded by hidden tests, and owes nothing to anyone whose model appears in it.
Pricing scales with the suite and the matrix: how many task classes, how many models, how many effort levels, how many repeats. A focused engagement, one work profile and a handful of task classes across a four-model matrix, starts in the mid four figures; broader matrices and bigger suites are more, and per-team suites add their suite build, not a whole new engagement. The compute is real and metered, and you see it, the same cost transparency as the public reports. Re-runs on an existing suite cost a fraction of the initial build.
A couple of sentences is enough to start: what your team builds, the languages and frameworks involved, and roughly what your task mix looks like. I'll come back with a proposed suite shape, a matrix, and a number. Building a model rather than choosing one? That's benchmark your model.
