Before diving into the methodology I have developed with VulcanBench, I think it probably makes sense to introduce myself. Hi, I’m Morgan, and I’m the Cofounder and CTO of Bold Metrics, an AI startup that helps some of the world’s leading apparel brands with fit and sizing. I lead an engineering team, and over the last 18 months we’ve gone head-first into agentic coding workflows.
What has been challenging for me as an engineering leader is knowing what benchmarks I can use to truly understand how a model will perform when my team uses it for the regular coding tasks we do every day. As I did a deeper dive into some of the current benchmarks out there, I found many of the evals had too many tasks that were things like puzzles or math problems, that didn’t reflect the kind of things my team would be doing with these models.
Looking on social media, I found a lot of people testing models by seeing how well a new model can build a working version of Flappy Bird or create an image of a pelican riding a bike, and well, yeah, that’s not as helpful for engineering teams like mine shipping production code that goes out to hundreds of millions of active users.
So I started benchmarking models myself, at first just testing our own engineering tasks across different models. Then I started to augment the effort level and had a bit of an aha moment. I realized that while so many model providers default to High effort, with the new class of models we now have, Low and Medium effort often produces the same result, faster, and cheaper.
As I started to get deeper into model benchmarking for us, I realized that all of the Python scripts I had created were getting messy, I needed a more clean and organized way to run my benchmarks. And well, I’m a big Star Trek fan, that’s one of the reasons why we’re called Bold Metrics, i.e. to boldly go where no one has gone before ;) I knew I wanted the name of whatever tool I built to have the word bench at the end, and I also knew I wanted a tool that at its core was about honest and transparent benchmarks, and I immediately thought, Vulcans!
Yup, and that’s how VulcanBench was formed.
My general thinking is that any benchmark that just runs one pass, and doesn’t take into account things like token use, cost, and time, is leaving out some really important pieces to the puzzle. If I am comparing two models, and one is 5% better than the other, but that model also has an overthinking problem and uses 5x the number of tokens and takes much longer to solve problems, that would be important to know, right?
And that’s my mission with VulcanBench. Open source in every way, every single eval, and the source code for the benchmarking tool itself, all available for everyone to see, critique, and help improve.
As you can probably now understand from the journey I’ve been on to get here, I’m not some kind of benchmarking expert, and yes, there are a lot of people much smarter than me working on really interesting benchmarks. So if you find an issue with one of my benchmarks, tell me, or better yet, put in a PR and help fix it.
I am dedicated to helping my engineering team use the best models out there, and that means I need to run as many evals as I can, and keep updating them every week to get even better. I hope I am able to help other teams like us that are looking for benchmarks that reflect the kind of work we do every day.
Most tasks are built from a real merged pull request in a production open-source project (flask, sqlglot, aiohttp, urllib3, poetry, redis, click, chi, and more). Each task ships a starting repository, a set of hidden tests never shown to the agent, declarative fail_to_pass and pass_to_pass test commands, and a gold-patch reference solution.
The repository is checked out at the pull request's base commit, the exact state a developer would face before the fix existed.
The agent receives a terse issue describing the symptom plus the full working tree, and the standard toolset: list, read, edit, search, and run commands.
The agent works the problem until it believes it is done, within a fixed step budget and a wall-clock limit. Its diff is captured as the answer.
The patch is scored by the project's own hidden tests, pass or fail, with resource consumption recorded alongside.
A benchmark built from public code risks measuring memory, not reasoning. So provenance is labeled and checked, never assumed: every task declares a source, either hand-authored or oss, plus an explicit decontaminated boolean, and the validator enforces the claim.
Open-source tasks are built from pull requests merged after the evaluated models' training cutoffs, so the solution post-dates what any model could have seen. Hand-authored tasks are original scenarios written for VulcanBench now, not derived from any public repository, so they are decontaminated by construction. Where a task is sourced verbatim from an existing repository whose fix predates model cutoffs, it is honestly labeled decontaminated: false, and every result scored against it is flagged in the integrity section of the report.
By default a task's correctness comes from hidden tests that are deterministic and exact. The fail_to_pass tests are validated to fail on the base commit and pass on the merged fix; the pass_to_pass tests guard against regressions. If any pass-to-pass test breaks, the functional score is gated to zero regardless of how many new tests pass: a fix that breaks working behavior is not a fix.
Validation proves each task is real before it is ever used: the gold patch must solve it (functional score of exactly 1.0), the fail-to-pass tests must genuinely fail before the fix, and scoring must be identical over repeated runs. There is no LLM judge in the default path, no rubric, and no partial credit for plausible-looking diffs.
A task may opt into an agentic grader when the point is to test terse, realistic prompts where the specification lives in the grader rather than the prompt. It judges the diff against hidden acceptance criteria, graded by a strong, independent model, never the model under test, by majority vote over several calls, and every such grader is itself evaluated for accuracy, self-consistency, and false-pass rate before it ships. Agentic grading is non-deterministic, so it is opt-in and never the default.
Every run produces five sub-scores on a 0–1 scale, combined into a weighted total. Functional correctness dominates, but a diff that passes tests can still be sloppy, insecure, or wasteful, and those are measured too.
Functional is pass/fail on the hidden tests. Quality runs real linters and complexity tools on the changed files. Security runs static analysis for introduced vulnerabilities. Efficiency maps token and step usage to a frugality score. Human-like is the only judged metric, an LLM ensemble, and it carries the least weight. The report tables on this site are graded functional-only, with judges disabled, so that no model's opinion touches the headline numbers.
Real runs execute in a Docker sandbox: a non-root container with the network switched off and CPU and memory capped. Tool calls and the functional grading both run inside it, so the agent cannot reach the internet to fetch a known answer and cannot escape the container. Tasks with heavy dependency stacks use per-task images with the upstream package's dependencies preinstalled, so a run measures the model's engineering, not its ability to fight a package manager. Network isolation is the default; opting out is deliberate and recorded.
Reasoning effort, the model's thinking-token budget, is a normalized control set to low, medium, high, or extra-high. It maps to each provider's native field (the Responses API effort setting for OpenAI, the Messages API effort setting for Anthropic). Holding a suite fixed and sweeping the dial turns a saturated benchmark into a discriminating one for free: the low-effort dropoffs below the high-effort ceiling are the signal.
The labels are not calibrated across vendors, and that matters. Effort is each provider's own scale: comparing two models at the same label compares each at its own setting, not a calibrated equivalence. So token counts always sit beside effort labels, and the per-task step budget is held identical across cells, so only the thinking budget moves.
Every run records cost, wall-clock time, agent steps, and token usage. Cost is computed from each provider's published list prices and, where a provider applies an automatic prompt-cache discount, that discount is folded into the reported total, since a naive token count can overstate a run's true spend several-fold. The headline efficiency metric is cost per solved task: total spend divided by tasks passed, which fairly charges a model for both its failures and its verbosity.
This is the axis that keeps mattering after accuracy converges. Across the published reports, models tying on correctness have differed by 3× to 4× in cost and up to an order of magnitude in tokens, differences driven by how much each agent thrashes, especially when navigating large repositories.
When a model's safety classifier declines a benign, security-adjacent task, say a parser fix flagged as cyber content, the run is scored as a failure, with no fallback model configured. An engineering team whose model refuses to touch parser code experiences that as a failure, not a neutral abstention. It is invisible on conventional accuracy benchmarks and highly visible in practice, so VulcanBench records it as a first-class outcome.
Every task, hidden test, gold patch, per-task Docker image, and the full grading harness are open source. Each run captures every step as a structured trace of model calls, tool calls, diffs, test results, and token usage, then writes a self-contained HTML transcript you can open in any browser, plus the exact command to replay it. Reproducing a published two-model run costs about ten dollars at list prices.
