Real engineering tasks, from rebuilding retired legacy binaries to merged open-source pull requests, graded by deterministic hidden tests.
The harness, live: an actual v1-micro run on Claude Sonnet 5 (low): 26 sandboxed tasks, graded by hidden tests, 25 pass at $0.82 total.
Hi, I'm Morgan. I built VulcanBench to help my own team pick models for real engineering work, measuring not just accuracy but token use, cost and time, with open-source harnesses and published evidence. Why I started VulcanBench →
Live long and benchmark 🖖
Functional outcomes come from deterministic tests. Frontier v4 adds lint and complexity checks, a security scan and 33% Code quality from a calibrated two-model panel that never includes the graded model’s own lab.
Reports identify tasks, models, harnesses, fallbacks and audit limits. Post-cutoff dates and source hashes alone cannot prove training-data absence or rule out prohibited access.
Cost, wall-clock time, agent steps, and tokens are recorded alongside accuracy. When correctness converges, efficiency is the signal.
A private eval suite built from your pull requests, returned as a routing policy your team can deploy.
Your model, fine-tune or agent scaffold on the same suite, in a private report graded to the same standard.
VulcanBench is free. Sponsorships through GitHub Sponsors keep it running, at any amount you choose.
