Real engineering tasks, from rebuilding retired legacy binaries to merged open-source pull requests, graded by deterministic hidden tests.
The harness, live: an actual v1-micro run on Claude Sonnet 5 (low): 26 sandboxed tasks, graded by hidden tests, 25 pass at $0.82 total.
Hi, I'm Morgan. I built VulcanBench to help my own team pick models for real engineering work, measuring not just accuracy but token use, cost and time, with open-source harnesses and published evidence. Why I started VulcanBench →
Live long and benchmark 🖖
Functional outcomes come from deterministic tests. Frontier v4 adds lint and complexity checks, a security scan and 33% Code quality from a calibrated two-model panel that never includes the graded model’s own lab.
Reports identify tasks, models, harnesses, fallbacks and audit limits. Post-cutoff dates and source hashes alone cannot prove training-data absence or rule out prohibited access.
Cost, wall-clock time, agent steps, and tokens are recorded alongside accuracy. When correctness converges, efficiency is the signal.
Point VulcanBench at your own repository, turn merged pull requests into tasks, and benchmark models on them.
Your model, fine-tune or agent scaffold on the same suite, in a private report graded to the same standard.
VulcanBench is free. Sponsorships through GitHub Sponsors keep it running, at any amount you choose.
