Hi I'm Morgan, and I run the open source benchmarking tool and lab that is VulcanBench. My motivation behind building this developed organically through my own need, as a Founder and CTO, to help my team leverage the best models for the work that we do.

The core problem I ran into with agentic coding benchmarks is that a good chunk of the tasks in the evals are things like puzzles and math problems, that don't represent the kind of work engineering teams do every day. At the same time, benchmarks that I like, don't show how models compared across token use and time, which is critical for engineering leaders to know.

So, I started testing models myself, on real engineering tasks, and looking at not just task completion scores, but also at token use, cost, and time to complete tasks. If I am comparing two models, and one is 5% better than the other, but that model also has an overthinking problem and uses 5x the number of tokens and takes much longer to solve problems, that would be important to know, right?

I also learned through my own testing that two models that have different scores on a popular benchmark, might actually produce the same accuracy on regular every day engineering tasks.

And what started as a bunch of Python scripts I was piecing together to test different models for my team, turned into fully open source benchmarking software, and open, transparent evals.

Every single benchmark you see on here you can run yourself. You can see every eval and all the source code for VulcanBench itself. When I publish a benchmark, it will never just look at model accuracy, while that is critically important, it will also look at how many tokens the models used to complete the tasks, what the cost was, and how long it took.

My goal is simple: let's build better benchmarks that represent the real work engineering teams do, and that go beyond scores, showing things like token use, cost, and time to complete a task.

Live long and benchmark 🖖

Have your own model, fine-tune, or agent scaffold measured on the same suite, in a private report graded to the same standard as everything published here.

Sponsorships run entirely through GitHub Sponsors, at whatever amount and cadence you choose. Sponsors are named on GitHub as a thank you.