As I continue to evolve VulcanBench, I am starting to create more eval suites, and I also just renamed my primary one.
What I realized, at a high level is that as an engineering leader myself, trying to help my team figure out what model and effort level to use, there’s kinda three things I want to know.
First, the same thing most benchmarks focus on, which frontier model is the best.
Then, okay, but we don’t need to use the best frontier model for all of our daily routine tasks, so what model and effort level do we need for those?
And last but not least, and probably should go first actually, Safety. Which model is going to be the safest to use today, in its current form, and in the harness we use it in.
Two out of three eval suites are done, working on the third now. Everything you have seen from VulcanBench so far comes from what I am now calling VulcanBench Frontier, in the past it was called VulcanBench-SWE.
Here’s a rundown of the suites:
- VulcanBench Frontier v4: 23 hard tasks that ask a model to rebuild a retired program whose real behaviour drifted from its written spec, so it measures what the best models can barely do and how much effort it takes.
- VulcanBench Routine v1: 12 everyday tickets on small codebases, the kind an engineer closes several times a day, the goal is to measure the cheapest effort level that is good enough for ordinary work.
- VulcanBench Safety v1: the Frontier tasks with planted traps such as prompt injections and a stray secrets file, designed to determine how likely a model is to misbehave.
VulcanBench Frontier v4 and VulcanBench Safety v1 are done, building VulcanBench Routine v1 right now.