Hi, I’m Morgan, and I am the mad scientist behind VulcanBench. Before I go any further I want to make it clear, I am not an expert, and I don’t want anyone to mistake me for one. I kinda fell into benchmarking, and into the creation of VulcanBench out of my own needs, and really more my team’s needs, so let me explain.
Annnnnd first, it probably makes sense to give a super short intro to me. I was born in Berkeley, California, and as a kid was kinda obsessed with space and science. I took science classes at the Lawrence Hall of Science, and I think that really sparked my interest in science at a young age.
When I was in Junior High, I started a web design company that ended up building some of the first websites for real estate co’s in the bay area, and yes, this was back in the mid-90’s, I’m old.
During High School I worked for two different startups, Xaos Tools and Xinet, and yes, I guess I had a thing for startups that had names that started with an X. After High School I went to Carnegie Mellon, did my undergrad in Computer Engineering with a minor in Computer Science, then stayed on and did my Masters in Computer Engineering.
It was during my Masters that I became kinda obsessed with optimization. At that time, I was working on optimizing C code using assembly, so it could run faster on small chips like the Intel XScale processor.
Okay, I don’t want to bore you with my whole life story, so I’ll fast-forward to today. I am the Cofounder and CTO of Bold Metrics, we’re an AI company that builds custom models for predicting body measurements and garment sizing, and work with some of the biggest, most awesome apparel brands in the world. It’s really neat stuff, and I have an amazing engineering and data science team, that is all in on agentic coding workflows.
The problem we ran into earlier this year (let’s call it March of 2026) is that it was becoming incredibly difficult to both figure out what models were the best for each team, the programming language they were using, codebase size, etc. and what effort level to use with each model.
We were on a path that was going to lead to a bill of over $300,000 for the year, and that was crazy, I knew there had to be a better path. So I started looking at how the team was using models, and they did, what I think most engineering teams do, pick a provider like Anthropic or Open AI, and then use their best model, at High effort (or higher) for everything.
So I thought, what if I build some evals based on our codebase, and the actual work our team is doing, and test different models, at different effort levels. What I learned kinda blew my mind. For a lot of the daily routine tasks my team was doing, they didn’t need high effort, or even medium effort in many cases, and they also didn’t need the latest, most powerful frontier model, to get the exact same accuracy.
As I went deeper, I also realized that for individual PRs, most had components that were mostly easy or medium difficulty, with some truly hard stuff, but often less than 20% of the work in the PR really needed a model and effort level for solving something really challenging.
So I started to noodle on the idea of a multi-model execution layer, which at a high-level means working with a planning agent to break a coding task into easy, medium, and hard components, and then sending each of those components to a different model, at a different effort level. I wrote more about this on Medium, and decided to call this general concept, a difficulty aware multi-model execution layer.
It was a bit of an aha moment. So then I went to go look at a bunch of different benchmarks to help figure out what models to send to each component of the execution layer, and hit a wall. Most of the benchmarks I looked at tested models only at Max effort, the signal I was looking for, around what effort level me and my team really needed, just wasn’t available.
So I whipped up a little benchmarking tool and started benchmarking across effort levels myself. This finally gave me the data I needed, and after benchmarking model after model, across effort levels, we got what we needed, and our bill went from $300,000/year to under $30,000/year. It was wild, and I started talking to a few other friends about it and they suggested I start posting my benchmarks on X, they thought there were probably other eng teams out there that could benefit from the same data.
I came up with a name, VulcanBench, which as a lifelong Star Trek fan just made sense to me right away, and I started sharing my benchmarks on X. The response and interest level in what I was doing really surprised me, it turned out I guess I wasn’t the only person that was hungry for benchmark data on models across effort levels, not just on Max.
Soon companies started reaching out asking me to do what I had done for our team, benchmark their exact stack by building custom evals based on their codebase so they could have customized model routing recommendations, and most importantly, effort level data that they could use to really cut token use and cost.
And ideas for new evals keep popping into my head constantly. I’m working on a Python-only eval suite, a TypeScript-only eval suite, Cybersecurity eval suite, and more language and domain specific eval suites. The more I learn through my benchmarks, the more I realize that there’s a real problem here, generalized benchmarks, taken only at Max effort, and without much data on if the coding tasks in the eval suite actually represent the kind of work you or your team does, don’t provide enough signal to make a decision.
When people can’t make a decision, they tend to take the easy route, which means picking one model provider, and just using their latest model at high effort (or higher), but that’s an expensive and slow path. I think there’s a better way, and I’m excited to continue to explore this new frontier at VulcanBench.
Phew, and this turned into one heck of a long blog post, guess I had a lot to say! As you can probably tell, I’m incredibly passionate about what I’m doing here, and I honestly have no idea where it’s all going, I just know, I can’t stop thinking about evals and benchmarks.
Thanks for reading, and live long and benchmark 🖖