VulcanBench VulcanBench
The published record

Benchmarks

Every measured run, newest first. Each links to its model card, findings, and report.

Report06
Opus 4.8 vs Haiku 4.5
Five runs per task: the cheapest model wins on dependability; pass@1 hides coin flips.
2026-07-07 · 15 evals · 135 runs · reliability study
→
Report05
Fable 5 vs Opus 4.8
A four-way pass@1 tie; effort helps Opus and backfires for Fable.
2026-07-04 · 15 evals · 60 runs · frontier-hard tier
→
Report04
Fable 5 vs Opus 4.8 vs GPT-5.5
Four configurations, fifteen evals, each misses a different task.
2026-07-02 · 15 evals · 60 runs · corrected costs
→
Report03
Fable 5 vs Sonnet 5 vs GLM-5.2
Accuracy converges; efficiency spans an order of magnitude.
2026-07-01 · 10 tasks · 30 runs
→
Report02
Sonnet 5 vs Opus 4.8
Does more thinking effort pay off? Cost versus correctness across low, medium, high.
2026-06-30 · 52 tasks · 936 runs
→
Report01
Sonnet 5 vs Opus 4.8
The pilot that established cost as the live axis once correctness saturates.
2026-07-01 · 4 tasks · 24 runs
→