https://vulcanbench.com/ Technical reports from VulcanBench: frontier models measured on real engineering work, graded by deterministic hidden tests, with cost, tokens, and time as first-class results. en https://vulcanbench.com/benchmarks/15-grok46-cursor-harness.html https://vulcanbench.com/benchmarks/15-grok46-cursor-harness.html Fri, 14 Aug 2026 12:00:00 +0000 VulcanBench's first model x harness study: the same Grok 4.6, the same 23 tasks, two delivery systems. Inside Cursor it scores up to 21 points higher at the same nominal effort, runs 2-5x faster, and its effort curve inverts from peak-and-collapse to monotone rise. https://vulcanbench.com/benchmarks/14-grok-46-effort.html https://vulcanbench.com/benchmarks/14-grok-46-effort.html Wed, 13 Aug 2026 12:00:00 +0000 Grok 4.6 peaks mid-knob on VulcanBench v3: 87.0% pass@1 at medium effort, 73.9% at its shipped default (high). Failures shift from wrong answers to unfinished runs as effort rises; the new xhigh level recovers less than half the drop. https://vulcanbench.com/benchmarks/13-deepseek-v4-pro.html https://vulcanbench.com/benchmarks/13-deepseek-v4-pro.html Sun, 09 Aug 2026 12:00:00 +0000 DeepSeek V4 Pro ties V4-Flash at 87.0% pass@1 on VulcanBench v3. Pro uses 55% fewer tokens and runs 38% faster, but its higher token price makes the sweep 29% more expensive. https://vulcanbench.com/benchmarks/12-qwen38-max.html https://vulcanbench.com/benchmarks/12-qwen38-max.html Tue, 04 Aug 2026 12:00:00 +0000 Qwen3.8-Max debuts on VulcanBench v3 with an effort knob that runs backwards: 81.2% at low, 71.0% at medium, 55.1% at its shipped default. Every failure at medium and xhigh is an unfinished run, not a wrong answer. https://vulcanbench.com/benchmarks/11-voice-tax.html https://vulcanbench.com/benchmarks/11-voice-tax.html Fri, 31 Jul 2026 12:00:00 +0000 First results from the VulcanBench Voice Eval Suite: the same 200 questions typed vs spoken. Grok Voice Think Fast 2.0 pays a +3.3 pp voice tax (99.0% text, 95.7% audio); GPT Realtime +4.0 pp. Both lose ~10 pp on spoken arithmetic. https://vulcanbench.com/benchmarks/09-contamination.html https://vulcanbench.com/benchmarks/09-contamination.html Wed, 29 Jul 2026 12:00:00 +0000 A controlled A/B on training-data contamination: 13 tasks Claude Opus 5 could have memorised vs 13 it cannot have seen. Result: 11/13 vs 12/13, one genuine miss each — no detectable effect. https://vulcanbench.com/benchmarks/10-opus5-effort.html https://vulcanbench.com/benchmarks/10-opus5-effort.html Tue, 28 Jul 2026 12:00:00 +0000 First measurement of Claude Opus 5 on the full VulcanBench v3 suite: its cheapest setting wins at 20/23 and $0.70 per solved task; score falls at every step up the effort knob while cost triples. https://vulcanbench.com/benchmarks/08-kimi-k3.html https://vulcanbench.com/benchmarks/08-kimi-k3.html Sun, 19 Jul 2026 12:00:00 +0000 Kimi K3 joins the VulcanBench v3 field: Grok 4.5 leads all eleven configurations at 91.3% with the lowest cost per solved task; K3 debuts at 74%, rising to 87% under an extended-budget ablation. https://vulcanbench.com/benchmarks/07-grok-fable-sol.html https://vulcanbench.com/benchmarks/07-grok-fable-sol.html Sun, 12 Jul 2026 12:00:00 +0000 Grok 4.5 leads at 91% from medium effort onward on the v3 suite; only GPT-5.6 Sol rewards the effort knob; two frontier-hard tasks go 0-for-27. https://vulcanbench.com/benchmarks/06-reliability.html https://vulcanbench.com/benchmarks/06-reliability.html Wed, 08 Jul 2026 12:00:00 +0000 The first reliability study on the suite: five runs per task. On pass^5 the cheapest model wins, the frontier model is the flaky one, and the frontier-hard tier holds at 0 solves in 30 attempts. https://vulcanbench.com/benchmarks/05-effort-matrix.html https://vulcanbench.com/benchmarks/05-effort-matrix.html Sat, 04 Jul 2026 12:00:00 +0000 Two models, two effort levels, on a suite with five frontier-hard tasks. Binary pass@1 is a four-way tie; effort helps Opus and backfires for Fable. https://vulcanbench.com/benchmarks/04-fable-opus-gpt55.html https://vulcanbench.com/benchmarks/04-fable-opus-gpt55.html Thu, 02 Jul 2026 12:00:00 +0000 Four frontier configurations on 15 real engineering evals. All score 14/15 but each misses a different task; cost spans 4x. https://vulcanbench.com/benchmarks/03-fable-sonnet-glm.html https://vulcanbench.com/benchmarks/03-fable-sonnet-glm.html Wed, 01 Jul 2026 12:00:00 +0000 Accuracy converges across three models while efficiency spans an order of magnitude. Fable 5 at low effort matches Sonnet 5 at high effort for a quarter of the cost. https://vulcanbench.com/benchmarks/01-cost-pilot.html https://vulcanbench.com/benchmarks/01-cost-pilot.html Wed, 01 Jul 2026 12:00:00 +0000 The pilot run that established cost as the live axis: both frontier models solve every real, decontaminated bug, and the gap is entirely economic. https://vulcanbench.com/benchmarks/02-effort-sweep.html https://vulcanbench.com/benchmarks/02-effort-sweep.html Tue, 30 Jun 2026 12:00:00 +0000 Does more reasoning effort pay off? A 936-run sweep of Sonnet 5 and Opus 4.8 across low, medium, and high effort.