VulcanBench Frontier v4 · current suite

Leaderboard

Updated 2026-11-19 · 23 tasks · 19 models · 91 model×effort columns · 2062 runs · v4 suite
Notes on the board and on individual columns

19 models, 91 model×effort columns, 2,062 runs. Combined score is 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under one frozen protocol (v3.4 to v3.7, v3.15 to v3.18, v3.20, v3.23, v3.24, v3.26, v3.27 to v3.29, v3.30.2, v3.30.3 and v3.31 apply the same rubric, controls and gates to each population; v3.20 seats GPT-6.1 Sol in place of Grok 4.6 to judge Grok 4.7, see ◊; v3.23 and v3.24 judge Grok 4.6 through a newer Cursor CLI, see †† and ‡‡; v3.26 to v3.26.5 seat Claude Opus 5.5 and GPT-6.1 Sol to judge Muse Wonder Full, see §§; v3.27 and v3.28 seat GPT-6.1 Sol in place of Grok 4.6 to judge DeepSeek V4.1 Flash and Reflection Beam, see ¶¶ and ◊◊; v3.30.2 and v3.30.3 seat GPT-6.1 Sol after Grok 4.6 failed its exam, to judge Claude Fable 5.2, see ‖‖). The shaded top-left corner of the chart is the quadrant to focus on: levels that score 80 or more for less than the median cost (or tokens, or minutes) of the plotted levels. Completion tokens are the model's own output per task, reasoning included. $/task is API-equivalent at list rates from the solver receipts; every model here ran on a subscription except DeepSeek V4.1 Flash, which ran on a metered DeepSeek API key and is shown at DeepSeek’s off-peak list price, see ¶¶. Grok 4.7 and Reflection Beam have no list price, so their $/task is unavailable and they appear on the Tokens and Minutes views only. Muse Wonder Full’s $/task is an estimate at the Muse Wonder 1.0 list price, see §§.

  • † Fable 5.1 runs include 11 disclosed Opus 4.8 fallbacks across the sweep; they stay in the population.
  • ‡ GPT-5.6 Terra at max includes paddockcore, run on September 17 on a second ChatGPT account after the first hit its quota window and judged under the v3.6.1 top-up with the same judges and calibration.
  • § GPT-5.6 Sol at max is judged on 22 of 23 tasks: on codeccore, Grok 4.6's intent probe quoted an excerpt absent from the code on both attempts, so the v3.7 protocol publishes no Code quality score for that run; the run passed its tests and is priced.
  • ¶ GPT-6 Luna at extra-high and max is judged on 21 and 19 of 23 tasks: the other 2 and 4 runs hit the flat 3-hour task bound while still working, have no finished code to judge, count as failed tasks and are unpriced ($/task covers the finished runs; Min/task covers all 23). Counting each timeout as a combined score of 0 over all 23 runs gives 71.94 at extra-high and 67.26 at max; its best tag and effort suggestions use these figures. Ran on Codex CLI 0.155.0, the first release that serves GPT-6 Luna on a ChatGPT plan; the extra-high pacecore run was retried after an 88-minute Codex client stall, and the retry counts.
  • ‖ GPT-6 Sol at medium is judged on 22 of 23 tasks: on codeccore, Grok 4.6's intent probe gave no valid answer (one malformed attempt, one quoting code absent from the run), so the v3.17 protocol publishes no Code quality score for that run; the run failed its tests, counts as a failed task in Passed (over all 23) and is priced. Ran on Codex CLI 0.155.0, the first release that serves GPT-6 Sol on a ChatGPT plan; no run reached the 3-hour bound.
  • * GPT-6.1 Sol at medium includes paddockcore with Code quality from Muse Spark 1.3 alone: Grok 4.6's review of that run had no valid response (both attempts quoted a changed line), so the v3.18 protocol scores it from the one valid review, as it does for any such run. Every GPT-6.1 Sol cell is judged on 23 runs. Ran on Codex CLI 0.159.0, the first release that serves GPT-6.1 Sol on a ChatGPT plan; no run reached the 3-hour bound.
  • ◊ Grok 4.7 (xAI) ran in Cursor's agent CLI 2026.10.01, which offers Low to Extra High and no Max. It is judged by Muse Spark 1.3 and GPT-6.1 Sol under v3.20, because Grok 4.6, the second judge for every other column, is not neutral for an xAI model. GPT-6.1 Sol rates Grok 4.7's code about 6 points above Muse does, so its Code quality and combined score are not strictly comparable with the other columns. Rescored from Muse Spark 1.3 alone for every column, Grok 4.7 scores 88.64 to 92.27: still first at Medium, High and Extra-high, and second at Low behind Fable 5.1 (89.46). Medium is judged on 22 of 23 tasks: lodgecore hit the 3-hour bound, counts as a failed task in Passed (over all 23) and in Min/task, and has no score. $/task is unavailable, not $0: VulcanBench has no list price for Grok 4.7 and the sweep ran on the Cursor subscription, so it is not on the cost chart and its effort suggestions are picked on time alone.
  • †† Sonnet 5.5 ran in Claude Code 2.1.291 to 2.1.293 (low 21 tasks on 2.1.291 and 2 on 2.1.292; medium and high on 2.1.292; extra-high 22 on 2.1.292 and paddockcore on 2.1.293; max on 2.1.293) with the refusal fallback on; no reply came from another model. The sweep predates the harness's tagged-worktree rule, so its run summaries carry no source block and record task hashes in an older format that counted __pycache__ files; task content was verified identical to the frozen suite lock and judging admitted the runs through a committed hash bridge (all 107 cached .pyc files an agent could see were byte-identical to compiling the starting source). Judged under v3.23 with judge settings identical to the original v3.3 and v3.4 protocol files (recovered from a private backup, hashes equal to the published ones). Muse Spark 1.3 ran the same binary; Grok 4.6 ran through Cursor CLI 2026.10.01, which updates itself (the v3.3 round recorded 2026.09.02), with the same model id (Cursor now labels it "Grok 4.6 Medium", a rename the judging wrapper accepts). Both judges passed calibration using the one-gate allowance. $/task is Claude Code's own reported cost, because Anthropic's pricing page gives two cache-read rates for Sonnet 5.5.
  • ‡‡ Haiku 5.5 ran in Claude Code 2.1.293 to 2.1.296 (low 22 tasks on 2.1.293 and 1 on 2.1.294; medium 3 on 2.1.294 and 20 on 2.1.295; high on 2.1.295; extra-high 2 on 2.1.295 and 21 on 2.1.296; max on 2.1.296) with the refusal fallback on; no reply came from another model. Medium, extra-high and max are judged on 22, 22 and 21 of 23 tasks: on paddockcore at each a safety classifier stopped the session while the model probed the legacy binary, leaving an empty patch, and at max codeccore has no published Code quality score because Grok 4.6's intent probe gave no valid answer. All four count in Passed (over all 23): the paddockcore runs as failed, codeccore as passed. Haiku 5.5 costs five times as much for prompts over 100,000 tokens and receipts hold session totals, so $/task is Claude Code's own total over every run, which sits between a floor and a ceiling given in its report. Judged under v3.24 with the recovered original judge settings; Grok 4.6 ran through Cursor CLI 2026.10.01, pinned by version for the whole round. Both judges passed calibration with no gate missed. The sweep paused twice between tasks, leaving run timings intact.
  • §§ Muse Wonder Full (Meta) ran in Muse Code in Contributor mode with its skill reminder off: low and 21 medium tasks on Muse Code 1.4.6, then the last 2 medium tasks, high, extra-high and max on Muse Code 1.4.9, which runs its goal and verify reminder observers on Muse Spark 1.3. Max ran later, on 3 November 2026, after a probe showed Contributor mode honors it. It is judged by Claude Opus 5.5 and GPT-6.1 Sol under v3.26 to v3.26.5, not by Muse Spark 1.3 and Grok 4.6: Muse Spark 1.3 is a Meta model and is not neutral for a Meta submission. Its Code quality and combined score are therefore not strictly comparable with the other columns; Opus 5.5 rates its reviewed Code quality about 17 points below GPT-6.1 Sol. High is judged on 22 of 23 tasks: paddockcore hit the flat 3-hour task bound and has no finished code to judge; it counts as a failed task in Passed (over all 23) and is included in Min/task and $/task. Counting the timeout as a combined score of 0 over all 23 runs gives 84.36 at high; its best tag and effort suggestions use that figure. Extra-high depotcore first timed out while another job shared the machine; it was rerun alone and the rerun, judged in v3.26.4, replaces it. $/task is an estimate: Meta lists no API price for Muse Wonder Full, so its tokens are priced at the Muse Wonder 1.0 list price ($4.00 input, $0.20 cached, $20.00 output per million) and the 1.4.9 reminder tokens at Muse Spark 1.3's.
  • ¶¶ DeepSeek V4.1 Flash (deepseek-flash, which DeepSeek's pricing page mapped to DeepSeek-V4.1-Flash-1017) ran in DeepSeek Harness 0.2.1-rc.2, DeepSeek's own agent, a developer preview pinned by version and install hash, with its session-log and telemetry uploads off. DeepSeek's API offers low, high and max only, so it has no medium or extra-high column. It is judged by Muse Spark 1.3 and GPT-6.1 Sol under v3.27, not by Muse Spark 1.3 and Grok 4.6: Grok 4.6 judges through Cursor, whose account was out of usage, so the owner seated GPT-6.1 Sol, as v3.20 did for Grok 4.7. GPT-6.1 Sol rates its reviewed Code quality about 8 points above Muse (7.3 to 9.6 by level), so its Code quality and combined score are not strictly comparable with the other columns, and GPT-6.1 Sol judges a column it is shown beside. Every cell is judged on 23 runs; none reached the 3-hour bound. Unlike the other columns it ran on a metered API key, not a subscription: $/task is DeepSeek's off-peak list price ($0.15 input, $0.003 cached, $0.60 output per million); weekday peak hours bill 2x, so the billed amount was $0.068 to $0.101 per task.
  • ◊◊ Reflection Beam (Beam-1.0-501B-A23B, 501B total and 23B active parameters) ran in Mirror CLI 0.1.1, Reflection's own coding agent, an early beta pinned by version and install hash, through Reflection's API with analytics off and a private Mirror home per run. It is judged by Muse Spark 1.3 and GPT-6.1 Sol under v3.28, the v3.27 panel, not by Muse Spark 1.3 and Grok 4.6, because Grok 4.6's Cursor account was out of usage; GPT-6.1 Sol rates its reviewed Code quality about 14 points above Muse (10.1 to 17.6 by level), and GPT-6.1 Sol judges a column it is shown beside. Runs that left no code to review are not judged and count as failed tasks: 12 at low and 3 at medium (empty patches, or only file-mode and scratch changes), so those cells are judged on 11 and 20 of 23 runs; counting them as 0 over all 23 gives 18.90 at low and 38.95 at medium, and its effort suggestions use these figures. Max is judged on 22 of 23: hedgecore passed its tests, but Beam left 6.5M characters of scratch files beside its code, beyond GPT-6.1 Sol's input limit, and Muse's two reviews cited no code, so no judge scored it. Reflection publishes no Beam price, so $/task is unavailable and Beam appears on the Tokens and Minutes views only.
  • ** GPT-6.1 Terra ran in Codex CLI 0.159.0 on a ChatGPT subscription, from the day after its release, and is judged by Muse Spark 1.3 and Grok 4.6 under v3.29. At max, Muse Spark 1.3's intent probe of settlecore is published through the owner-approved escape rule first used for GPT-6.1 Sol: both attempts quoted a byte-string line with its escapes decoded into control characters, and the rule writes them back as the code spells them. At extra-high, three runs printed Codex's notice that it had no model metadata for gpt-6.1-terra and used fallback settings; all of them passed their tests. Every cell is judged on 23 runs; no run reached the 3-hour bound.
  • ‖‖ Claude Fable 5.2 ran in Claude Code 2.1.321 with its refusal fallback on; the fallback never answered, and it solved all 115 runs, every cell judged on 23. It is judged by Muse Spark 1.3 and GPT-6.1 Sol under v3.30.2 (low to extra-high) and v3.30.3 (max), not by Muse Spark 1.3 and Grok 4.6: Grok 4.6 failed one calibration gate under v3.30 and judged nothing, so the owner seated GPT-6.1 Sol. GPT-6.1 Sol rates its reviewed Code quality about 7 points above Muse (6.7 to 8.1 by level), so its Code quality and combined score are not strictly comparable with the Muse and Grok columns; by Muse alone its combined score is 91.09 to 92.78. The Fable 5.1 and Opus 5.5 rows keep their published Muse and Grok scores. Its runs recorded thinking summaries, a run condition earlier Claude columns did not have. $/task is Claude Code's own list-price total; max waited for the Fable usage limit to reset.
  • *** GPT-6.2 Sol ran in Codex CLI 0.166.0 on a ChatGPT subscription (the login reported plan prolite; earlier GPT columns ran on Pro), on its release day and the next, and is judged by Muse Spark 1.3 and Grok 4.6 under v3.31. At low, granarycore has Code quality from Muse Spark 1.3 alone: Grok 4.6's review of that run had no valid response (both attempts quoted code the submission does not contain), so the protocol scores it from the one valid review, as it did for GPT-6.1 Sol at medium. It passed every hidden test in all 115 runs, every cell is judged on 23, and no run reached the 3-hour bound.

SE is one task standard error of the combined score. Astra’s $/task is the central estimate; its report carries a long-context upper bound. A dot marked best is that model’s highest-scoring effort level. Per-run records, judge sub-scores and pricing are in each report’s evidence bundle: Astra vs. Fable 5.1, GPT-5.5 vs. Luna, Terra, Sol, Opus 5.5, GPT-6 Luna, GPT-6 Sol, GPT-6.1 Sol, Grok 4.7, Sonnet 5.5, Haiku 5.5, Muse Wonder Full, DeepSeek V4.1 Flash, Reflection Beam, GPT-6.1 Terra, Claude Fable 5.2, GPT-6.2 Sol. Download the board as CSV.