Leaderboard

11 models, 53 model×effort columns, 1,209 runs. Combined score is 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under one frozen protocol (v3.4 to v3.7, v3.15 to v3.18 and v3.20 apply the same rubric, controls and gates to each population; v3.20 seats GPT-6.1 Sol in place of Grok 4.6 to judge Grok 4.7, see ◊). The chart plots combined score against cost per task, $0 on the left, one line per model from Low to Max; the cost axis shows up to $5 per task and scrolls sideways for anything costlier. The table below carries every column. Completion tokens are the model's own output per task, reasoning included. $/task is API-equivalent at list rates from the solver receipts; every model here ran on a subscription. Grok 4.7 has no list price, so its $/task is unavailable and it appears on the Tokens and Minutes views only.

† Fable 5.1 runs include 11 disclosed Opus 4.8 fallbacks across the sweep; they stay in the population. ‡ GPT-5.6 Terra at max includes paddockcore, run on September 17 on a second ChatGPT account after the first hit its quota window and judged under the v3.6.1 top-up with the same judges and calibration. § GPT-5.6 Sol at max is judged on 22 of 23 tasks: on codeccore, Grok 4.6's intent probe quoted an excerpt absent from the code on both attempts, so the v3.7 protocol publishes no Code quality score for that run; the run passed its tests and is priced. ¶ GPT-6 Luna at extra-high and max is judged on 21 and 19 of 23 tasks: the other 2 and 4 runs hit the flat 3-hour task bound while still working, have no finished code to judge, count as failed tasks and are unpriced ($/task covers the finished runs; Min/task covers all 23). Counting each timeout as a combined score of 0 over all 23 runs gives 71.94 at extra-high and 67.26 at max; its best tag and effort suggestions use these figures. Ran on Codex CLI 0.155.0, the first release that serves GPT-6 Luna on a ChatGPT plan; the extra-high pacecore run was retried after an 88-minute Codex client stall, and the retry counts. ‖ GPT-6 Sol at medium is judged on 22 of 23 tasks: on codeccore, Grok 4.6's intent probe gave no valid answer (one malformed attempt, one quoting code absent from the run), so the v3.17 protocol publishes no Code quality score for that run; the run failed its tests, counts as a failed task in Passed (over all 23) and is priced. Ran on Codex CLI 0.155.0, the first release that serves GPT-6 Sol on a ChatGPT plan; no run reached the 3-hour bound. * GPT-6.1 Sol at medium includes paddockcore with Code quality from Muse Spark 1.3 alone: Grok 4.6's review of that run had no valid response (both attempts quoted a changed line), so the v3.18 protocol scores it from the one valid review, as it does for any such run. Every GPT-6.1 Sol cell is judged on 23 runs. Ran on Codex CLI 0.159.0, the first release that serves GPT-6.1 Sol on a ChatGPT plan; no run reached the 3-hour bound. ◊ Grok 4.7 (xAI) ran in Cursor's agent CLI 2026.10.01, which offers Low to Extra High and no Max. It is judged by Muse Spark 1.3 and GPT-6.1 Sol under v3.20, because Grok 4.6, the second judge for every other column, is not neutral for an xAI model. GPT-6.1 Sol rates Grok 4.7's code about 6 points above Muse does, so its Code quality and combined score are not strictly comparable with the other columns. Rescored from Muse Spark 1.3 alone for every column, Grok 4.7 scores 88.64 to 92.27: still first at Medium, High and Extra-high, and second at Low behind Fable 5.1 (89.46). Medium is judged on 22 of 23 tasks: lodgecore hit the 3-hour bound, counts as a failed task in Passed (over all 23) and in Min/task, and has no score. $/task is unavailable, not $0: VulcanBench has no list price for Grok 4.7 and the sweep ran on the Cursor subscription, so it is not on the cost chart and its effort suggestions are picked on time alone. SE is one task standard error of the combined score. Astra’s $/task is the central estimate; its report carries a long-context upper bound. The best tag marks each model’s highest-scoring effort level. Per-run records, judge sub-scores and pricing are in each report’s evidence bundle: Astra vs. Fable 5.1, GPT-5.5 vs. Luna, Terra, Sol, Opus 5.5, GPT-6 Luna, GPT-6 Sol, GPT-6.1 Sol, Grok 4.7. Download the board as CSV.

Frontier v4 asks which model and effort level can do hard work. Routine v1 asks the everyday question: what is the cheapest effort level that is enough for an ordinary ticket. It is 12 private routine tickets on small Python packages (a targeted bug fix, a small feature behind a flag, input validation, an edge case in dates, money or text), each admitted because a frontier model at its lowest effort finds it easy. 8 models, 38 model×effort columns, 456 runs, one attempt per task and level, each model through its own CLI on a subscription. The tasks stay private so they cannot leak into training data, so only per-model, per-effort aggregates are published.

Suggested effort for routine work, by the same rule as the Frontier board: the cheapest level, then the fastest, within 3 combined-score points of the model’s best.

• Fable 5.1: low (12/12 passed, combined 94.2, its lowest level)
• Opus 5.5: low (12/12 passed, combined 94.2, its lowest level)
• GPT-6 Astra: low (12/12 passed, combined 94.8, its lowest level)
• GPT-5.6 Terra: low (12/12 passed, combined 93.1, its lowest level)
• GPT-5.6 Sol: low (12/12 passed, combined 94.4, its lowest level)
• GPT-5.6 Luna: low (12/12 passed, combined 92.5, its lowest level)
• GPT-5.5: low (12/12 passed, combined 94.9, its lowest level)
• Grok 4.7: low (12/12 passed, combined 97.3, its lowest level)

Combined score uses the Frontier weights: 50% functional correctness from hidden tests, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged by Muse Spark 1.3 and Grok 4.6 under Code quality protocol v3.8 (Opus 5.5 under v3.14, the same protocol on its own population, judged in a separate session) with the same rubric, controls, gates and calibration exam as Frontier v4. Grok 4.7 is judged under v3.22 by Muse Spark 1.3 alone: Grok 4.6 is not neutral for an xAI model, and GPT-6.1 Sol, seated in its place, failed that protocol’s calibration exam on two gates, so the single-panel rule publishes from Muse. Muse rates code above Grok 4.6, so rescoring every other column from Muse alone puts their lowest tested levels at 93.1 to 95.6, against Grok 4.7’s 97.3 at Low: still first. Routine and Frontier Code quality are not comparable. On Frontier v4 part of Code quality measures whether a reviewer can recover each task’s deliberate legacy quirks; routine tickets have no such quirks by design, so the protocol’s own pre-registered rule scores Routine Code quality from the reviewed panel alone. Compare levels and models within this table, never across the two boards. SE is one task standard error of the combined score. Sec/task is mean wall clock. $/task is API-equivalent at list rates from the solver receipts, not a subscription bill; Grok 4.7 has no list price and ran on the Cursor subscription, so its cost is unavailable, not $0, and its suggestion is made on time alone. Download this table as CSV.

The final SWE v3 board, fifteen models and 42 model×effort columns on 23 tasks from real merged open-source PRs ranked by pass@1, is frozen and published as machine-readable JSON. The numbered reports on the Benchmarks page carry its per-task detail. SWE v3 scores are not comparable with Frontier v4.