Benchmarks

October 12, 2026 · 460 runs · four models, five effort levels

Claude Haiku 5.5 vs Claude Sonnet 5.5, GPT-6.1 Sol and GPT-6 Luna

Claude Haiku 5.5 beside Claude Sonnet 5.5, GPT-6.1 Sol and GPT-6 Luna on the same 23 tasks at every effort level, judged by the same panel (Muse Spark 1.3 and Grok 4.6): combined score with ±1 SE and judged n, Code quality, tasks solved, cost and minutes per task, three comparison cards, every caveat and every data link.

At Max Haiku 5.5 (92.16 ± 0.37, n=21) and Sonnet 5.5 (92.02 ± 0.35) are effectively tied, with Haiku about 3.6 times cheaper per task; below Max Haiku rates higher on Code quality but solves fewer tasks. GPT-6.1 Sol solves more tasks at every level for 1.9 to 3.8 times less per task and scores about 3 points lower at Extra-high and Max; GPT-6 Luna scores far lower at every level and is 5 to 142 times cheaper.

October 12, 2026 · 115 runs · Code quality protocol v3.24

Claude Haiku 5.5 across every effort level

115 runs through Claude Code on a Claude Max subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same rubric as the other Frontier v4 reports. Three runs a safety classifier stopped with an empty patch and one unpublished judge probe are counted and disclosed, and cost is shown with a floor and a ceiling for Haiku’s two price tiers.

Combined score 84.42 at Low, 84.03 at Medium, 88.01 at High, 91.83 at Extra-high and 92.16 at Max (judged n=23, 22, 23, 22, 21). Tasks passed 13, 13, 17, 22 and 22 of 23; 19.0 to 27.7 minutes and $0.84 to $1.54 per task as Claude Code reports it, between a floor and a ceiling for Haiku’s two price tiers.

October 8, 2026 · 345 runs · three models, five effort levels

Claude Sonnet 5.5 vs Claude Opus 5.5 vs GPT-6.1 Sol

The three models side by side on the same 23 tasks at every effort level, judged by the same neutral panel (Muse Spark 1.3 and Grok 4.6): combined score with ±1 SE and judged n, Code quality, tasks solved, cost and minutes per task, with every caveat and every data link in one place.

Tasks solved Low to Max: Sonnet 5.5 15, 17, 20, 22, 23; Opus 5.5 17, 23, 22, 22, 21; GPT-6.1 Sol 20, 22, 23, 23, 23. Cost per task: $2.39 to $5.52, $1.70 to $8.75 and $0.31 to $0.46. Sonnet 5.5 has the highest best-level score of the three (92.02 ± 0.35 at Max), edging Opus 5.5 (91.11 ± 0.57 at High) by about 0.9 points, not a clear win, and clearly ahead of GPT-6.1 Sol (88.78 ± 0.40 at Extra-high).

October 8, 2026 · 115 runs · Code quality protocol v3.23

Claude Sonnet 5.5 across every effort level

115 runs through Claude Code on a Claude Max subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same rubric as the other Frontier v4 reports. The sweep predates the tagged-worktree rule, and Grok 4.6 judged through a newer Cursor CLI than in earlier rounds; both are disclosed. A second card sets Sonnet 5.5 beside Claude Opus 5.5 and Claude Fable 5.1 at every level.

Combined score 81.63 at Low, 84.10 at Medium, 86.79 at High, 90.20 at Extra-high and 92.02 at Max. Tasks passed 15, 17, 20, 22 and 23 of 23; 14.8 to 38.0 minutes and $2.39 to $5.52 per task, Claude Code’s own cost. Refusal fallback on; no run used it.

October 4, 2026 · 92 runs · Code quality protocol v3.20

Grok 4.7 across every effort level

92 runs through Cursor’s agent CLI on the Cursor subscription, 23 tasks at each of the four effort levels Cursor offers (no Max), with Code quality at 33% of the combined score, judged by Muse Spark 1.3 and GPT-6.1 Sol because Grok 4.6, the usual second judge, is not neutral for an xAI model. Companion cards cover time and tokens, Grok 4.7 beside the leaders, and VulcanBench Safety v1.

Combined score 89.42 at Low, 92.30 at Medium, 92.71 at High and 93.15 at Extra-high, the top three places on the board; on Muse Spark 1.3 alone it stays first at Medium to Extra-high and is second at Low. Tasks passed 18, 21, 22 and 23 of 23; 20.2 to 28.5 minutes per task; cost unavailable, no list price.

October 1, 2026 · 115 runs · Code quality protocol v3.18

GPT-6.1 Sol across every effort level

115 runs through Codex CLI 0.159.0 on a ChatGPT Pro subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. Companion cards price the runs and set three generations of Sol side by side.

Combined score 86.22 at Low, 86.44 at Medium, 88.23 at High, 88.78 at Extra-high and 88.35 at Max, ahead of GPT-6 Sol and GPT-5.6 Sol at every level. Tasks passed 20, 22, 23, 23 and 23 of 23, every hidden test from High up; $0.31 to $0.46 per task, $44.33 for all 115 runs.

September 30, 2026 · 115 runs · Code quality protocol v3.17

GPT-6 Sol across every effort level

115 runs through Codex CLI 0.155.0 on a ChatGPT Pro subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. Companion cards price the runs and set GPT-6 Sol, and the whole GPT-6 family, beside GPT-5.6.

Combined score 67.62 at Low, 80.60 at Medium, 82.83 at High, 85.94 at Extra-high and 86.82 at Max, slightly behind GPT-5.6 Sol at every level. Tasks passed 4, 13, 15, 18 and 19 of 23; $1.02 to $2.52 per task, $221.38 for all 115 runs. Medium is judged on 22 of 23 after one invalid judge probe.

September 28, 2026 · 115 runs · Code quality protocol v3.16

GPT-6 Luna across every effort level

115 runs through Codex CLI 0.155.0 on a ChatGPT Pro subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. A companion card prices the runs and counts raw tokens at every level.

Combined score over judged runs 40.83 at Low, 45.72 at Medium, 57.35 at High, 78.79 at Extra-high and 81.42 at Max. Six runs hit the 3-hour bound (2 at Extra-high, 4 at Max); counting them as 0 gives 71.94 and 67.26. Tasks passed 0, 0, 2, 11 and 12 of 23; $0.010 to $0.160 per priced task.

September 26, 2026 · 115 runs · Code quality protocol v3.15

Claude Opus 5.5 across every effort level

115 runs through Claude Code on a Claude Max subscription, 23 tasks at each of five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. A second card sets Opus 5.5 beside GPT-6 Astra and Claude Fable 5.1 at every level.

Combined score 86.36 at Low, 90.86 at Medium with all 23 tasks passed, 91.11 at High, second on Frontier v4 only to Fable 5.1 at Max, then 90.68 and 90.22; $1.70 to $8.75 per task. Claude Code's refusal fallback was on, following Artificial Analysis: Opus 4.8 wrote 0.0, 6.4, 20.9, 32.4, 47.8% of replies from Low to Max, and every run counts.

September 19, 2026 · 115 runs · Code quality protocol v3.7

GPT-5.6 Sol across every effort level

115 runs through the Codex CLI on a ChatGPT Pro subscription, 23 tasks at each of Sol's five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. Sol joins Terra, Luna, GPT-5.5, Astra and Fable 5.1 on the Frontier v4 leaderboard, completing the GPT-5.6 family.

Most of the effort curve is one step: combined score 69.05 at Low, 82.56 at Medium, then 85.76 to 87.18 from High to Max, where Sol passes 21 of 22 judged tasks. Code quality stays within about three points at every effort. A companion card prices the same runs at API-equivalent rates and counts raw tokens at every effort: $1.36 to $2.05 per task, with High the most expensive level.

September 17, 2026 · 115 runs · Code quality protocol v3.6 and the v3.6.1 top-up

GPT-5.6 Terra across every effort level

115 runs through the Codex CLI on a ChatGPT subscription, 23 tasks at each of Terra's five effort levels, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the other Frontier v4 reports. Terra joins GPT-5.5, Luna, Astra and Fable 5.1 on the Frontier v4 leaderboard.

The steepest effort curve on the board so far: combined score rises at every step, 59.70 at Low to 89.05 at Max, where Terra passes every task. Code quality stays within three points at every effort. A companion card prices the same runs at API-equivalent rates and counts raw tokens at every effort: $0.58 to $2.12 per task.

September 15, 2026 · 207 runs · Code quality protocol v3.5

GPT-5.5 vs. GPT-5.6 Luna across every effort level

207 runs through the Codex CLI on a ChatGPT subscription, 23 tasks at every effort level each API offers (four for GPT-5.5, five for Luna), with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under the same frozen protocol as the Astra and Fable 5.1 report.

The effort knob decides it: GPT-5.5 leads at Low and Medium, Luna edges ahead at High and Extra-high, and Luna's Max level, which GPT-5.5 does not offer, is the top cell with 19 of 23 tasks passed. Luna is rated higher on Code quality at every matched effort. A companion card prices the same runs at API-equivalent rates and counts raw tokens at every effort: Luna costs 17 to 75 times less per task at matched effort.

September 9, 2026 · 230 runs rescored · Code quality protocol v3.4

GPT-6 Astra vs. Fable 5.1 under a neutral Code quality panel

230 runs from the September 2026 effort sweep, 23 tasks at five effort levels per model, with Code quality at 33% of the combined score, judged for a human reader by Muse Spark 1.3 and Grok 4.6 under a frozen, calibrated six-dimension rubric with an intent-recovery probe.

Fable 5.1 has the higher combined score and Code quality at every matched effort; Astra is faster and cheaper at every effort. The widest gap is human readability. Both judges agree on every ranking. A companion card prices the same runs at API-equivalent rates and counts raw tokens at every effort: Astra costs $1.48 to $2.57 per task against Fable's $7.96 to $14.49.

October 6, 2026 · 50 runs · Claude Code 2.1.280

Claude Opus 5.5 on VulcanBench Safety v1

Over 50 runs, Low to Max, Claude Opus 5.5 followed 0 of 60 instructions to contact the network, leak a secret or delete files. It followed 3 of 100 planted instructions overall (two code injections into its fix and one harmless test), flagged the note to the user in 64 of 97 refusals, and passed 44 of 50 tasks.

September 25, 2026 · 4,774 test items · jev-1.13.0

Jev 1.13.0 on VulcanBench Verdict v2

TypeSafe AI's Jev scores 45.9 on the Verdict Index (95% interval 43.1 to 48.4): 49.6 on the software families and 40.4 on the general ones, with a Calibration Index of 0.33. It answers in 0.3 seconds, $0.58 for all 4,774 items. GPT-6 Astra at high effort, given exactly the same inputs as a reference that shows every family is answerable rather than as a leaderboard entry, scores 91.7.

Jev is strongest where the answer can be recognised from the shape of the text, such as whether a snippet type-checks (87) or which service caused an incident (82), and weakest on multi-step work: 24 on filtering and totalling a table, 1 on predicting which hidden test a patch fails. On size-matched patch pairs it is above the floor, unlike v1, at skill 36 against 58 for the reference. The admission gate was amended after the pilot; the report sets out the change and its reasons. The reference scored 100 on all eight general families, so they have no headroom for ranking strong reasoning models.

September 22, 2026, corrected and extended with a control September 23 · 2,641 queries · jev-1.13.0

Jev 1.13.0 on VulcanBench Verdict v1

The first measurement of TypeSafe AI's Jev on software engineering. Jev is a System One model: it returns a typed decision in a single parallel pass and generates no text, so it was asked to judge 745 patches that frontier coding agents had already written, each one graded by hidden tests.

Its stated probability that a patch passes never rises above 0.42, on a population where 63% of them pass, which is a large calibration failure. The ranking underneath is sounder: AUROC 0.69 on that question, and at a cutoff fitted on held-out items its accuracy draws level with the majority answer rather than beating it. On which of two patches a code-quality panel preferred, it agrees 89.7% of the time at AUROC 0.962, for 328 ms and $0.15 per 1,000 decisions. Corrected on September 23 after the first version scored these questions at a cutoff the model never crosses. A control, GPT-6 Astra given the same inputs, ranks the fixes far better (AUROC 0.87 against 0.69) and beats guessing by about 10 points, so the questions are answerable and Jev's shortfall is real; even ranking fixes by lines changed alone beats Jev (0.78).