Claude Haiku 5.5 across every effort level

Across every effort level

Claude Haiku 5.5 (claude-haiku-5-5) ran every task once at each of its five effort levels, 23 tasks per cell, through Claude Code 2.1.293 to 2.1.296 on a Claude Max subscription, from October 8, 10:40 UTC, to October 11, 22:49 UTC, 2026, on the owner’s Mac in the local sandbox, one task at a time, with two pauses between tasks. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a named human reader by Muse Spark 1.3 and Grok 4.6 with a ground-truth intent-recovery probe. No run reached the flat 3-hour task bound; the longest, Medium depotcore, took 126 minutes. Anthropic’s default effort for Haiku 5.5 is Medium, on the API and in Claude Code.

Under the prior 50/15/15/20 profile applied to the same Code quality scores the ladder runs 85.92 to 93.31. Runtime and cost here cover every run; the score card averages the judged runs, so its minutes and cost at Medium, Extra-high and Max (24.9, 19.4 and 27.9 minutes; $1.41, $0.87 and $1.55) differ slightly from these, and its tasks-passed row at Max counts the 21 judged runs (21 of 21).

Four runs without a Code quality score

• Three safety-classifier stops. On paddockcore at Medium, Extra-high and Max, a safety classifier stopped the session while Haiku 5.5 probed the legacy binary, after 4.7, 9.3 and 29.7 minutes, and the run ended with an empty patch: there is no code to review. As for Claude Opus 5.5’s High depotcore, these runs are excluded from judging, count as failed tasks, and are priced and timed. The combined score at those three levels is over the judged runs, so it does not include these failures; tasks passed does.
• Max codeccore is unpublished. Grok 4.6’s intent probe for this run gave no valid answer after the protocol’s single retry: attempt 1 quoted a code line absent from the run, which no recovery rule accepts, and attempt 2 was not valid JSON. The frozen protocol therefore publishes no Code quality or combined score for the run, exactly as for the same task under v3.7 and v3.17. The run passed its tests, so it counts as a passed task, and it is priced and timed.

Max is therefore judged on 21 of 23 runs, and Medium and Extra-high on 22.

Cost, tokens and Haiku’s price tiers

Haiku 5.5 has two list prices per token. Anthropic’s pricing page, checked October 7, 2026, charges $0.10 input, $0.125 for 5-minute cache writes, $0.01 for cache reads and $0.50 output per million tokens for prompts up to 100,000 tokens, and five times as much on every line ($0.50, $0.625, $0.05 and $2.50) for prompts over 100,000 tokens. Claude Code’s receipts hold the session’s cumulative usage, not the size of each request, so a run cannot be split between the tiers after the fact. Each run therefore carries three figures: Claude Code’s own list-price total (cli_reported_cost_usd, which prices each request at its own tier and is the published cost), a floor that prices every request at the standard tier, and a ceiling that prices every request at the higher one. Haiku 5.5 ran on a Claude Max subscription, so all three are API-equivalent estimates, not bills; judging is excluded.

On the Frontier v4 board

Haiku 5.5’s best level, Max at 92.16 ± 0.37 (n=21), ranks 4th of the board’s 63 model and effort columns, behind only Grok 4.7’s three top levels, which are judged by a different panel (Muse Spark 1.3 and GPT-6.1 Sol). Among columns judged by Muse Spark 1.3 and Grok 4.6 it is the highest point estimate: 0.14 points above Claude Sonnet 5.5 at Max (92.02 ± 0.35) and 0.32 above Claude Fable 5.1 at Max (91.84 ± 0.47), both well within the errors. Extra-high ranks 7th, High 25th, Low 37th and Medium 40th. The leaderboard carries every column; Haiku 5.5’s cells carry the ‡‡ footnote with the disclosures below.

The board’s combined scores are over judged runs. Haiku 5.5’s Medium, Extra-high and Max cells leave out the failed paddockcore runs that a safety classifier stopped, so read them beside tasks passed, which counts every run.

What Code quality measures

Every judge receives the same rubric, frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel carries 24 points. An intent-recovery probe carries 9: the judge, given only the specification and the code, lists where the code departs from the specification, and a separate call matches that list against a frozen answer key, over the departures the submission passed tests for. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force. Frontier Code quality is therefore the reviewed layer plus intent recovery, and it is never compared with Routine v1 Code quality, which is scored on a different suite and population.

Both sub-scores rise with effort, human readability from 71.1 to 78.7 and maintainability from 78.2 to 82.6, and readability is the weaker of the two at every level. The two judges agree closely: within 2.1 points at every level, Grok 4.6 higher at Low and Muse Spark 1.3 higher from Medium up. Per-run sub-scores from each judge are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, and Grok 4.6 (xAI) through the Cursor CLI at medium effort, with equal weight. Both judges are from labs other than Anthropic, so neither grades a relative of Haiku 5.5. Every session is fresh, tools are disabled, the workspace is empty, model identity is checked per call from the CLI’s own records, and solver labels are withheld. Protocol v3.24 changes nothing in the rubric, controls, quirk keys, gates, repeats, seed, weights or judge settings from v3.23; it applies the protocol to this population, and both judges retook the calibration exam under it before any counted call.

Judge settings and versions. The judge settings are read from the original v3.3 and v3.4 protocol files, recovered from the owner’s private backup, whose sha256 equal the published values (v3.3 b82da598, v3.4 1d80e097). Muse Spark 1.3 ran the same binary as every earlier round (sha256 match). Grok 4.6 ran through Cursor CLI 2026.10.01-e373342, pinned by version for the whole round: the round judged through that version’s own launcher and re-checked a digest of its version directory before every stage, so the CLI could not change mid-round. It is the same Cursor CLI that judged Claude Sonnet 5.5 under v3.23. The Grok model id (cursor-grok-4.6-medium) is the same as in every earlier round; Cursor reports its display name as “Grok 4.6 Medium” where the frozen settings say “Cursor Grok 4.6 Medium”, a rename the judging wrapper accepts and records (426 times in this round).

Before scoring a single submission, each judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point.

Both judges passed every gate with no allowance used. Under v3.23, for Sonnet 5.5, each needed the one-gate allowance. Every gate value and control mean is in calibration.json.

Each judge made 80 calibration calls, 112 primary reviews, 5 repeats, 8 pairwise checks and 112 intent probes; Muse made 112 answer-key matches and Grok 111, since its probe on Max codeccore had no valid answer. Muse needed a second attempt on nine calls and Grok on five, under the protocol’s single-retry rule. One Grok review was recovered by a standing operator rule: on Extra-high vaultcore both attempts failed only because a quoted code line is hard-wrapped in the source, so the wrapper selected attempt 1 with the quote re-wrapped to the source’s line breaks; scores are untouched and the original quote is kept in the receipt. Neither judge produced a reviewer fallback. Every attempt is archived beside its replacement in the harness run directory.

How it was run

• From a tagged run worktree. The sweep ran from bench/2026-10-07-haiku55-frontier-v4, so every run summary carries a clean source block and records the frozen suite lock’s task hash. No hash bridge was needed, unlike Sonnet 5.5.
• Claude Code versions. Claude Code updated itself during the sweep. From each run’s own start-up record: Low ran 22 tasks on 2.1.293 and 1 on 2.1.294; Medium 3 on 2.1.294 and 20 on 2.1.295; High on 2.1.295; Extra-high 2 on 2.1.295 and 21 on 2.1.296; Max on 2.1.296. Earlier Claude columns used older Claude Code releases (Sonnet 5.5 on 2.1.291 to 2.1.293).
• Refusal fallback on, never used. Claude Code’s refusal fallback stayed at its default (on), as for Opus 5.5 and Sonnet 5.5. No run used it: every reply in every run came from claude-haiku-5-5.
• Two pauses between tasks. The owner needed the laptop, so the sweep was stopped between tasks from October 9, 15:43 UTC, to October 10, 09:20 UTC, and from October 10, 19:48 UTC, to October 11, 15:50 UTC. Each time the one run in progress was set aside and re-run from the start, and no counted run was in progress across a pause, so every run’s timing is intact. The host’s clock moved from PDT to CEST during the sweep, so all times here are UTC. When the first pause ended, Claude Code replaced itself (2.1.295 to 2.1.296) and left 25 extra-high folders holding only a start record; every one of those tasks then ran to completion in its own folder, and the stubs are not part of the population.
• Where and how. --billing subscription on a Claude Max plan, the local sandbox on the owner’s Mac (the same host as every published Frontier v4 column), a flat 3-hour task timeout, one task at a time. The sweep ran October 8, 10:40 UTC, to October 11, 22:49 UTC.
• Judging window. Judging under v3.24 ran October 11, 22:52 UTC, to October 12, 10:11 UTC, in one window with no stops, and no solver sweep ran during it.
• One attempt per task and level. Runs were not repeated, and effort labels are Claude Code’s own.

Evidence and reproduction

Public files contain all 115 run measurements, both judges’ six-dimension sub-scores and intent-recovery scores per judged run, hidden behaviours fixed, integrity-audit verdicts, each run’s task hash, Claude Code version, replies by serving model, raw tokens, cost with its tier floor and ceiling, five aggregates, every calibration gate value, the exact judge protocol text, the judge settings and Cursor version record, the operator record for the unpublished probe, and source hashes. Each run’s Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, where both terms are the equal mean of the two judges. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol documents, the recovered judge protocols and the decision record. Redacted agent traces are published at vulcanbench-haiku55-traces.

No human rated anything; the scores are model judgment for a human reader, not human validation. Hash checks bind the export to frozen files; they do not prove that judges were unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), reviewer session identifiers and usage receipts, the unused Grok probe attempts on Max codeccore, and host binary paths. Summed solver time is 46.15 hours; judging is excluded.