GPT-5.6 Sol across every effort level

Across every effort level

Sol ran every task once at each of its five effort levels, 23 tasks per cell, through the Codex CLI on a ChatGPT Pro subscription on September 17 to 18, 2026. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a named human reader by Muse Spark 1.3 and Grok 4.6, two judges from labs with no model on this board, with a ground-truth intent-recovery probe. Judging ran September 18 to 19. One Max run, codeccore, has no published Code quality score: Grok 4.6's intent probe quoted an excerpt absent from the code on both attempts and the frozen protocol accepts no invented characters, so the Max cell's Code quality and combined score cover 22 of its 23 runs. The run passed its tests and is counted in every runtime, token and cost figure.

Under the prior 50/15/15/20 profile applied to the same Code quality scores the ladder is the same shape, 72.24 to 89.90. The card and the full per-effort table follow.

Cost and tokens across effort levels

The same 115 runs priced from their Codex receipts at list API rates, cache-aware, with judging excluded. These are API-equivalent estimates, not subscription charges: Sol ran on a ChatGPT Pro subscription, and what the plan actually bills is not observable from the receipts. Cost per task does not follow the ladder: $1.36 at Low, $1.72 at Medium, $2.05 at High, the most expensive level, then $1.61 at Extra-high and $1.77 at Max. Runtime moves the same way, from 9.8 to 12.0 minutes with High the slowest. The whole sweep comes to $195.55 for 115 runs, every one of them priced, including the Max run without a published Code quality score.

What Code quality measures

Every judge receives the same rubric, frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel carries 24 points. An intent-recovery probe carries 9: every task in this suite is a rewrite of a retired binary whose real behaviour departs from its written specification in documented ways, and the judge, given only the specification and the code, must list where the code departs from the specification. A separate call matches that list against a frozen answer key, over the departures the submission passed tests for. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force and is stated on the card.

Code quality sits between 64.98 and 68.19 at every effort, with the two judges within four points of each other in every cell and Grok 4.6 the higher rater at each level. Intent recovery, the ground-truth layer, rises from Low to High as more of the hidden contract is implemented and holds there; human readability and maintainability do not move with the knob. Per-run sub-scores from each judge are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, and Grok 4.6 (xAI) through the Cursor CLI at medium effort, with equal weight. Neither lab has a model on this board, so neither judge grades a relative. Every session is fresh, tools are disabled, the workspace is empty, model identity is checked per call from the CLI's own records, and solver labels are withheld. Protocol v3.7 changes nothing in the rubric, controls, quirk keys, gates, repeats, seed or judge settings from v3.6; it applies the protocol to this population, and both judges retook the calibration exam under it before any counted call.

Before scoring a single submission, each judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point.

Both judges passed every gate with no allowance used; under v3.5 and v3.6 Muse Spark 1.3 had needed the pre-registered allowance on the repeatability gate, and under v3.7 it did not. Both judges separate the clear program from the compressed one by more than two points on naming and by 2.0 to 2.1 points on presentation, credit formatting mainly on presentation, penalise misleading comments on intent, and penalise hidden state on verifiability. Every gate value and control mean is in calibration.json.

Under v3.7 Muse Spark 1.3 made 440 calls: 80 in calibration, 115 primary reviews, 5 repeats, 10 pairwise checks, 115 intent probes and 115 answer-key matches. Grok 4.6 made 439, with 114 matches, since no match was made for the invalid codeccore probe. Muse needed a second attempt on nine calls (six primary reviews and three probes): four unsupported excerpts and five malformed JSON responses. Grok needed a second attempt on nine: five unsupported excerpts (two primary reviews, one calibration control and two probes, one of them the codeccore probe, which was unsupported on both attempts and is recorded as invalid), one malformed JSON probe, two transport faults when the Cursor CLI could not resolve its API host (a primary review and a calibration pair), each retried under the network-fault rule, and one calibration probe the provider blocked before any model output, retried once under a new wrapper rule recorded in the operations log. Neither judge produced a reviewer fallback. Every attempt is archived beside its replacement in the harness run directory.

Evidence and reproduction

Public files contain all 115 run measurements, both judges' six-dimension sub-scores and intent-recovery scores per published run, raw tokens and API-equivalent cost estimates per run, five aggregates, every calibration gate value, the exact judge protocol text, the operator's finding on the unpublished row and source hashes. Each run's Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, where both terms are the equal mean of the two judges, or the reviewed score alone where intent recovery has no denominator. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol documents, controls, runner and operator wrapper.

No human rated anything; the scores are model judgment for a human reader, not human validation. Runs were not repeated, and effort labels are Codex's own. The Max cell's Code quality and combined score are means over 22 of 23 runs, for the reason given under the effort table; the frozen summary's publication flag is false only because of that strict count, and the export asserts the shortfall is exactly that row. Hash checks bind the export to frozen files, including the operator's invalid-probe marker; they do not prove that judges were unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), and reviewer session identifiers and usage receipts. Summed solver time is 21.11 hours; judging is excluded.