GPT-6.1 Sol across every effort level

Across every effort level

GPT-6.1 Sol ran every task once at each of its five effort levels, 23 tasks per cell, through Codex CLI 0.159.0 on a ChatGPT Pro subscription from September 29 to 30, 2026. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a named human reader by Muse Spark 1.3 and Grok 4.6 with a ground-truth intent-recovery probe. Judging ran September 30 to October 1. No run reached the flat 3-hour task bound; the longest, Medium paddockcore, took 80 minutes. All 115 runs carry a published Code quality score. One of them, Medium paddockcore, is scored from Muse Spark 1.3 alone, because Grok 4.6's review of it had no valid response; see the note under the results table.

Under the prior 50/15/15/20 profile applied to the same Code quality scores the ladder has the same shape, 87.93 to 89.76. The card and the full per-effort table follow.

Perfect hidden tests from High up

From High to Max, GPT-6.1 Sol passes every hidden test on every task: 23 of 23 tasks and all 231 tested behaviours at each of the three levels. Frontier v4 therefore no longer separates GPT-6.1 Sol's top levels on correctness. What still separates them is Code quality, the lint and security scans, and cost: Extra-high scores 0.55 points above High on the combined score and costs 32% more per task, and Max is 0.43 points below Extra-high at 5% more.

We checked the two obvious alternative explanations. The harness's integrity audit is clean on all 115 runs: no web access, and no reads of benchmark data or answer-key paths. (The filesystem audit records paths outside the task workspace on every run, as it does for every Codex column, such as the shell and temporary scratch folders; none of them is benchmark data or an answer key.) And OpenAI's model page gives GPT-6.1 Sol a knowledge cutoff of April 30, 2026, before the Frontier v4 tasks were built and admitted in August 2026. Neither rules out every form of overlap, but the perfect results are not explained by the run reaching the answers.

The suite's charter already plans for this. Its saturation-pruning rule says that when a new frontier model generation lands, the suite is re-gated with three runs per model per task, and tasks that the weaker reference now solves every time, or that the stronger reference finishes in a median under 10 minutes, are pruned. Re-gating Frontier v4 against this generation is a planned follow-up; until then, read GPT-6.1 Sol's High to Max columns as at the suite's correctness ceiling.

Cost and tokens across effort levels

All 115 runs priced from their Codex receipts at list API rates, cache-aware, with judging excluded. These are API-equivalent estimates, not subscription charges: GPT-6.1 Sol ran on a ChatGPT Pro subscription, and what the plan actually bills is not observable from the receipts. Cost per task is $0.31 at Low, $0.40 at Medium, $0.33 at High, $0.43 at Extra-high and $0.46 at Max. Medium uses the most tokens per task (1.62M) and High the fewest (0.91M), which is why High costs less than Medium. The whole sweep comes to $44.33 for 115 runs.

Three generations of Sol

GPT-6 Sol and GPT-5.6 Sol ran the same 23 tasks through Codex in September and were judged by the same two judges under v3.17 and v3.7. GPT-6.1 Sol leads both at every effort level on combined score, tasks passed and cost per task. Its combined score is 86.22 to 88.78, against 67.62 to 86.82 for GPT-6 Sol and 69.05 to 87.18 for GPT-5.6 Sol; the lead is largest at Low (18.60 and 17.17 points) and smallest at Max (1.53 and 1.17). It costs 14% to 30% of GPT-6 Sol's price per task and 16% to 27% of GPT-5.6 Sol's, using 0.91M to 1.62M raw tokens per task against 3.26M to 8.32M and 1.79M to 2.83M. Its Code quality is also higher at every level (70.25 to 74.99, against 62.53 to 70.78 and 64.98 to 68.19).

On speed the picture is mixed: GPT-6.1 Sol is faster than GPT-6 Sol at every level and faster than GPT-5.6 Sol at Low, Medium and High, but slower at Extra-high (15.4 against 10.6 minutes) and Max (14.5 against 11.6). Three cautions on the speed figures: the three sweeps ran on different Codex CLI versions (0.159.0, 0.155.0 and 0.153.4); GPT-6.1 Sol's Low, Medium and High runs overlapped GPT-6 Sol's judging on the same machine, as GPT-6 Sol's sweep had overlapped GPT-6 Luna's (see How it was run); and its Extra-high and Max runs, the two slower than GPT-5.6 Sol, mostly ran after that window. Each model was judged in its own protocol run of the same frozen rubric.

On the Frontier v4 board

GPT-6.1 Sol's best level, Extra-high at 88.78, ranks 13th of the board's 49 model and effort columns, with Max 14th and High 15th. Above it are every Fable 5.1 level, Opus 5.5 from Medium to Max, GPT-6 Astra at Max and Extra-high, and GPT-5.6 Terra at Max. The leaders are Claude Fable 5.1 at Max (91.84), Opus 5.5 at High (91.11) and GPT-6 Astra at Max (89.30), 3.06, 2.33 and 0.52 points above GPT-6.1 Sol's best.

The gap is not correctness: Fable 5.1 at Max and GPT-6 Astra at Max also pass 23 of 23, and Opus 5.5 at High passes all 22 of its judged runs, so on functional correctness they tie. The gap is in the judged and scanned factors, Code quality above all. GPT-6.1 Sol's best Code quality is 74.99, against 82.43 for Fable 5.1 at Max, 77.32 to 79.98 for Opus 5.5 from Medium to Max, and 76.26 for GPT-6 Astra at Max. Against Fable 5.1 at Max, Code quality accounts for 2.46 of the 3.06 points and security for 0.48; against GPT-6 Astra at Max, Code quality accounts for 0.42 of 0.52. Against Opus 5.5 at High the security scan matters as much: Code quality accounts for 0.99 of the 2.33 points and security for 1.29 (99.55 against 84.35). GPT-5.6 Terra at Max has lower Code quality (72.52) and ranks just above GPT-6.1 Sol on its security score (99.13). On cost the order reverses: of the 16 columns scoring 88 or more, GPT-6.1 Sol's three are the cheapest, $0.33 to $0.46 per task against $1.71 to $14.49 for the rest. The leaderboard carries every column.

What Code quality measures

Every judge receives the same rubric, frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel carries 24 points. An intent-recovery probe carries 9: the judge, given only the specification and the code, lists where the code departs from the specification, and a separate call matches that list against a frozen answer key, over the departures the submission passed tests for. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force.

Code quality is about 70.5 at Low and Medium and about 74 from High up, with Grok 4.6 the higher rater at every level. Grok's Medium mean covers the 22 runs it reviewed validly. Human readability is the weaker sub-score at every level, 60.9 to 66.6 against 72.7 to 77.7 for maintainability. Per-run sub-scores from each judge are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, and Grok 4.6 (xAI) through the Cursor CLI at medium effort, with equal weight, the same pinned binaries as under v3.7. Neither lab has a model on this board, so neither judge grades a relative. Every session is fresh, tools are disabled, the workspace is empty, model identity is checked per call from the CLI's own records, and solver labels are withheld. Protocol v3.18 changes nothing in the rubric, controls, quirk keys, gates, repeats, seed, weights or judges from v3.7; it applies the protocol to this population, and both judges retook the calibration exam under it before any counted call.

Before scoring a single submission, each judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point.

Both judges passed every gate with no allowance used, as under v3.17. Every gate value and control mean is in calibration.json.

Under v3.18 each judge made 440 counted calls: 80 in calibration, 115 primary reviews, 5 repeats, 10 pairwise checks, 115 intent probes and 115 answer-key matches. One of Grok 4.6's 115 primary reviews, Medium paddockcore, has no valid response, as described above. Muse needed a second attempt on eight calls and Grok on nine. Muse's probe on High codeccore was recovered by a new operator rule, described under How it was run. Grok's calls reported the display label "Grok 4.6 Medium" for the pinned model id, which the operator wrapper's display-rename rule accepts, as under earlier amendments. Neither judge produced a reviewer fallback. Every attempt is archived beside its replacement in the harness run directory.

How it was run

• Codex CLI 0.159.0. Codex 0.155.0, 0.157.0 and 0.158.0 refuse GPT-6.1 Sol on a ChatGPT account before any work; 0.159.0 is the first release that serves it, so the column runs on 0.159.0 from its own install, and the earlier Codex columns keep their versions.
• The sweep overlapped another model's judging. The GPT-6.1 Sol solver sweep ran from September 29, 19:18 PDT, to September 30, 17:21 PDT. At the owner's request its first 11.5 hours, to September 30, 06:51 PDT, ran alongside GPT-6 Sol's v3.17 Code quality judging on the same machine. Every Low, Medium and High run and the first five Extra-high runs started inside that window, which matters for the wall-clock speed figures.
• Three evidence rebuilds. Three runs add test fixtures that the saved text patch cannot re-apply: Low payrollcore and Medium lodgecore add fixtures git treats as binary, and Low cellarcore's fixture text was altered by the text-mode capture. The v3.18 runner rebuilt their judging evidence from the saved workspace's index diff after checking it against the run's patch. Only test fixtures differ; the reviewed module is rebuilt as for every other run. The three method records are hashed in the evidence bundle.
• One escaping recovery on a Muse probe. On High codeccore, Muse Spark 1.3's first probe attempt wrote an invalid JSON escape, and its second quoted a code line with the string escapes decoded into control characters, so the excerpt no longer matched the source text. By owner decision this was treated as a JSON-escaping slip, not an invented quote: a new operator rule respells such an excerpt with the source's escapes, after which it must be verbatim and the attempt must pass every other check. The second attempt was selected, and the original excerpt is kept in the receipt.
• One Grok review scored from Muse alone. Grok 4.6's primary review of Medium paddockcore quoted a changed line on both attempts and was marked invalid under the owner's v3.14 precedent, so that run's Code quality is Muse Spark 1.3's alone, as described under the results table.
• Judging windows. Judging ran September 30, 20:58 PDT, to October 1, 06:01 PDT, and resumed from 06:04 to 12:52 PDT after the Grok finding above. No solver sweep ran during judging; the Grok 4.7 sweep waited for it to finish.
• One attempt per task and level. Runs were not repeated, no run was retried, and effort labels are Codex's own.

Evidence and reproduction

Public files contain all 115 run measurements, both judges' six-dimension sub-scores and intent-recovery scores per run (Muse's alone on Medium paddockcore), hidden behaviours fixed, integrity-audit verdicts, raw tokens and API-equivalent cost estimates per run, five aggregates, every calibration gate value, the exact judge protocol text, the operator records for the invalid Grok review, the Muse escaping recovery and the three evidence rebuilds, and source hashes. Each run's Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, where both terms are the equal mean of the judges with a valid review. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol documents, controls, runner, operator wrapper and decision records.

No human rated anything; the scores are model judgment for a human reader, not human validation. Hash checks bind the export to frozen files, including the operator's invalid-review marker and the recovered probe; they do not prove that judges were unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), reviewer session identifiers and usage receipts, and Grok's unused probe answer on Medium paddockcore. Summed solver time is 22.05 hours; judging is excluded.