GPT-6 Astra vs. Fable 5.1 under a neutral Code quality panel

Across five effort levels

Each model ran every task at Low, Medium, High, Extra-high and Max effort, 23 tasks per cell. No solver ran for this report: functional, lint and complexity, and security measurements come from that sweep. What is new is the Code quality factor, now 33% of the combined score, scored for a named human reader by Muse Spark 1.3 and Grok 4.6, two judges from labs with no model on this board, with a ground-truth intent-recovery probe. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality.

Applying the prior 50/15/15/20 profile to the same Code quality scores still puts Fable ahead at every effort, so the ranking does not depend on the weight change. The card and the full per-effort table follow.

Cost and tokens across effort levels

The same 230 runs priced from their solver receipts at list API rates, cache-aware, with judging excluded. These are API-equivalent estimates, not subscription charges: both models ran on subscriptions, and what a plan actually bills is not observable from the receipts. Astra is cheaper at every effort, by 3.5 to 6.3 times per task, and its cost rises with effort while Fable's does not track it.

What Code quality measures now

Every judge receives the same rubric, frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel above carries 24 points. An intent-recovery probe carries 9: every task in this suite is a rewrite of a retired binary whose real behaviour departs from its written specification in documented ways, and the judge, given only the specification and the code, must list where the code departs from the specification. A separate call matches that list against a frozen answer key. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force and is stated on the card.

Both judges agree on every ranking and differ by a few points on levels. The largest gap between the models is human readability. Intent recovery, the only layer with ground truth, is close. Per-run sub-scores from each judge are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, and Grok 4.6 (xAI) through the Cursor CLI at medium effort, with equal weight. Neither lab has a model on this board, so neither judge grades a relative. Every session is fresh, tools are disabled, the workspace is empty, model identity is checked per call from the CLI's own records, and solver labels are withheld. Astra and Claude Opus 5 also passed the calibration exam under an earlier protocol version but were retired from the score by the benchmark owner, because Astra would otherwise have judged its own code while Fable never did.

Before scoring a single submission, each judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point.

GLM 5.3 was tried first and failed on validity after fabricating an excerpt in both attempts on one control; Muse Spark 1.3 replaced it under amendment v3.4. Both scored judges separate the clear program from the compressed one by more than two points on naming and presentation, credit formatting on presentation but not on naming, penalise misleading comments on intent, and penalise hidden state on verifiability. Every gate value and control mean is in calibration.json.

Evidence and reproduction

Public files contain all 230 run measurements, both judges' six-dimension sub-scores and intent-recovery scores per run, raw tokens and API-equivalent cost estimates per run, ten aggregates, every calibration gate value, the exact judge protocol text and source hashes. Each run's Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, where both terms are the equal mean of the two judges. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol documents, controls, runner and operator wrapper.

No human rated anything; the scores are model judgment for a human reader, not human validation. Runs were not repeated, CLI versions were not held fixed across weeks, and effort labels are harness-specific. Hash checks bind the export to frozen files; they do not prove that judges were unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), reviewer session identifiers and usage receipts, and the retired Astra and Opus 5 reviews. Summed solver time is unchanged at 70.75 hours; judging is excluded.