Reflection Beam across every effort level

Across every effort level

Reflection Beam ran every task once at each of the five reasoning levels its API takes (Extra-high is Beam’s xhigh), 23 tasks per cell, through Mirror CLI 0.1.1 on Reflection’s API, from October 23, 21:15 UTC, to October 25, 04:31 UTC, 2026, on the owner’s Mac in the local sandbox, one task at a time. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a named human reader by Muse Spark 1.3 and GPT-6.1 Sol with a ground-truth intent-recovery probe.

Under the prior 50/15/15/20 profile applied to the same Code quality scores the ladder runs 41.15 to 67.00. Code quality falls slightly as effort rises (72.92 at Low to 65.81 at Max), but Low’s figure rests on 11 runs that mostly changed little code; see What Code quality measures.

Beside Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra

At every level Reflection Beam is far below the three frontier models. Its best, 64.72 at Extra-high, compares with 90.68 for Claude Opus 5.5, 88.78 for GPT-6.1 Sol and 89.16 for GPT-6 Astra at the same level; at Max it is 63.92 against 90.22, 88.35 and 89.30. It fully solves 4 and 6 tasks at Extra-high and Max, against 22 and 21 for Opus 5.5 and 23 at both levels for GPT-6.1 Sol and GPT-6 Astra. It is the fastest of the four at Low and Medium only because it stops early; at Max it takes 22.4 minutes a task, faster than Opus 5.5 (37.5) but slower than GPT-6.1 Sol (14.5) and GPT-6 Astra (10.3). It also uses far more tokens: 17.29M a task at Max, about twice Opus 5.5’s 8.72M and about 17 and 18 times GPT-6.1 Sol’s and GPT-6 Astra’s, most of it cached input.

Time and tokens

Reflection publishes no price for Beam (its public API beta is free to start), so no run is priced: cost is unavailable, not zero, and the board shows Beam on its Tokens and Minutes views only. Tokens are summed from Mirror’s per-call usage records, input (cached included) plus output; Mirror reports no separate reasoning count. Judging is excluded.

On the Frontier v4 board

Reflection Beam’s best level, Extra-high at 64.72 ± 3.76 (n=23), ranks 63rd of the board’s 75 model and effort columns, below every column of Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra and just below GPT-5.6 Terra at Medium. By level it ranks Low 75th, Medium 72nd, High 68th, Extra-high 63rd, Max 64th; Low is last on the board. Extra-high carries the best tag, and with no price the board picks levels on time alone: its critical, routine and rough effort suggestions all pick Extra-high, because Max, the only level within 5 points, is slower. The leaderboard carries every column; Reflection Beam’s cells carry the ◊◊ footnote with the disclosures on this page, including the Low and Medium figures with no-code runs as 0.

Read these placements with the panel in mind. Most columns, Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra included, are judged by Muse Spark 1.3 and Grok 4.6; Grok 4.7’s, DeepSeek V4.1 Flash’s and Reflection Beam’s by Muse Spark 1.3 and GPT-6.1 Sol; Muse Wonder Full’s by Claude Opus 5.5 and GPT-6.1 Sol. Scores from different panels are not calibrated against each other, so gaps of a point or two between Reflection Beam and its neighbours are not a reliable order.

What Code quality measures

Every judge receives the same rubric, frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel carries 24 points. An intent-recovery probe carries 9: the judge, given only the specification and the code, lists where the code departs from the specification, and a separate call matches that list against a frozen answer key, over the departures the submission passed tests for. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force. Frontier Code quality is therefore the reviewed layer plus intent recovery, and it is never compared with Routine v1 Code quality, which is scored on a different suite and population.

Code quality is highest at Low and drifts down as Beam writes more code, from 72.92 to 65.81. At Low it rests on 11 runs, most of which changed little, and 8 of them passed no quirk family, so their score is the reviewed layer alone. Human readability falls most, from 71.8 at Low to 57.1 at Extra-high. Per-run sub-scores from each judge are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

GPT-6.1 Sol in Grok 4.6’s seat. The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, with its v3.4 settings and binary pin, and GPT-6.1 Sol (OpenAI) at medium effort through Codex CLI 0.159.0, its launcher and native binary pinned by sha256, with equal weight. Grok 4.6, the second judge for most columns, runs through Cursor, and a probe through the pinned Cursor CLI on October 24, 2026 still found the account out of usage. The owner kept GPT-6.1 Sol in the seat, as protocol v3.27 did for DeepSeek V4.1 Flash; this is the same pair, with the same settings and pins. Neither lab is Reflection. The Muse binary is a byte-identical backup of the v3.4 binary, matched by sha256 and re-checked before every stage. Every session is fresh, tools are disabled, the workspace is empty and solver labels are withheld. Muse’s identity is checked per call from the CLI’s own records; Codex does not record the serving model, so GPT-6.1 Sol’s identity is the model requested.

A different panel from the columns it is shown beside. Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra’s own columns, like most columns on the board, are judged by Muse Spark 1.3 and Grok 4.6. Panels are not calibrated against each other, so Reflection Beam’s Code quality and combined score are not strictly comparable with theirs. GPT-6.1 Sol therefore judges a column it is shown beside; its own column was judged without it. Protocol v3.28 changes nothing in the rubric, controls, quirk keys, gates, repeats, seed or weights from v3.27; it applies the protocol to this population with this judge pair.

Before scoring a single submission, each judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point. Both judges retook the exam under v3.28 before any counted call.

Both judges passed every gate with no allowance used. Every gate value and control mean is in calibration.json.

The judges disagree on level. GPT-6.1 Sol rates Reflection Beam’s reviewed Code quality about 14 points above Muse Spark 1.3: 15.2 at Low, 17.6 at Medium, 14.0 at High, 10.7 at Extra-high, 10.1 at Max. Both passed the same exam, so this is a difference in how strictly each scores real code, not a failed judge; GPT-6.1 Sol rated DeepSeek V4.1 Flash about 8 points above Muse in the same way. The published score is their mean. Scored by one judge alone, the combined score would be (Muse or Sol) 37.44 or 41.58 at Low; 42.21 or 47.37 at Medium; 52.90 or 57.12 at High; 63.24 or 66.19 at Extra-high; 62.28 or 64.72 at Max; either way the ladder climbs from Low to Extra-high. Each run’s single-judge figures are in runs.csv.

Each judge was sent 112 primary reviews, 112 intent probes, 5 repeats and 10 pairwise checks, and Muse 112 answer-key matches to Sol’s 111 (Sol had no probe for hedgecore). Muse needed a second attempt on two calls and Sol on 39, under the protocol’s single-retry rule: 36 answer-key matches and three primary reviews that first failed when Codex returned 429 after the ChatGPT usage window ran out. 24 of Sol’s answer-key matches are published through a standing operator rule: both attempts failed only because Sol numbered the departures from one instead of zero, so the wrapper moved the indexes down by one with every status untouched and kept the original in the receipt. Every operator marker is published in judge-protocols.json. Neither judge produced a reviewer fallback. Judging calls ran on October 25, 2026, from 04:31 to 20:21 UTC; the no-source-change exclusions were applied and the summary written on October 26.

How it was run

• The model. Reflection Beam 1.0 through Reflection’s API as Beam-1.0-501B-A23B, the versioned id Mirror uses by default. The API’s model list shows only the alias Beam-501B-A23B, which the sweep checked before each level; requesting the versioned id keeps a later Beam version behind the alias from slipping in. Every inference in Mirror’s conversation database names the requested model, so model identity is harness-recorded.
• Reflection’s own harness. Mirror CLI (reflection-mirror, Reflection’s open-source coding agent, Apache-2.0), version 0.1.1, an early beta, in its one-shot exec mode, pinned by exact version and a content hash of its installed package. It ran inside VulcanBench’s macOS outer sandbox with Mirror’s tool approvals automatic, since Mirror’s own sandbox workspaces need a microsandbox VM, and updates off. Mirror refuses to run until its usage notice is acknowledged and the folder trusted; at the owner’s approval each run had a private Mirror home with the notice acknowledged and only that run’s workspace trusted. Usage analytics were off and web tools absent, as for every harness on the board.
• Five levels. Beam’s reasoning_effort takes low, medium, high, xhigh and max; the board’s Extra-high is xhigh.
• A misleading sandbox note. Task files are staged without execute bits, as for every column on the board, and models that hit Permission denied on the legacy binary usually run chmod. When that happens in Mirror, it appends a note saying the sandbox denied access. Two Low runs ended saying the sandbox blocked the binary, which it does not, and two other Low answers said the full reconstruction was beyond the session’s scope. The adapter archived model messages and usage but not tool calls, so how often Beam saw the note cannot be counted.
• A daily token limit. Reflection’s API has a daily token limit. It stopped the sweep on October 24 at 13:38 UTC after 13 Max tasks; the 1,200 attempts it cut short were set aside unscored and the sweep resumed after the 00:00 UTC reset. No finished run was affected. Six individual model requests failed in five runs and Mirror carried on.
• From a tagged run worktree. Every run ran from bench/2026-10-23-beam-mirror-frontier-v4, so every run summary carries a clean source block and records the frozen suite lock’s task hash.
• No timeouts, no overlapping jobs. Every run finished inside the flat 3-hour bound; the longest, Extra-high blendcore, took 57.1 minutes. A per-minute log watched for any other local benchmark job beside the sweep and recorded none.
• Where and how. The local sandbox on the owner’s Mac, one task at a time, from October 23, 21:15 UTC, to October 25, 04:31 UTC. The API key was in the agent’s environment and the shell network was not isolated, as recorded on every run. Every integrity audit is clean: no web access, and no benchmark data or answer-key paths.
• One attempt per task and level. Runs were not repeated.

Evidence and reproduction

Public files contain all 115 run measurements, both judges’ six-dimension sub-scores and intent-recovery scores per judged run, the combined score from each judge alone, the reason for every unjudged run, hidden behaviours fixed, integrity-audit verdicts, each run’s task hash and harness version, model calls, failed requests and raw tokens, five aggregates, every calibration gate value, the exact judge protocol text, every operator marker and source hashes. Each run’s Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, where both terms are the equal mean of the judges with a valid review, or the reviewed score alone when the run passed no quirk family. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol document, the decision record, the Mirror CLI adapter and the code in the harness repository.

No human rated anything; the scores are model judgment for a human reader, not human validation. Hash checks bind the export to frozen files; they do not prove that judges were unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), reviewer session identifiers and usage receipts, Mirror conversation databases and solver traces, the 1,200 attempts cut short by the daily token limit, and host binary paths. Summed solver time is 20.21 hours; judging is excluded.