Devin SWE-2 across every effort level

Across every effort level

Devin SWE-2 ran every task once at each of its three effort levels, 23 tasks per cell, through the Devin CLI (devin 3000.10.31) on a Devin subscription on September 18 to 21, 2026. SWE-2's catalog lists exactly three levels, medium, high and max; there is no lower level and no level above max. Combined score weights are 50% functional correctness, 8.5% lint and complexity, 8.5% security and 33% Code quality, judged for a named human reader by Muse Spark 1.3 with a ground-truth intent-recovery probe. Judging ran on September 21.

Medium and high are indistinguishable. They score 82.43 and 81.42, their standard errors overlap, and both pass 15 of 23 tasks. Only Max moves: 86.14, and 21 of 23 tasks passed. Devin is also slow and token-heavy next to the Codex models measured on this suite: 35 to 57 minutes and 7.53M to 16.80M raw tokens per task, against about 10 to 12 minutes and 1.8M to 2.8M for GPT-5.6 Sol. Its cost cannot be reported: Cognition publishes no rate for SWE-2.

The tasks-passed row counts every run the sweep attempted, since all four unjudged runs failed their tests. Under the prior 50/15/15/20 profile applied to the same Code quality scores the shape is the same, 85.46 at medium, 84.01 at high and 88.10 at Max. The card and the full per-effort table follow.

What was judged, and by whom

One judge, not two. Every other Frontier v4 entry is scored by a neutral two-model panel: Muse Spark 1.3 (Meta) and Grok 4.6 (xAI). Devin SWE-2 is scored by Muse Spark 1.3 alone. Grok 4.6 failed the calibration exam under protocol v3.9 and GPT-5.6 Sol, admitted under v3.10 to fill the second seat for this population only, failed under v3.10. Both failed the same gate, gate 16, the probe on the clear control: a program with no documented departure from its specification must draw an empty probe on at least four of five repeats, and each judge reported invented departures instead, Grok on two repeats and Sol on four. Gate 16 is boolean, so the pre-registered one-gate allowance cannot excuse it, and under the protocol's pre-registered single-panel rule no judge retakes a gate it failed. Both verdicts are published: they are in calibration.json beside Muse's passing one, and the harness records are in the operations log. Read every Code quality number on this page as one judge's rating.

65 of 69 runs are judged. Four runs produced nothing to judge and the population builder excluded them rather than scoring them. All four failed their tests and count as fails in the sweep's pass counts, so the exclusions do not flatter Devin's functional results.

A run that produced no code has nothing to judge, and the sweep's automated quality and security metrics are undefined for it by construction, so protocol v3.11 excludes such a run the way it excludes an unfinished one and lists it with the reason. The judged cells are 23 at medium, 20 at high and 22 at Max. Code quality and the combined scores are means over those judged runs; runtime and tokens cover all 68 finished runs, the three that changed no source file included. The one run that did not finish has no receipt, so its 3.0 hours of wall clock sit outside the token and runtime totals and are recorded separately in runs.json.

Cost and tokens across effort levels

Cost per task is unavailable, and that is the published value. Cognition publishes no per-token rate for SWE-2. The cost tier "Free" in Devin's catalog is a promotion dated through 2026-10-10, not a rate, so a figure taken from it would read as a measured price and would stop being true in under three weeks. No cost is estimated here, on this page or in the evidence bundle, and SWE-2 is not compared on cost with the priced models on the Frontier v4 leaderboard. Tokens, runtime and Devin's own credit and ACU counters are recorded instead.

What the sweep does spend is time and tokens, and it spends a lot of both: 776 million raw tokens and 53.7 hours of solver wall clock over 68 finished runs, an average of 11.41M tokens and 47.3 minutes per task. Neither follows the effort ladder. Medium is the heaviest level at 16.80M tokens per task, high the lightest at 7.53M and the fastest at 35.1 minutes, and Max sits between them on tokens while being the slowest at 56.6 minutes.

What Code quality measures

The judge receives a rubric frozen by hash before any review. It names the reader it scores for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. It tells the judge that its own ease at parsing dense code is not evidence of readability. Six dimensions are scored 0 to 4 in half steps against written anchors: naming, presentation and intent form the human readability sub-score; structure, changeability and verifiability form the maintainability sub-score. Every score must cite an exact excerpt from the code and a concrete consequence for that reader. The host computes the sub-scores; the judge does no arithmetic.

The 33 points have three designed layers. The reviewed panel carries 24 points. An intent-recovery probe carries 9: every task in this suite is a rewrite of a retired binary whose real behaviour departs from its written specification in documented ways, and the judge, given only the specification and the code, must list where the code departs from the specification. A separate call matches that list against a frozen answer key, over the departures the submission passed tests for. The third layer, measured maintenance, is designed for 12 points but is not yet built, so the pre-registered 24 plus 9 split is in force and is stated on the card.

Code quality runs 65.14 to 69.17 across the three levels. The rise is carried by the reviewed layer, whose human readability sub-score climbs 7.0 points from medium to Max; intent recovery, the ground-truth layer, is flat at 76.4 to 77.4. With standard errors of 1.81 to 2.91 the medium to high step is inside the noise, and only Max is clearly above medium. The panel note under the effort table applies to every figure here. Per-run sub-scores are in runs.csv; the exact rubric, system text, probe and match instructions are in judge-protocols.json.

Judges and calibration

The scored panel is Muse Spark 1.3 (Meta) through the Muse CLI on its Standard tier, which does not train on prompts or completions. Meta has no model on this board, so the judge does not grade a relative. Every session is fresh, tools are disabled, the workspace is empty, model identity is checked per call from the CLI's own records, and solver labels are withheld. Protocol v3.11 changes nothing in the rubric, controls, quirk keys, gates, repeats, seed or judge settings; it applies the protocol to the judged population of this sweep.

Before scoring a single submission, a judge reviews ten held-out programs that implement the same ledger specification: clear, compressed, compressed then auto-formatted, verbose with duplicated policy, needlessly abstracted, misleadingly commented, narrated with a comment on every line, a documented legacy quirk, an embedded instruction to give full marks, and hidden module state. Each program is reviewed five times in a seeded order. Twenty gates fixed in advance check that the judge sees the construct; a judge may miss at most one gate by at most half a point, and gate 16 is boolean, so the allowance cannot cover it.

Muse's verdict comes from the v3.9 freeze rather than a fresh exam: the exam is per judge and control set and neither changed between v3.9 and v3.11, so on the v3.6.1 precedent the runner reuses the verdict and refuses to freeze unless it passed. Muse passed 19 of the 20 gates outright and used the pre-registered one-gate allowance on the repeatability gate, within the allowance as written. It separates the clear program from the compressed one by more than two points on naming, credits formatting mainly on presentation, penalises misleading comments on intent and penalises hidden state on verifiability. Grok 4.6 and GPT-5.6 Sol each passed every other gate, repeatability included, and failed only gate 16, the one call that rewards saying nothing. Every gate value and control mean for all three judges is in calibration.json.

Under v3.11 Muse Spark 1.3 made 204 counted calls: 65 primary reviews, 3 repeats, 6 pairwise checks, 65 intent probes and 65 answer-key matches. No call needed a second attempt, no operator rule was invoked and no reviewer fallback occurred. The v3.9 and v3.10 passes against the earlier freezes, including the two failed exams, stay archived in the harness run directories.

Evidence and reproduction

Public files contain all 68 finished run measurements, the judge's six-dimension sub-scores and intent-recovery score per published run, raw tokens and Devin's own credit and ACU counters per run (and a null cost, since none is available), three aggregates, every calibration gate value for all three judges, the exact judge protocol text and the population record with its four exclusions. Each run's Code quality is (0.24 × reviewed score + 0.09 × intent recovery) / 0.33, both terms from the scored panel, or the reviewed score alone where intent recovery has no denominator. The combined score is 100 × (0.50F + 0.085Q + 0.085S + 0.33C / 100). The reproduction guide gives the public arithmetic checks and links the protocol documents, controls, runner and operator wrapper.

No human rated anything; the scores are model judgment for a human reader, not human validation. Runs were not repeated, and effort labels are Devin's own. The high cell's Code quality and combined score are means over 20 of 23 runs and the Max cell's over 22 of 23, for the reasons given above. Hash checks bind the export to frozen files; they do not prove that the judge was unbiased or that no training overlap exists.

Withheld: raw prompts and responses, submitted patches and reconstructed sources, the quirk answer keys (they describe hidden-test behaviour), and reviewer session identifiers and usage receipts. Summed solver time is 53.66 hours over the finished runs; judging is excluded.