Report No. 02

How Gemini 4 Neon thinks

2026-11-21 · Morgan Linton · VulcanBench Frontier v4, 92 runs, Low to Extra-high effort · trace analysis

Gemini 4 Neon solved 86 of its 92 VulcanBench Frontier v4 runs: every task at Medium, High and Extra-high, and 17 of 23 at Low. This report is about how. Its traces show a model that collects before it reasons. It reads the repository, runs the program, then reads the machine code, and only after all of that does it think at length and write the rewrite, usually in a single edit. Turning up the effort setting makes it read more, not check more.

~75%of a run gone before the first edit, at every effort level (Claude Fable 5.2: 60% at Low, 50% at Extra-high)
4.4xmore thinking from Low to Extra-high (9.1k to 40.3k tokens a task); thinking rises from 45% to 71% of output
32% → 71%of its thinking spent before reading code or binary, Low against Extra-high
0 of 92runs that mutation-tested their checks, built the Linux reference or timed the rewrite

The task in one paragraph

Each Frontier v4 task hands the model a small repository: a module that is supposed to reproduce a retired program, a written specification, and the retired program itself as compiled binaries for macOS and Linux. The module is wrong in ways the issue only hints at, and the specification may have drifted from what the program really does. Hidden tests compare the repaired module with the binary, byte for byte, on inputs the model never sees. The binary is the ground truth; everything else is a claim about it. (Methodology has the full design, and Report 01 looks at Claude Fable 5.2 on the same tasks.)

What this report does not show. Neon's Antigravity sessions recorded a summary of its thinking with each model response. Those summaries are full of what it learned about each program, which would hand future models the answers, so this report shows only their structure: counts, lengths, timing and the actions they lead to. It never quotes them.

Anatomy of a solve

Most Neon runs follow the same arc, and the phases come in this order at every level:

  1. GatherOne command lists the repository, then a batch of single-file reads takes in the module, the specification, the README and the legacy notes. Median: four or five files read before any machine code.
  2. Run the programAt Low and Medium it usually feeds the reference a few hand-typed lines before reading its machine code (16 of 23 runs at each level). At Extra-high it mostly skips this step (4 of 23).
  3. Read the machine codeSymbols and strings, then a disassembly saved to scratch space and read in passes. Median: its 9th or 10th tool call, still inside the first minute. From High up it reads both the macOS and the Linux build in 18 of 23 runs.
  4. ThinkThe longest stretch of reasoning in the run, holding about half of its thinking. At Low it leads straight into the rewrite; from High up it usually sends Neon back to the machine code for more.
  5. ExperimentScripted batches of inputs through the reference, often before the module is touched at all (14 to 17 runs of 23 at every level).
  6. Write once, check, finishThe rewrite, usually in one or two edits with the edit tool, then a randomized comparison with the reference and a final look at the diff. Every one of the 92 patches changes only the module.

The chart below lays one task, blendcore, end to end at each level, with Claude Fable 5.2's Low and Extra-high runs on the same task underneath for contrast. Each tick is a tool call, coloured by what it did; each bar above the line is the thinking that came before it.

Reading code and binaryProbing the binaryEditing the moduleTestingWrap-up
One task, blendcore, solved by Gemini 4 Neon at each effort level and by Claude Fable 5.2 at Low and Extra-high: each tick is a tool call, each bar the thinking before it0%25%50%75%100%share of the run, start to finishNeon, Low3.4 min, 15 calls3.4 minutes on the wall clock0.2 min: reading code and binary, 24 thinking tokens before it0.2 min: reading code and binary0.2 min: reading code and binary, 24 thinking tokens before it0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.4 min: probing the binary, 113 thinking tokens before it0.4 min: probing the binary0.6 min: reading code and binary, 23 thinking tokens before it0.6 min: reading code and binary0.8 min: reading code and binary, 31 thinking tokens before it0.8 min: reading code and binary0.9 min: probing the binary, 72 thinking tokens before it0.9 min: probing the binary1.0 min: reading code and binary, 97 thinking tokens before it1.0 min: reading code and binary1.1 min: reading code and binary, 47 thinking tokens before it1.1 min: reading code and binary1.3 min: reading code and binary, 62 thinking tokens before it1.3 min: reading code and binary1.4 min: reading code and binary, 76 thinking tokens before it1.4 min: reading code and binary2.7 min: editing the module, 6,239 thinking tokens before it2.7 min: editing the module3.0 min: testing, 36 thinking tokens before it3.0 min: testing3.1 min: wrap-up, 53 thinking tokens before it3.1 min: wrap-upNeon, Medium4.7 min, 17 calls4.7 minutes on the wall clock0.2 min: reading code and binary, 28 thinking tokens before it0.2 min: reading code and binary0.3 min: reading code and binary, 35 thinking tokens before it0.3 min: reading code and binary0.3 min: reading code and binary0.3 min: reading code and binary0.3 min: reading code and binary0.3 min: reading code and binary0.5 min: testing, 109 thinking tokens before it0.5 min: testing0.5 min: probing the binary, 19 thinking tokens before it0.5 min: probing the binary0.7 min: probing the binary, 74 thinking tokens before it0.7 min: probing the binary0.8 min: reading code and binary, 80 thinking tokens before it0.8 min: reading code and binary0.9 min: reading code and binary, 76 thinking tokens before it0.9 min: reading code and binary1.1 min: reading code and binary, 71 thinking tokens before it1.1 min: reading code and binary1.1 min: reading code and binary3.1 min: reading code and binary, 14,118 thinking tokens before it3.1 min: reading code and binary3.7 min: editing the module, 2,472 thinking tokens before it3.7 min: editing the module4.1 min: reading code and binary, 273 thinking tokens before it4.1 min: reading code and binary4.3 min: testing, 85 thinking tokens before it4.3 min: testingNeon, High6.3 min, 23 calls6.3 minutes on the wall clock0.1 min: reading code and binary, 35 thinking tokens before it0.1 min: reading code and binary0.2 min: reading code and binary, 34 thinking tokens before it0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.3 min: reading code and binary, 121 thinking tokens before it0.3 min: reading code and binary0.4 min: reading code and binary, 46 thinking tokens before it0.4 min: reading code and binary0.6 min: reading code and binary, 97 thinking tokens before it0.6 min: reading code and binary0.6 min: reading code and binary0.7 min: reading code and binary, 61 thinking tokens before it0.7 min: reading code and binary2.8 min: reading code and binary, 18,626 thinking tokens before it2.8 min: reading code and binary2.9 min: reading code and binary, 62 thinking tokens before it2.9 min: reading code and binary3.0 min: reading code and binary, 69 thinking tokens before it3.0 min: reading code and binary3.7 min: reading code and binary, 3,690 thinking tokens before it3.7 min: reading code and binary3.9 min: reading code and binary, 1,445 thinking tokens before it3.9 min: reading code and binary4.0 min: reading code and binary, 583 thinking tokens before it4.0 min: reading code and binary4.7 min: editing the module, 2,833 thinking tokens before it4.7 min: editing the module5.4 min: testing, 362 thinking tokens before it5.4 min: testing5.5 min: testing, 211 thinking tokens before it5.5 min: testing5.7 min: testing, 116 thinking tokens before it5.7 min: testing5.8 min: wrap-up, 558 thinking tokens before it5.8 min: wrap-up5.9 min: testing, 47 thinking tokens before it5.9 min: testingNeon, Extra-high10.3 min, 32 calls10.3 minutes on the wall clock0.1 min: reading code and binary, 55 thinking tokens before it0.1 min: reading code and binary0.2 min: reading code and binary, 55 thinking tokens before it0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.2 min: reading code and binary0.3 min: reading code and binary, 263 thinking tokens before it0.3 min: reading code and binary0.4 min: reading code and binary, 70 thinking tokens before it0.4 min: reading code and binary0.5 min: reading code and binary, 126 thinking tokens before it0.5 min: reading code and binary0.7 min: reading code and binary, 177 thinking tokens before it0.7 min: reading code and binary0.8 min: reading code and binary, 125 thinking tokens before it0.8 min: reading code and binary0.8 min: reading code and binary, 57 thinking tokens before it0.8 min: reading code and binary0.8 min: reading code and binary2.9 min: reading code and binary, 17,273 thinking tokens before it2.9 min: reading code and binary3.4 min: reading code and binary, 3,009 thinking tokens before it3.4 min: reading code and binary3.5 min: reading code and binary, 109 thinking tokens before it3.5 min: reading code and binary3.6 min: reading code and binary, 39 thinking tokens before it3.6 min: reading code and binary3.7 min: reading code and binary, 77 thinking tokens before it3.7 min: reading code and binary3.8 min: reading code and binary, 51 thinking tokens before it3.8 min: reading code and binary3.9 min: reading code and binary, 41 thinking tokens before it3.9 min: reading code and binary4.0 min: reading code and binary, 186 thinking tokens before it4.0 min: reading code and binary5.0 min: testing, 5,792 thinking tokens before it5.0 min: testing5.4 min: reading code and binary, 142 thinking tokens before it5.4 min: reading code and binary5.7 min: testing, 108 thinking tokens before it5.7 min: testing6.2 min: testing, 2,409 thinking tokens before it6.2 min: testing7.1 min: testing, 4,407 thinking tokens before it7.1 min: testing8.0 min: editing the module, 3,914 thinking tokens before it8.0 min: editing the module8.7 min: testing, 324 thinking tokens before it8.7 min: testing9.2 min: wrap-up, 1,555 thinking tokens before it9.2 min: wrap-up9.4 min: editing the module, 69 thinking tokens before it9.4 min: editing the module9.7 min: testing, 151 thinking tokens before it9.7 min: testing9.8 min: wrap-up, 64 thinking tokens before it9.8 min: wrap-upFable 5.2, Low4.3 min, 9 calls4.3 minutes on the wall clock0.0 min: reading code and binary0.1 min: reading code and binary, about 257 thinking tokens before it0.1 min: reading code and binary0.2 min: reading code and binary, about 295 thinking tokens before it0.2 min: reading code and binary0.2 min: reading code and binary, about 256 thinking tokens before it0.2 min: reading code and binary0.5 min: reading code and binary, about 1,274 thinking tokens before it0.5 min: reading code and binary1.8 min: editing the module, about 4,049 thinking tokens before it1.8 min: editing the module2.1 min: testing, about 376 thinking tokens before it2.1 min: testing2.4 min: testing, about 638 thinking tokens before it2.4 min: testing4.0 min: editing the module, about 588 thinking tokens before it4.0 min: editing the moduleFable 5.2, Extra-high10.7 min, 22 calls10.7 minutes on the wall clock0.1 min: reading code and binary, about 240 thinking tokens before it0.1 min: reading code and binary0.1 min: reading code and binary0.2 min: probing the binary, about 457 thinking tokens before it0.2 min: probing the binary0.3 min: reading code and binary, about 267 thinking tokens before it0.3 min: reading code and binary0.3 min: reading code and binary, about 254 thinking tokens before it0.3 min: reading code and binary0.4 min: reading code and binary, about 257 thinking tokens before it0.4 min: reading code and binary0.4 min: reading code and binary0.4 min: reading code and binary4.3 min: probing the binary, about 19,873 thinking tokens before it4.3 min: probing the binary4.9 min: editing the module, about 799 thinking tokens before it4.9 min: editing the module5.5 min: editing the module, about 637 thinking tokens before it5.5 min: editing the module5.9 min: testing, about 416 thinking tokens before it5.9 min: testing6.4 min: testing, about 841 thinking tokens before it6.4 min: testing6.9 min: editing the module, about 786 thinking tokens before it6.9 min: editing the module7.1 min: testing, about 488 thinking tokens before it7.1 min: testing7.2 min: reading code and binary, about 961 thinking tokens before it7.2 min: reading code and binary7.4 min: probing the binary, about 534 thinking tokens before it7.4 min: probing the binary7.9 min: wrap-up, about 503 thinking tokens before it7.9 min: wrap-up8.6 min: testing9.0 min: editing the module, about 1,194 thinking tokens before it9.0 min: editing the module10.2 min: testing, about 648 thinking tokens before it10.2 min: testing10.3 min: wrap-up

Ticks: tool calls, coloured by activity. Bars above the line: thinking tokens spent before that call (height is the square root, so small bursts stay visible). The short bar under each label is the run's wall-clock length, on one scale for all six runs.

Every Neon row opens with a run of reading ticks and almost no thinking: it collects first. The tallest bar then moves earlier as effort rises, from about four fifths of the way through the Low run, right before the one edit, to just over a quarter of the way through at Extra-high, where it is followed by more reading and experiments. Fable 5.2's rows reach the machine code within seconds and, at Extra-high, fill the second half with probing and checking.

Walkthrough: three minutes at Low

Blendcore at Low took Neon 3.4 minutes, 15 tool calls and 6,926 thinking tokens. The run, step by step, with times from the start:

Across all 23 Low tasks Neon averages 27 tool calls, 3 edits and 8 test runs, more than at any higher level, with the first edit at a median of 2.7 minutes, 72% of the way through the run. Its biggest burst of thinking follows a binary inspection in 19 of 23 runs and comes right before an edit in 13.

Walkthrough: ten minutes at Extra-high

The same task at Extra-high took 10.3 minutes, 32 tool calls and 40,750 thinking tokens, nearly six times the Low run. The rewrite it shipped is about the same size. Here is where the extra work went:

Most of the extra six minutes went into understanding the program before writing anything: a second binary, a second look at the first, and experiments against the reference. The checking after the rewrite is about as long as at Low. That is the Extra-high pattern across all 23 tasks: the first edit still lands about 75% of the way in, and the module gets a mean of 1.3 edits.

Where the extra thinking goes

Neon's thinking per task grows 4.4 times from Low to Extra-high and its time doubles, from 5.1 to 10.0 minutes, while thinking grows from 45% of output to 71%. To see what that thinking is for, each response's thinking is credited to the action that followed it: reading, probing the binary by hand, editing the module, testing, or wrapping up. Report 01 did the same for Fable 5.2, which is shown underneath.

Reading code and binaryProbing the binaryEditing the moduleTestingWrap-up
Share of thinking tokens by the activity they preceded, by effort levelGemini 4 NeonLown=23Gemini 4 Neon, Low: 32% of thinking before reading code and binary32%Gemini 4 Neon, Low: 4% of thinking before probing the binaryGemini 4 Neon, Low: 29% of thinking before editing the module29%Gemini 4 Neon, Low: 34% of thinking before testing34%Gemini 4 Neon, Low: 1% of thinking before wrap-upMediumn=23Gemini 4 Neon, Medium: 42% of thinking before reading code and binary42%Gemini 4 Neon, Medium: 5% of thinking before probing the binaryGemini 4 Neon, Medium: 23% of thinking before editing the module23%Gemini 4 Neon, Medium: 31% of thinking before testing31%Gemini 4 Neon, Medium: 1% of thinking before wrap-upHighn=23Gemini 4 Neon, High: 64% of thinking before reading code and binary64%Gemini 4 Neon, High: 4% of thinking before probing the binaryGemini 4 Neon, High: 18% of thinking before editing the module18%Gemini 4 Neon, High: 13% of thinking before testing13%Gemini 4 Neon, High: 1% of thinking before wrap-upExtra-highn=23Gemini 4 Neon, Extra-high: 71% of thinking before reading code and binary71%Gemini 4 Neon, Extra-high: 3% of thinking before probing the binaryGemini 4 Neon, Extra-high: 10% of thinking before editing the module10%Gemini 4 Neon, Extra-high: 15% of thinking before testing15%Gemini 4 Neon, Extra-high: 1% of thinking before wrap-upClaude Fable 5.2Lown=23Claude Fable 5.2, Low: 35% of thinking before reading code and binary35%Claude Fable 5.2, Low: 1% of thinking before probing the binaryClaude Fable 5.2, Low: 44% of thinking before editing the module44%Claude Fable 5.2, Low: 19% of thinking before testing19%Claude Fable 5.2, Low: 1% of thinking before wrap-upMediumn=23Claude Fable 5.2, Medium: 47% of thinking before reading code and binary47%Claude Fable 5.2, Medium: 1% of thinking before probing the binaryClaude Fable 5.2, Medium: 44% of thinking before editing the module44%Claude Fable 5.2, Medium: 6% of thinking before testingClaude Fable 5.2, Medium: 2% of thinking before wrap-upHighn=23Claude Fable 5.2, High: 40% of thinking before reading code and binary40%Claude Fable 5.2, High: 12% of thinking before probing the binary12%Claude Fable 5.2, High: 26% of thinking before editing the module26%Claude Fable 5.2, High: 20% of thinking before testing20%Claude Fable 5.2, High: 3% of thinking before wrap-upExtra-highn=23Claude Fable 5.2, Extra-high: 30% of thinking before reading code and binary30%Claude Fable 5.2, Extra-high: 27% of thinking before probing the binary27%Claude Fable 5.2, Extra-high: 19% of thinking before editing the module19%Claude Fable 5.2, Extra-high: 20% of thinking before testing20%Claude Fable 5.2, Extra-high: 3% of thinking before wrap-up

The two models move in opposite directions. At Low, Neon's thinking is spread across reading (32%), editing (29%) and testing (34%). By Extra-high, 71% of it comes before reading the repository or the binary, and testing falls to 15%. Fable 5.2 goes the other way: its share before reading falls from 47% at Medium to 30% at Extra-high while probing and testing grow. Neon's biggest burst shows the same drift. At Low it comes right before an edit in 13 of 23 runs; at Extra-high it never does, and in 21 of 23 runs it comes right before more reading.

Over the wall clock, Neon thinks late. Only 4% to 10% of its thinking falls in the first fifth of a run, where it is gathering files and running the program, against 10% to 24% for Fable 5.2 from Low to Extra-high, and 61% to 80% lands between 40% and 80% of the way through. At higher effort the peak moves earlier, toward the reading of the machine code.

Gemini 4 Neon: share of each level's thinking in each tenth of the run
Gemini 4 Neon: share of a level's thinking in each tenth of the run0-10%10-20%20-30%30-40%40-50%50-60%60-70%70-80%80-90%90-100%LowLow: 3% of thinking in the 0 to 10% slice of the run3%Low: 7% of thinking in the 10 to 20% slice of the run7%Low: 8% of thinking in the 20 to 30% slice of the run8%Low: 9% of thinking in the 30 to 40% slice of the run9%Low: 8% of thinking in the 40 to 50% slice of the run8%Low: 19% of thinking in the 50 to 60% slice of the run19%Low: 17% of thinking in the 60 to 70% slice of the run17%Low: 17% of thinking in the 70 to 80% slice of the run17%Low: 9% of thinking in the 80 to 90% slice of the run9%Low: 2% of thinking in the 90 to 100% slice of the run2%MediumMedium: 2% of thinking in the 0 to 10% slice of the run2%Medium: 3% of thinking in the 10 to 20% slice of the run3%Medium: 3% of thinking in the 20 to 30% slice of the run3%Medium: 5% of thinking in the 30 to 40% slice of the run5%Medium: 8% of thinking in the 40 to 50% slice of the run8%Medium: 39% of thinking in the 50 to 60% slice of the run39%Medium: 20% of thinking in the 60 to 70% slice of the run20%Medium: 13% of thinking in the 70 to 80% slice of the run13%Medium: 5% of thinking in the 80 to 90% slice of the run5%Medium: 2% of thinking in the 90 to 100% slice of the run2%HighHigh: 2% of thinking in the 0 to 10% slice of the run2%High: 2% of thinking in the 10 to 20% slice of the run2%High: 4% of thinking in the 20 to 30% slice of the run4%High: 11% of thinking in the 30 to 40% slice of the run11%High: 31% of thinking in the 40 to 50% slice of the run31%High: 18% of thinking in the 50 to 60% slice of the run18%High: 17% of thinking in the 60 to 70% slice of the run17%High: 8% of thinking in the 70 to 80% slice of the run8%High: 5% of thinking in the 80 to 90% slice of the run5%High: 2% of thinking in the 90 to 100% slice of the run2%Extra-highExtra-high: 2% of thinking in the 0 to 10% slice of the run2%Extra-high: 2% of thinking in the 10 to 20% slice of the run2%Extra-high: 7% of thinking in the 20 to 30% slice of the run7%Extra-high: 21% of thinking in the 30 to 40% slice of the run21%Extra-high: 24% of thinking in the 40 to 50% slice of the run24%Extra-high: 18% of thinking in the 50 to 60% slice of the run18%Extra-high: 13% of thinking in the 60 to 70% slice of the run13%Extra-high: 9% of thinking in the 70 to 80% slice of the run9%Extra-high: 3% of thinking in the 80 to 90% slice of the run3%Extra-high: 3% of thinking in the 90 to 100% slice of the run3%tenth of the run, by wall-clock time

The extra reading is concrete. From High up, Neon reads the Linux build as well as the macOS build in 18 of 23 runs, against 4 or 5 for Fable 5.2. It also runs scripted experiments before its first edit at every level, where Fable 5.2 mostly does so only at Extra-high and Max. And it stops typing input into the reference by hand before reading the machine code: 16 runs at Low, 4 at Extra-high, while Fable 5.2 climbs from 1 to 14.

Runs thatModelLowMediumHighExtra-high
Ran the reference before reading its machine codeNeon16/2316/2311/234/23
Fable 5.21/232/2310/2314/23
Ran scripted experiments before the first editNeon17/2314/2316/2316/23
Fable 5.24/233/239/2316/23
Read both the macOS and the Linux binaryNeon0/231/2318/2318/23
Fable 5.20/230/234/235/23
Read the machine code before the first editNeon21/2323/2323/2323/23
Fable 5.223/2323/2323/2323/23

Machine code read before the first edit: the two Low runs that never read it, queuecore and schedcore, solved their tasks anyway by running the reference instead.

What does not grow is the checking after the rewrite. Counted from the commands each run executed, none of Neon's 92 runs used any of the four extra checks that mark Fable 5.2's high-effort runs: mutation testing (planting deliberate bugs to see whether the checks catch them), running the reference's Linux build in a container, a throughput check, or a second Python version. Its test runs per task fall from 7.7 at Low to 3.3 at High and 4.6 at Extra-high.

CheckModelLowMediumHighExtra-high
Mutation testingNeon0/230/230/230/23
Fable 5.20/230/233/2312/23
Linux build in a containerNeon0/230/230/230/23
Fable 5.20/230/235/2313/23
Throughput checkNeon0/230/230/230/23
Fable 5.20/230/232/233/23
Second Python versionNeon0/230/230/230/23
Fable 5.20/230/230/231/23

Its thinking summaries read differently too. Keyword counts over the summaries (the same rough rules Report 01 used) find Neon hedging in almost every run at every level: words such as likely, probably and hypothesis appear in 20 to 23 runs of 23, where Fable 5.2 used them in about half its runs until Extra-high. What grows for Fable 5.2 and barely moves for Neon is talk about the job around the code: doubting the specification (7 or 8 Neon runs at every level, against Fable 5.2's climb from 3 to 19), keeping the change small (1 to 11, against 4 to 21) and cleaning up scratch files (0 to 8, against 2 to 21).

Runs whose thinking mentionsModelLowMediumHighExtra-high
Machine code and disassemblyNeon21/2323/2323/2323/23
Fable 5.223/2323/2323/2323/23
Hypotheses (likely, probably, I suspect)Neon20/2323/2323/2323/23
Fable 5.211/2312/2312/2321/23
Checking itself (verify, confirm)Neon22/2323/2323/2323/23
Fable 5.223/2318/2323/2323/23
Edge casesNeon20/2323/2323/2323/23
Fable 5.222/2322/2323/2323/23
Comparing against the referenceNeon20/2320/2322/2322/23
Fable 5.223/2323/2323/2323/23
Probing the programNeon16/2313/2312/2315/23
Fable 5.220/2318/2316/2318/23
Distrusting the specificationNeon7/237/238/238/23
Fable 5.23/234/2310/2319/23
Keeping the change smallNeon1/237/235/2311/23
Fable 5.24/238/2313/2321/23
Cleaning upNeon0/234/233/238/23
Fable 5.22/234/2317/2321/23

All of Neon's per-level means in one place:

Per task, meanLowMediumHighExtra-high
Solved (all hidden tests pass)17/2323/2323/2323/23
Minutes5.15.96.410.1
Output tokens20.1k28.4k38.2k56.9k
of which thinking9.1k (45%)15.6k (55%)25.3k (66%)40.3k (71%)
Model responses with a thinking summary14121218
Largest single burst, share of thinking45%60%53%48%
Tool calls27.523.824.531.0
Binary inspection calls8.06.97.69.5
Calls that run the reference by hand1.51.61.61.0
Test runs7.75.13.34.6
Edits to the module3.01.71.31.3
Edit-then-test cycles1.51.01.01.0
First binary inspection, median tool call91099
First edit, median minute2.73.64.57.5
Share of the run before the first edit, median72%75%75%75%

The six misses at Low

Neon left hidden tests failing in 6 of its 23 Low runs (cellarcore, hedgecore, lodgecore, matchcore, quotacore and tallycore, with functional scores from 0.75 to 0.92) and in none above Low. All six read the machine code before editing, so the misses are not a black-box habit. They differ from the 17 solved Low runs in how the thinking was spent:

17 solved Low runs

median per run
  • Biggest burst: 56% of the run's thinking.
  • About 6,900 thinking tokens in 3.6 minutes.
  • 3 test runs and 2 edits.

6 Low runs with failing tests

median per run
  • Biggest burst: 25% of the run's thinking.
  • About 7,400 thinking tokens in 4.2 minutes.
  • 7 test runs and 3.5 edits.

The misses used about as much thinking as the solves but in smaller pieces, with more rounds of edit and test, as if Neon were patching by trial instead of reasoning its way to the rewrite first. At Medium it solved all six. For four of them (hedgecore, matchcore, quotacore and tallycore) the Medium run spent two to three times the thinking of the Low run, about 14,000 to 15,000 tokens against 4,900 to 7,500, in under two minutes more. The other two, cellarcore and lodgecore, were already Neon's longest Low runs, at 12.2 and 9.0 minutes. With six cases this is a pattern worth noting, not proof of a cause; harder tasks may invite both scattered thinking and failure.

Against Claude Fable 5.2

Fable 5.2 is the natural comparison: it solved all 115 of its runs, its streams are complete, and Report 01 measured them with the same rules used here. Neon has no Max level, so the comparison runs from Low to Extra-high.

Gemini 4 NeonClaude Fable 5.2
Thinking tokens and minutes per task by effort levelThinking tokensmean per task, log scale, whiskers ±1 standard error3k5k10k20k50kLowMediumHighXHighGemini 4 Neon, Low: 9.1k (±0.9k, n=23)Gemini 4 Neon, Medium: 15.6k (±0.8k, n=23)Gemini 4 Neon, High: 25.3k (±0.8k, n=23)Gemini 4 Neon, Extra-high: 40.3k (±1.1k, n=23)Claude Fable 5.2, Low: 7.7k (±1.1k, n=23)Claude Fable 5.2, Medium: 7.1k (±0.6k, n=23)Claude Fable 5.2, High: 18.3k (±1.3k, n=23)Claude Fable 5.2, Extra-high: 40.8k (±2.3k, n=23)Fable 5.2NeonMinutesmean per task, log scale, whiskers ±1 standard error351020LowMediumHighXHighGemini 4 Neon, Low: 5.1 min (±0.6 min, n=23)Gemini 4 Neon, Medium: 5.9 min (±0.7 min, n=23)Gemini 4 Neon, High: 6.4 min (±0.2 min, n=23)Gemini 4 Neon, Extra-high: 10.1 min (±0.2 min, n=23)Claude Fable 5.2, Low: 4.6 min (±0.6 min, n=23)Claude Fable 5.2, Medium: 4.3 min (±0.5 min, n=23)Claude Fable 5.2, High: 7.8 min (±0.8 min, n=23)Claude Fable 5.2, Extra-high: 14.2 min (±0.8 min, n=23)Fable 5.2Neon

Gemini 4 Neon

Antigravity CLI · 86 of 92 solved
  • Gathers files and runs the program first; first machine-code read at a median of its 9th or 10th call, 0.6 to 0.8 minutes in.
  • About 75% of the run passes before the first edit, at every level.
  • Extra effort goes to reading: a second binary, a second look at the first, experiments before writing.
  • Patches change only the module and grow with effort, from a median of 51 added lines at Low to 83 to 86 from High up, with comments on 1% to 2% of them.

Claude Fable 5.2

Claude Code · 115 of 115 solved
  • Goes straight to the machine code; first read at a median of its 2nd to 5th call, 0.1 to 0.3 minutes in.
  • The first edit comes earlier as effort rises: 60% of the way through at Low, 50% at Extra-high.
  • Extra effort goes to proving the rewrite: mutation testing, the Linux build in a container, throughput.
  • Patches change only the module and stay at about 100 added lines at every level, with comments on 12% to 19% of them.

At the two ends of the dial the two are close on thinking: 9.1k against 7.7k tokens a task at Low, and about 40k each at Extra-high, with thinking at 71% and 66% of output there. In between Neon thinks more, twice as much at Medium (15.6k against 7.1k). Neon is a little slower at Low and Medium and faster at High and Extra-high (10.0 minutes against 14.0 at Extra-high). The difference is not how much they think but what for. Neon front-loads understanding and checks lightly after writing; Fable 5.2 reads quickly, writes earlier and keeps checking, more so as effort rises.

Fable 5.2 also never left a hidden test failing, at any level. Neon's six misses all came at its cheapest setting, where its thinking was most scattered and its test runs were at their most numerous, which suggests that at Low the trial-and-error loop is standing in for reasoning it did not do up front.

Per taskModelLowMediumHighExtra-high
Thinking tokens, meanNeon9.1k15.6k25.3k40.3k
Fable 5.27.7k7.1k18.3k40.8k
Thinking share of outputNeon45%55%66%71%
Fable 5.248%47%59%66%
Minutes, meanNeon5.15.96.410.1
Fable 5.24.64.37.814.2
Tool calls, meanNeon27.523.824.531.0
Fable 5.210.710.316.427.0
First binary inspection, median tool callNeon91099
Fable 5.22245
Share of the run before the first edit, medianNeon72%75%75%75%
Fable 5.260%58%53%50%
Test runs, meanNeon7.75.13.34.6
Fable 5.23.01.33.06.5
Edits to the module, meanNeon3.01.71.31.3
Fable 5.21.81.82.44.3

The two models ran in different harnesses with different tools. Antigravity's read_file reads one file per call, so Neon's tool-call counts and the call number of its first machine-code read run higher than Claude Code's for the same work; compare the minutes instead.

What it adds up to

  1. It gathers before it thinks. Neon reads the repository's files and, at lower effort, runs the program before it disassembles anything, and only 4% to 10% of its thinking falls in the first fifth of a run. It still reaches the machine code inside the first minute, in 90 of 92 runs before its first edit.
  2. It thinks late and writes once. About three quarters of every run passes before the module is touched, at every effort level, and from Medium up the module gets one or two edits.
  3. Effort buys understanding, not proof. From Low to Extra-high the share of thinking spent before reading rises from 32% to 71%, the Linux build joins the macOS build, and experiments move ahead of the rewrite. The checking after the rewrite does not grow, and none of the 92 runs used the heavier checks Fable 5.2 reaches for.
  4. Low is where it slips. All six of its failures came at Low, in runs whose thinking was split into small pieces and spent on rounds of patching. One level up, the same tasks got two to three times the thinking and were solved.

Methods and caveats

Runs. Gemini 4 Neon (gemini-4-neon) in Antigravity CLI 1.5.12 on a Google AI Pro subscription, at four effort levels (Neon has no Max), one run per task per level, from November 20 to 21, 2026, from bench/2026-11-20-gemini4-neon-frontier-v4. Neon ran through a custom Antigravity agent with file and shell tools only and no web access. The AI Pro quota refused 40 attempts during the sweep, four of them at Extra-high after the run had started work; every refused attempt was discarded and the task rerun from scratch, so each published run is one uninterrupted session. Claude Fable 5.2's runs are the ones in Report 01 (Claude Code 2.1.321, November 9 to 11).

Sources. For each of Neon's 92 runs: the Antigravity event stream (tool calls and per-response token counts), the tool-call log with start and end times, the Antigravity session database (for the thinking summaries' count, length and keywords) and the final patch. Fable 5.2's figures are Report 01's published data, plus three order measures added here (running the reference before reading its machine code, scripted experiments before the first edit, reading both binaries), computed from its raw streams with the same code.

Activities. Each Antigravity tool is mapped onto Report 01's vocabulary (run_command as a shell command, read_file as a read, edit_file and write_file as edits) and every call goes through Report 01's classification rules unchanged. The same caveats apply: commands that do two things get one label, scripts are counted as tests whether they send input to the reference, the module or both, and the shares are approximate.

Thinking. Antigravity reports thinking tokens for every model response, so no estimate is needed: each response's thinking goes to the first tool call it made, and a response with no tool call is the closing report. Its output tokens include thinking, as Claude Code's do. Fable 5.2's per-activity thinking in Report 01 is a rescaled estimate, which makes the split chart a close but not exact comparison. Thinking summaries are summaries, not the raw reasoning; theme counts are rough keyword matches.

Scores. This report is about how Neon works, not how it ranks. Neon's combined score runs from 84.32 at Low to 87.87 at High (87.48 at Medium, 87.70 at Extra high), with Code quality judged by Muse Spark 1.3 and Grok 4.6 (round v3.32). Two codeccore runs, one at Low and one at Extra high, have no published Code quality score: Grok's follow-up check returned malformed output twice on each, and the frozen protocol leaves such a run unpublished, so those two levels are judged on 22 runs. Both runs are included in this report's trace figures. Fable 5.2 was judged by a different panel (Muse Spark 1.3 and GPT-6.1 Sol), so the two models' Code quality scores are not a like-for-like comparison. Full results are on the Gemini 4 Neon benchmark page.

Data. summary.json (every figure on this page) and runs-traces.csv (one row per Neon run). They are built by scripts/build_report02_data.py from the private run records and drawn by scripts/build_report02_charts.py. Neither file contains thinking text or task content.

← All reports