The harness is worth more than the effort knob: at the same nominal high setting, Grok 4.6 scores 94.9% inside Cursor against 73.9% through VulcanBench’s deliberately minimal reference loop, a 21.0-point gap. Through the raw API the effort curve peaks at medium and collapses; inside Cursor it rises monotonically (79.7 → 94.2 → 94.9 → 95.7). The overthinking failure mode disappears: the raw API produced ten budget-cut runs above low effort, Cursor produced zero at any level. The harness also changes which tasks are solvable. The two tasks no raw-API run has ever solved both fall inside Cursor, while a task every raw-API level solves becomes Cursor’s only recurring failure. Only at low effort does the bare-bones loop win, by 2.9 points.
| Effort | In Cursor | Bare-bones API | Delta | Cursor time/task | API time/task |
|---|---|---|---|---|---|
| low | 79.7% ±8.1 | 82.6% ±8.1 | −2.9 | 2.9 min | 4.8 min |
| medium | 94.2% ±4.4 | 87.0% ±7.2 | +7.2 | 2.9 min | 14.2 min |
| high | 94.9% ±4.4 | 73.9% ±9.4 | +21.0 | 4.2 min | 16.3 min |
| xhigh | 95.7% ±4.3 | 78.3% ±8.8 | +17.4 | 4.2 min | 15.8 min |
System A (bare-bones) is Report No. 14: the xAI API through VulcanBench’s minimal reference loop, one attempt per task, effort sent per request. System B is Cursor’s own agent via cursor-agent, three attempts per task (84/84/79/75 runs per level), effort baked into the model id (cursor-grok-4.6-low … -xhigh), non-fast ids. Both systems are graded by the same deterministic hidden tests in Docker. pass@1 is the mean per-task success rate, so uneven attempt counts do not bias it; times are medians and include provider latency.
The harness is worth more than the effort knob. The largest gap the knob produced in Report No. 14 was 13.1 points. The harness produces 21.0 at the same nominal setting, and every Cursor level from medium up beats every bare-bones level.
The overthinking failure mode disappears. Raw-API failures above low effort are dominated by runs cut off at a step or wall-clock budget (2, 4, and 4 at medium, high, xhigh). Inside Cursor there are zero budget-cut runs at any level, and the same nominal level finishes 1.6–4.9× faster.
The harness changes which tasks are solvable. pennylane-trotter-fragmented and sqlglot-canonicalize-internal-names, unsolved by any raw-API run across Reports No. 12–14, are both solved inside Cursor. In the other direction, itertools-strip-prefix, solved at every raw-API level, fails at medium, high, and xhigh inside Cursor.
Low effort is the exception that sharpens the question. The bare-bones loop wins low by 2.9 points. Whatever Cursor’s scaffold adds, it pays off where the model reasons long and costs a little where it does not.
This is a product measurement, not a weights measurement. Read it as “Grok 4.6 as Cursor ships it” against “Grok 4.6 as the API hands it to you.” The 21-point delta is far outside both columns’ uncertainty; its cause is not identifiable from outside this wall (see Caveats).
The gap could come from Cursor’s scaffold (context management, tool design, stopping rules), from how Cursor’s effort-suffixed model ids map to the API’s reasoning_effort values, or from different serving. “Cursor Grok 4.6” is Cursor’s own branding and may be a tuned or differently-hosted deployment. These explanations are indistinguishable from outside, and per-request token counts, which Cursor does not report, would settle it. Attempt counts differ between systems (3 vs 1), so per-task flips carry more weight on the Cursor side. Bare-bones costs in Report No. 14 are lower bounds: xAI reports reasoning tokens outside completion_tokens and the harness under-counted them at the time of that sweep (since fixed). Cursor-side spend is not reportable by the CLI and appears only in the account dashboard, so this study reports no dollar comparison.
VulcanBench v3: 23 tasks from real merged post-cutoff PRs (Python 9, TypeScript 4, Rust 4, Go 3, JavaScript 3). Both systems are graded by the same deterministic hidden tests in Docker; model judges disabled. System A ran 2026-08-12 (Report No. 14). System B ran 2026-08-13/14 with cursor-agent 2026.08, Cursor’s sandbox enabled, and VulcanBench setup and verification in Docker over the agent’s host workspace; about a fifth of the low and medium runs are recovered attempts re-verified in Docker after a harness fix, and all runs at every level were verified in the same environment. This is the first VulcanBench study to hold the model fixed and vary the harness; the Harness Study series continues with other model × harness combinations.
