This table was retained from the earlier methodology page. Its accompanying text described an eighteen-task reference set. It is not the current 23-task Astra sweep and should not be used as a direct comparison with that run. The original page did not identify an exact snapshot date, suite hash, or table-specific run count, so this is historical context rather than a fully specified benchmark result.
| Model (harness) | Full budget | @10 min | @30 min | Median tokens/run |
|---|---|---|---|---|
| Claude Opus 5 (Claude Code CLI) | 85.6% | 5.9% | 77.4% | 0.57M |
| GPT 5.6 Sol (Codex CLI) | 4.4% | 3.3% | 4.4% | 1.14M |
The time thresholds describe results calculated from recorded durations. They did not terminate attempts. For current comparisons, use a report that identifies the full task set, attempt counts, and execution configuration.
Earlier sweeps exposed two distinct risks. In an initial Cursor sweep, the recorded investigation reported that 46% of runs fetched their task's source pull request or fix commit, with a median patch matching the reference solution. In a separate sweep with web access blocked, 46 runs accessed their task's answer key through the filesystem, and all 46 solved.
These are historical findings retained from the prior methodology page, not estimates of contamination in current runs. They motivated separate web and filesystem controls, workspaces outside the benchmark checkout, and audits of available execution traces.
An earlier audit also flagged four clean runs because it treated a tool call's start as completed access. Audits must correlate requests with their outcomes: a rejected request does not establish that protected content was retrieved.
Integrity flags annotate the evidence; they do not automatically rescore or exclude a run. A report must state how flagged results were treated.
The original methodology described Codex workspace permissions, Claude Code permission modes, Cursor's sandbox, a custom Seatbelt/Landlock profile for Grok Build, and ZCode without a kernel sandbox. These details can change with product and harness versions. Each run's recorded configuration is the relevant evidence.
Placing a workspace outside the benchmark checkout avoids exposing reference material in its ancestor directories. It does not by itself prevent access through absolute paths. Filesystem permissions, web controls, and trace audits must be considered together.
API sandbox runs and optional agent-in-container runs have different boundaries from agents running on the host. Reports should identify agent execution and verification environments separately.
