September 22, 2026, corrected September 23 · 2,641 queries · VulcanBench Verdict v1
Jev 1.13.0 on VulcanBench Verdict v1
TypeSafe AI's Jev is a System One model: it does not generate text, it returns a typed decision, either one option from a fixed set, a score on an ordered scale, or the probability that a statement is true, in a single parallel pass. It cannot write code, so it cannot take VulcanBench Frontier v4. It can judge code, which is what this suite asks it to do. Every headline answer here was produced by running tests, not by asking another model.
Correction, September 23, 2026. The first version of this report scored yes or no questions at a 0.5 cutoff. Jev's probability that a patch passes never exceeds 0.42, so that cutoff sat outside its output range and no item could be answered "passes". Those figures were an artifact of the decision rule, not of the model. This version reports the ranking, which needs no cutoff, alongside accuracy at a cutoff fitted on the development split. Two claims did not survive and are withdrawn; see what changed.
How the test works
An AI agent fixes a bug
Frontier coding agents wrote 745 distinct fixes for 23 engineering tasks during our Frontier v4 runs.
The fix is run against hidden tests
Each task has a test suite the agent never saw. It records whether the fix passes every test, passes some, or breaks something that used to work.
Jev reviews the fix
Jev gets the bug report and the fix, never the test results, and answers one question with a probability, such as "does this fix pass every test?"
We score the answer
Jev's answer is compared with what the tests showed, and with the score for always giving the most common answer.
Every fix yields three questions about its test results, plus a separate question comparing two fixes for the same task on how well they are written, which is checked against AI reviewers rather than tests. 2,163 questions in all. For yes or no questions, the probability Jev needs to count as "yes" is chosen on a separate set of fixes that are never scored, so Jev is judged at its best cutoff without seeing the answers.
The four questions
The fixes come from 23 behavioral-reconstruction tasks, where each agent rebuilt a retired program to match its real behaviour. Jev sees one fix, or one pair of fixes, per question.
Does this patch pass?
611 test items
A yes or no question with a probability attached. The reference is the hidden test suite: did every target test pass, with nothing regressing.
Does it break something?
611 test items
The same patches, asked whether a test that passed before the patch now fails. Only 36 of them really regress.
Which of four outcomes?
611 test items
All pass, partial fix, regression or no progress. The same information as the first two questions, asked as one choice.
Which patch is better written?
311 test items
Two patches for one task. The reference is the Muse Spark 1.3 and Grok 4.6 code-quality panel, which is opinion, so this family is reported apart and excluded from the overall score.
A fifth family, picking which file the fix belongs in, is kept only as a wiring probe. Jev answered 19 of 19 correctly, and then we checked the items: every issue names the answer file's stem in its own text, so the question is string matching. It supports no claim about ability. Items are split by task, so no task appears in both the development and test splits, and the test split is what is published here.
Results
One query per item, model pinned to jev-1.13.0 rather than the jev-latest alias, because the alias moves and calibration is per version. Jev exposes no effort, temperature or seed setting, so it has a single column. The floor is the score for always answering a family's most common label.
| Question | Items | Floor | Jev, fitted cutoff | AUROC | Brier | ECE | At a 0.5 cutoff† |
|---|---|---|---|---|---|---|---|
| Does this patch pass every hidden test? | 611 | 62.7% | 62.8% | 0.690 | 0.383 | 0.409 | 37.3% |
| Does it break a test that passed before? | 611 | 94.1% | 93.1% | 0.652 | 0.284 | 0.268 | 33.4% |
| Which of four outcomes did the tests report? | 611 | 62.7% | 49.8%‡ | 0.651 | 0.650 | 0.186 | 49.8% |
| Overall, ground truth only | 1,852 | 72.4% | 68.9% | 0.434 | 0.278 | 40.8% | |
| Which of two patches is more maintainable?§ | 311 | 57.9% | 89.7%‡ | 0.962 | 0.162 | 0.082 | 89.7% |
†The last column is the withdrawn number. It is accuracy when a yes needs a stated probability above 0.5, which Jev never produces on these items. It is kept in the table so the correction is visible rather than quietly removed. ‡ marks the two families decided by top-ranked option, where no cutoff applies and both columns are the same number.
§The style row is not ground truth. Its reference is agreement with the Muse Spark 1.3 and Grok 4.6 code-quality panel, so it is excluded from the overall row and from any claim about correctness. The overall row covers the three patch families plus the 19 localization probe items.
Reading the numbers. The fitted cutoffs, 0.17 for the pass question and 0.72 for the regression question, were chosen on the development split and never on the published items. AUROC is the chance that the model ranks a true item above a false one, so it does not depend on where the cutoff sits; 0.5 is chance. Brier and ECE score the probabilities as stated, which is where Jev does badly: Brier is the squared error of the stated probability, and ECE is the average gap between stated confidence and observed accuracy over ten bins. Brier is binary for the yes or no families and multiclass for the choice families, so compare it within a row, never down the column. The three patch families ask different questions about the same 745 patches, so their errors are correlated.
Operations. 2,641 queries across both splits, zero failures, zero retries, $0.447 in total, $0.15 per 1,000 decisions. Latency was 328 ms at the median and 471 ms at the 95th percentile, measured as wall clock from a machine in California and including the network round trip to a service hosted on the US West Coast. Latency is published as indicative and is not used to rank anything.
Probabilities that never reach the boundary, and a ranking that does
Across all 611 patch-verdict items, the probability Jev gives for "this patch passes" runs from 0.09 to 0.42, with a mean of 0.22. In reality 383 of those patches, 62.7% of them, pass every hidden test. There is no input in this suite for which Jev states that a patch is more likely to pass than not. The regression question is compressed the same way from the other side: the probability that a patch breaks something runs from 0.24 to 0.80 with a mean of 0.53, on a population where the true rate is 5.9%.
That is a calibration failure, and a large one. A model whose stated probabilities are meant to be believed, asked about a class that occurs 63% of the time, never assigns that class more than 0.42. The expected calibration error of 0.409 on the pass question is the largest number in this report.
The ordering underneath is a different story. Ranked by that same probability, Jev puts passing patches above failing ones well clear of chance.
| Question | AUROC |
|---|---|
| Does this patch pass every hidden test? | 0.690 |
| Does it break a test that passed before? | 0.652 |
| Which of four outcomes (macro, one against the rest) | 0.651 |
| Which of two patches is more maintainable? | 0.962 |
Ordering is not the same as usefulness. With a cutoff fitted on held-out items, accuracy on the two yes or no families lands at 62.8% and 93.1%, against floors of 62.7% and 94.1%. The signal is real, but the classes are imbalanced enough that acting on it gains nothing over answering the most common label.
The wording is not the cause. The same 134 development-split patches were asked the verdict question four ways, including an inverted form where agreeing means the opposite.
| Wording | Mean probability of passing |
|---|---|
| This patch makes every hidden target test pass | 0.220 |
| This patch is a correct and complete fix | 0.208 |
| A careful maintainer would merge this patch | 0.276 |
| This patch is broken (inverted) | 0.221 |
The four agree with each other on 96% to 100% of patches, and the inverted form mirrors the original almost exactly. TypeSafe's consistency claim holds up cleanly here. Rewording moves the mean by at most 0.06 and never lifts the distribution across 0.5, so the compression is a property of the model on this task rather than an artifact of how the question was put.
Where it works
On the 311 pairs where both code-quality judges preferred the same patch by at least 10 points out of 100, Jev picked the same patch 89.7% of the time, with an AUROC of 0.962 and a well-calibrated expected calibration error of 0.082. That is the clearest positive signal in the suite, and it survives the two obvious explanations. Picking the longer patch agrees with the judges only 44.7% of the time, so length is not the shortcut. Jev's split between the two presented positions, 183 to 128, tracks the true split of 180 to 131, so there is no position bias. Patches are ordered by content hash, not by score, so position carries no signal to find.
Judges are opinion, not ground truth, so this is not a correctness result and it does not belong on the same scale as the rows above. What it suggests is a real and narrow competence: Jev can read the surface of code, its naming, structure, comments and duplication, even though it cannot evaluate what the code does. It is also the one place in this suite where its probabilities are both discriminating and honestly scaled.
What this means
Jev cannot be dropped in as a merge gate or a continuous-integration pre-filter on the strength of its own probabilities. Read literally, it says every patch is more likely to fail than pass, which would block everything. Read as a ranking, with a cutoff fitted on your own labelled data, it comes out about level with answering "it passes" every time.
Where the judgment is about the shape of the code rather than its behaviour, the picture changes: 89.7% agreement with a calibrated judge panel, an AUROC of 0.962, at 328 ms and $0.15 per 1,000 decisions. That is roughly the cost and latency of a database query, for a judgment that currently costs a full model call.
The methodological point is one this report had to learn the hard way. A model that returns probabilities can be self-consistent, well ordered and badly scaled all at once, and which of those you see depends entirely on the metric. Accuracy at a fixed cutoff measures the scaling and hides the ordering. AUROC measures the ordering and hides the scaling. Reporting one without the other produces a confident, wrong headline, which is what the first version of this page did.
What changed in the correction
Published September 22, corrected September 23, 2026. Two claims are withdrawn.
- "Jev scores 40.8% against a 72.4% floor." That was accuracy at a 0.5 cutoff the model never crosses. At a fitted cutoff it is 68.9% against the same floor. Both numbers are in the table above and in the data file; the fitted one is the fair comparison, and Jev is still below the floor, by 3.5 points rather than 32.
- "Accuracy falls as the patch gets longer." It does not. The share of patches that pass rises with size, 54%, 89% and 90% across the three size buckets, and at a fitted cutoff Jev's accuracy in each bucket equals that bucket's majority baseline exactly. The original finding was the class balance moving, seen through a broken decision rule.
Retained, with the framing corrected: the compression of the probabilities, now reported as the calibration failure it is rather than as "it called every patch broken"; the four-wording consistency result; the style-pair result; and the localization probe.
Changed in the suite itself: the scorer now reports AUROC and the range of probabilities the model actually produced, fits yes or no cutoffs on the development split, and prints the majority baseline beside every subgroup. Everything was recomputed from the stored predictions, so the correction cost no further queries and no number moved for any other reason.
Method, limits and evidence
Items were mined from seven completed Frontier v4 sweeps: 745 distinct agent patches across 23 tasks, with contaminated runs, empty patches and duplicate patches excluded. Answers come from the verifier's own record of which target tests passed and whether anything regressed, from the gold patch for the localization probe, and from the code-quality panel for the style pairs. Scoring covers AUROC, accuracy at the top-ranked answer, accuracy at a development-fitted cutoff, the majority-label share, Brier, log loss, expected calibration error over ten bins, the range of probabilities produced, and latency and cost per 1,000 decisions. The reproduction guide gives the public arithmetic checks against results.json, and the suite code is in the VulcanBench repository.
Limits. The fitted cutoffs come from 134 development items per family; they are held out from the published split, but they are not many, and a different development sample would move them. No other model has been run on these items yet, so the reference points here are the majority floor and, for style, the judge panel. In particular, an AUROC of 0.962 on style is not yet known to be better or worse than what a frontier model would score on the same pairs. Every item comes from one suite, whose tasks skew toward binary-parity reimplementation work in Python, so this patch population is not a sample of open-source pull requests in general. Any aggregate across the three patch families should be bootstrapped by patch rather than by item, because they ask different questions about the same patches. Latency depends on where the caller sits.
Withheld. The item file embeds Frontier v4 issue text and agent patches, so it stays private and carries the suite canary; publishing it would leak the task set. This bundle contains metrics only. Results are pinned to jev-1.13.0 and to the item set built on September 22, 2026; a later model version is a new measurement, never an update of this one.