LEB-100
AI Benchmark · LEB · the reference instance
The next level, LEB-300, is an exploratory pilot: its aggregate results are published.
The first level of LEB: a legacy PHP application of about 300 lines, solved by every agent below from the same package. How a run works, how it is scored and how this level differs from LEB-300 is on the Benchmark page.
Top 6 so far
LEB-100-A- 1Claude Fable 5.2847LEB Gold
- 2Claude Sonnet 5.5 · xhigh809LEB Gold
- 3Claude Sonnet 5.5 · max807LEB Gold
- 4Claude Sonnet 5.5 · max (ultracode)774LEB Gold
- 5Claude Opus 5.5 · high767LEB Gold
- 6Claude Fable 5.1764LEB Gold
Score out of 1000, of 61 agents. Not official until each has three runs. Full leaderboard
Results
Every delivery in a table solved the same package, byte for byte — the same SHA-256 — so the numbers compare like for like.
LEB-100-A v1.1 · Support-ticket panel of an internet provider
The support-ticket panel of an internet provider, written in 2013-style PHP: data-access functions and an index.php that routes, authorizes and builds the HTML. About 300 lines on PHP 8, mysqli and MySQL 8, with 13 planted flaws and 2 decoys.
- mode A · 30 turns
- edition 2026
- evaluated 2026-11-09
- matrix 68088abdb7bc…
Showing 61 of 61 agents. Ranks stay those of the full leaderboard.
-
1
Claude Fable 5.2 Anthropic · effort xhigh 1 of 3 runs847 of 1000 LEB Gold
- Security250/250
- Architecture50/200
- Bugs150/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation47/50
Comment and details
One run so far: 847, Gold, the highest score on this instance, through the owner's personal Claude account (the client's list-price cost, about US$ 6.50, is not a charge), in 26 minutes and 1.8 million tokens. It reported 12 of the 13 planted flaws and fixed 11: security 250 of 250, bugs 150, performance 150 and clean code 100, with compatibility 100 and no penalty. Architecture was 50 of 200: it identified the dispatcher (ARCH-002) and declined to refactor it, and it did not name the magic numbers (ARCH-009). Its report scored 47 of 50 and its confidence was well calibrated (Brier 0.025).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Bugs, Performance, Clean code, Compatibility
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 84711 of 13 flaws fixedno false positivecontract kept22/22 checks26minUS$ 6.50Tokens: 3,365 in · 155,879 out · 1,656,313 cache · 1,815,557 totalScorecard
-
2
Claude Sonnet 5.5 Anthropic · effort xhigh 3 of 3 runs (825 · 809 · 724)809 of 1000 LEB Gold
- Security250/250
- Architecture25/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
The most complete single-agent result: security at 250 of 250 in its first two runs, performance at full marks, compatibility untouched, and the CSV formula injection fixed the way the answer key expects. Its third run left three security flaws found but unfixed and fell to 724; like every agent, it left the dispatcher whole.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance, Clean code, Compatibility
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 82511 of 13 flaws fixedno false positivecontract kept22/22 checks19minUS$ 3.60Tokens: 1,533 in · 138,245 out · 7,176,279 cache · 7,316,057 totalScorecard
- Run 2 · 80911 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 2.96Tokens: 1,092 in · 131,538 out · 4,673,192 cache · 4,805,822 totalScorecard
- Run 3 · 7248 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 2.91Tokens: 1,090 in · 127,242 out · 4,686,490 cache · 4,814,822 totalScorecard
-
3
Claude Sonnet 5.5 Anthropic · effort max 3 of 3 runs (807 · 773 · 820)807 of 1000 LEB Gold
- Security250/250
- Architecture12/200
- Bugs150/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
The most consistent of the top three: 773 to 820 across three runs, bugs and performance at full marks in all of them, no false positive and no contract broken. More effort than xhigh bought nothing measurable, at about one and a half times the cost.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Bugs, Performance, Clean code, Compatibility
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 80711 of 13 flaws fixedno false positivecontract kept22/22 checks27minUS$ 3.86Tokens: 966 in · 214,201 out · 3,527,234 cache · 3,742,401 totalScorecard
- Run 2 · 77310 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 4.75Tokens: 1,078 in · 229,079 out · 6,339,029 cache · 6,569,186 totalScorecard
- Run 3 · 82011 of 13 flaws fixedno false positivecontract kept22/22 checks40minUS$ 6.65Tokens: 5,597 in · 352,423 out · 7,098,658 cache · 7,456,678 totalScorecard
-
4
Claude Sonnet 5.5 Anthropic · effort max (ultracode) 3 of 3 runs (774 · 820 · 759)774 of 1000 LEB Gold
- Security250/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
Claude Code's multi-agent mode: 68 to 145 subagents and 2h to 7h 55min a run, at US$ 119 to 233 each, for 774, 820 and 759. At best it tied the same model working alone in half an hour, with the best-calibrated report of the Claude agents and the same untouched architecture.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 77411 of 13 flaws fixedno false positivecontract kept22/22 checks7h 55minUS$ 233.08Tokens: 8,703 in · 6,819,101 out · 627,624,423 cache · 634,452,227 totalScorecard
- Run 2 · 82011 of 13 flaws fixedno false positivecontract kept22/22 checks2hUS$ 171.18Tokens: 90,631 in · 6,961,681 out · 329,949,168 cache · 337,001,480 totalScorecard
- Run 3 · 75910 of 13 flaws fixedno false positivecontract kept22/22 checks3h 34minUS$ 119.07Tokens: 4,150 in · 3,825,335 out · 270,341,344 cache · 274,170,829 totalScorecard
-
5
Claude Opus 5.5 Anthropic · effort high 3 of 3 runs (733 · 767 · 804)767 of 1000 LEB Gold
- Security233/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
Fixes 8 to 10 of the 13 flaws in 6 to 8 minutes and US$ 1.14 to 1.59 a run: both injections, the XSS, the session fixation, the IDOR, the hardcoded secrets, the division by zero, the file-handle leak and the N+1 query every time; the second and third also migrate the MD5 passwords and flatten the nested ifs. It reports the CSV formula injection in all three runs and leaves it on purpose, and architecture is what swings (50, 0 and 25 of 200): the dispatcher is named in two runs and never refactored. Compatibility stays at 100 with no penalty. It scores 50 points above the same model at xhigh (717).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7338 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 1.14Tokens: 40 in · 33,929 out · 838,605 cache · 872,574 totalScorecard
- Run 2 · 76710 of 13 flaws fixedno false positivecontract kept22/22 checks8minUS$ 1.59Tokens: 46 in · 43,399 out · 1,009,913 cache · 1,053,358 totalScorecard
- Run 3 · 80410 of 13 flaws fixedno false positivecontract kept22/22 checks7minUS$ 1.38Tokens: 44 in · 35,147 out · 1,041,719 cache · 1,076,910 totalScorecard
-
6
Claude Fable 5.1 Anthropic · effort xhigh 2 of 3 runs (781 · 764)764 of 1000 LEB Gold
- Security233/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation42/50
Comment and details
Finds as much as Sonnet (11 to 12 of the 13 flaws) and fixes less of it: it explained the CSV formula injection and left it in place on purpose, to protect the file's consumers. Two runs so far; its client switched part of a third attempt to another model, which voided it.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7819 of 13 flaws fixedno false positivecontract kept22/22 checks16minTokens: 932 in · 71,351 out · 2,551,395 cache · 2,623,678 totalScorecard
- Run 2 · 76410 of 13 flaws fixedno false positivecontract kept22/22 checks22minUS$ 8.04Tokens: 1,630 in · 93,132 out · 2,502,648 cache · 2,597,410 totalScorecard
-
7
Claude Sonnet 5.5 Anthropic · effort high 3 of 3 runs (795 · 737 · 747)747 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
Fixes 8 to 10 of the 13 flaws in 3 to 5 minutes and US$ 0.40 to 0.69 a run, and the first run's 795 does not repeat: 737 and 747 leave the official score at 747. MD5 stays in all three runs and the CSV formula injection in two; the third fixes the injection but also rewrites the empty technician cell as '- (PEN-001, −15). The second keeps a literal fallback for the database password, and architecture is 0 in both later runs. Compatibility stays at 100 with no COMP-003. The same model scores 809 at xhigh and 667 at medium.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7959 of 13 flaws fixedno false positivecontract kept22/22 checks4minUS$ 0.51Tokens: 42 in · 25,328 out · 794,214 cache · 819,584 totalScorecard
- Run 2 · 7378 of 13 flaws fixedno false positivecontract kept22/22 checks5minUS$ 0.69Tokens: 48 in · 28,027 out · 1,032,378 cache · 1,060,453 totalScorecard
- Run 3 · 74710 of 13 flaws fixedno false positivecontract kept22/22 checks3minUS$ 0.40Tokens: 22 in · 19,924 out · 363,306 cache · 383,252 totalScorecard
-
8
Claude Opus 5.5 Anthropic · effort xhigh 3 of 3 runs (711 · 717 · 805)717 of 1000 LEB Silver
- Security233/250
- Architecture50/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation45/50
Comment and details
The highest architecture score in an official run (50 of 200), and the widest spread among the Claude models: 711 and 717, then 805. It fixed 9 flaws in every run and never flattened the nested ifs in the run that counts, which is where its gap to Sonnet comes from.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Clean code
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7119 of 13 flaws fixedno false positive1 business value changed22/22 checks16minTokens: 40 in · 80,368 out · 1,949,756 cache · 2,030,164 totalScorecard
- Run 2 · 7179 of 13 flaws fixedno false positivecontract kept22/22 checks18minUS$ 3.01Tokens: 962 in · 83,343 out · 1,940,114 cache · 2,024,419 totalScorecard
- Run 3 · 8059 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 2.71Tokens: 1,074 in · 69,811 out · 2,537,939 cache · 2,608,824 totalScorecard
-
9
Claude Sonnet 5.5 Anthropic · effort medium 3 of 3 runs (718 · 667 · 645)667 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code25/100
- Compatibility100/100
- Explanation42/50
Comment and details
The three fastest runs of the instance, a minute and a half and under US$ 0.20 each, for 667. It fixes both injections and the XSS, the session, the IDOR, the division by zero and the N+1 query, and keeps compatibility at 100 in all three runs. It reports the CSV formula injection and the MD5 passwords and leaves both, and touches nothing in architecture; the same model scores 809 at xhigh.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7188 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.17Tokens: 10 in · 8,897 out · 128,702 cache · 137,609 totalScorecard
- Run 2 · 6678 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.19Tokens: 10 in · 10,071 out · 130,914 cache · 140,995 totalScorecard
- Run 3 · 6457 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.17Tokens: 10 in · 8,627 out · 126,710 cache · 135,347 totalScorecard
-
10
GPT-6-astra OpenAI · effort ultra 3 of 3 runs (668 · 666 · 651)666 of 1000 LEB Silver
- Security224/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation43/50
Comment and details
The strongest GPT agent, with two or three subagents in 10 to 15 minutes a run and three runs within 17 points of each other. It changed one business value in every run (−30 each), and its reports are among the best calibrated here (Brier 0.000).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6689 of 13 flaws fixedno false positive1 business value changed22/22 checks11minTokens: 174,060 in · 38,839 out · 3,108,480 cache · 3,321,379 totalScorecard
- Run 2 · 6669 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 142,970 in · 36,517 out · 2,383,872 cache · 2,563,359 totalScorecard
- Run 3 · 6518 of 13 flaws fixedno false positive1 business value changed22/22 checks10minTokens: 171,561 in · 25,580 out · 1,841,536 cache · 2,038,677 totalScorecard
-
11
Kimi K3.1 Pro Moonshot AI · effort high (permissions auto-approved) 3 of 3 runs (632 · 666 · 665)665 of 1000 LEB Silver
- Security250/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation41/50
Comment and details
Three runs, 632, 666 and 665 (official 665, Silver), in 7.5 to 7.8 minutes and US$ 0.47 to 0.51 at the client's list price (it ran through the Kimi Code membership, which charges nothing per token). Each found 10 of the 13 planted flaws and fixed 8, with 246 to 250 of 250 in security and full marks in performance, but nothing in architecture or clean code. Every run lost 15 points for a new defect in its own fix, and the first also changed a contracted behavior (compatibility 70).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 63210 of 13 flaws fixedno false positive1 business value changed22/22 checks8minUS$ 0.50Tokens: 42,366 in · 14,948 out · 854,016 cache · 920,467 totalScorecard
- Run 2 · 66610 of 13 flaws fixedno false positivecontract kept22/22 checks8minUS$ 0.51Tokens: 42,384 in · 13,989 out · 895,901 cache · 962,726 totalScorecard
- Run 3 · 66510 of 13 flaws fixedno false positivecontract kept22/22 checks8minUS$ 0.47Tokens: 39,902 in · 14,602 out · 798,621 cache · 861,133 totalScorecard
-
12
GPT-6.1-sol OpenAI · effort xhigh 3 of 3 runs (666 · 653 · 661)661 of 1000 LEB Silver
- Security246/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation41/50
Comment and details
Security at 246 of 250, the best outside Claude, with the CSV formula injection and MD5 both fixed in the official run. What holds it below the Claude models is one business value changed in each run (−30) and the nested ifs left as they were.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6669 of 13 flaws fixedno false positive1 business value changed22/22 checks27minTokens: 73,937 in · 43,294 out · 1,233,792 cache · 1,351,023 totalScorecard
- Run 2 · 6539 of 13 flaws fixedno false positive1 business value changed22/22 checks26minTokens: 73,874 in · 33,443 out · 887,296 cache · 994,613 totalScorecard
- Run 3 · 66110 of 13 flaws fixedno false positive1 business value changed22/22 checks24minTokens: 60,781 in · 34,648 out · 953,728 cache · 1,049,157 totalScorecard
-
13
MiniMax M3.1 Flash MiniMax · effort high (permissions auto-approved) 3 of 3 runs (661 · 639 · 663)661 of 1000 LEB Silver
- Security246/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation41/50
Comment and details
Three runs, 661, 639 and 663 (official 661, Silver), at US$ 0.14 to 0.17 a run and 17 to 27 minutes, through MiniMax's own API. Each found 8 to 10 of the 13 planted flaws and fixed 7 or 8, with 224 to 246 of 250 in security and full marks in performance; it scored a little in architecture in the first run and in clean code in the second, and changed one contracted behavior in the first two runs (compatibility 70).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 66110 of 13 flaws fixedno false positive1 business value changed22/22 checks17minUS$ 0.17Tokens: 60,315 in · 39,592 out · 7,334,528 cache · 7,481,318 totalScorecard
- Run 2 · 6399 of 13 flaws fixedno false positive1 business value changed22/22 checks27minUS$ 0.16Tokens: 61,719 in · 44,818 out · 6,705,920 cache · 6,851,549 totalScorecard
- Run 3 · 6639 of 13 flaws fixedno false positivecontract kept22/22 checks26minUS$ 0.14Tokens: 56,336 in · 37,134 out · 5,565,696 cache · 5,699,029 totalScorecard
-
14
GPT-6.1-sol pro OpenAI · effort xhigh 1 of 3 runs654 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation42/50
Comment and details
One run so far, through opencode, at US$ 1.01: 654, close to GPT-6.1-sol in Codex CLI. It left the CSV formula injection in place and changed one business value (−30).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6549 of 13 flaws fixedno false positive1 business value changed22/22 checks16minUS$ 1.01Tokens: 126,508 in · 41,237 out · 810,414 cache · 1,000,316 totalScorecard
-
15
Claude Haiku 5.5 Anthropic · effort xhigh 3 of 3 runs (646 · 627 · 643)643 of 1000 LEB Silver
- Security250/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation44/50
Comment and details
The small Claude with more effort: 8 to 10 of the 13 flaws fixed in 10 to 14 minutes and US$ 0.37 to 0.65 a run. Its third run fixes all seven security flaws, the CSV formula injection and the MD5 passwords among them (250 of 250); the first leaves both. The nested ifs are never named (clean code 0) and the file-handle leak is fixed without being reported in two runs. Two runs lose 30 points because formatarStatus now returns 'Desconhecido' for statuses outside 1 to 3. About twice the score of Haiku 4.5 (317).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6468 of 13 flaws fixedno false positivecontract kept22/22 checks11minUS$ 0.52Tokens: 106 in · 134,985 out · 5,725,449 cache · 5,860,540 totalScorecard
- Run 2 · 6278 of 13 flaws fixedno false positive1 business value changed22/22 checks10minUS$ 0.37Tokens: 74 in · 121,413 out · 3,661,714 cache · 3,783,201 totalScorecard
- Run 3 · 64310 of 13 flaws fixedno false positive1 business value changed22/22 checks14minUS$ 0.65Tokens: 106 in · 156,239 out · 6,118,984 cache · 6,275,329 totalScorecard
-
16
GPT-6-luna OpenAI · effort xhigh 3 of 3 runs (633 · 643 · 655)643 of 1000 LEB Silver
- Security224/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
Fixes 9 or 10 of the 13 flaws, and in its third run all seven security flaws, the CSV formula injection and the MD5 passwords among them (250 of 250). Two runs lose 15 points because the CSV guard rewrites the empty technician cell as '-, the file-handle leak is fixed without being reported, and nothing is done on architecture or clean code. Compatibility stays at 100.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6339 of 13 flaws fixedno false positivecontract kept22/22 checks7minTokens: 63,253 in · 29,862 out · 579,456 cache · 672,571 totalScorecard
- Run 2 · 6439 of 13 flaws fixedno false positivecontract kept22/22 checks6minTokens: 68,820 in · 34,856 out · 965,760 cache · 1,069,436 totalScorecard
- Run 3 · 65510 of 13 flaws fixedno false positivecontract kept22/22 checks5minTokens: 106,034 in · 27,936 out · 551,168 cache · 685,138 totalScorecard
-
17
Grok 4.7 xAI · effort high 3 of 3 runs (638 · 663 · 607)638 of 1000 LEB Silver
- Security207/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation42/50
Comment and details
The strongest model from outside Anthropic and OpenAI, with compatibility at 100 in two of three runs and reports at 42 of 50. It kept the unsalted MD5 passwords in every run; in its first it was the only agent to look something up on the web, the PHP manual page for fputcsv.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6388 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 2.43Tokens: 375,533 in · 19,450 out · 2,298,496 cache · 2,761,606 totalScorecard
- Run 2 · 6638 of 13 flaws fixedno false positivecontract kept22/22 checks24minUS$ 2.88Tokens: 464,772 in · 17,294 out · 2,760,320 cache · 3,320,861 totalScorecard
- Run 3 · 6078 of 13 flaws fixedno false positive1 business value changed22/22 checks26minUS$ 2.57Tokens: 388,897 in · 16,860 out · 2,347,904 cache · 2,839,851 totalScorecard
-
18
GPT-6.1-sol OpenAI · effort high 3 of 3 runs (626 · 637 · 727)637 of 1000 LEB Silver
- Security207/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation46/50
Comment and details
The best explanations of the three agents added on 2026-10-26 (44 to 46 of 50) and the best single run among them, 727, which also fixes the CSV formula injection and flattens the nested ifs. In all three runs it scopes the SLA average to the logged-in client, a contracted value changed (−30), as the same model did at xhigh; architecture stays at 0.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 6268 of 13 flaws fixedno false positive1 business value changed22/22 checks10minTokens: 33,136 in · 25,058 out · 643,968 cache · 702,162 totalScorecard
- Run 2 · 6378 of 13 flaws fixedno false positive1 business value changed22/22 checks11minTokens: 40,422 in · 24,392 out · 617,600 cache · 682,414 totalScorecard
- Run 3 · 72710 of 13 flaws fixedno false positive1 business value changed22/22 checks9minTokens: 39,593 in · 19,951 out · 580,608 cache · 640,152 totalScorecard
-
19
Claude Haiku 5.5 Anthropic · effort high 3 of 3 runs (633 · 644 · 622)633 of 1000 LEB Silver
- Security246/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility40/100
- Explanation43/50
Comment and details
The small Claude at high: 9 or 10 of the 13 flaws fixed in 6 to 9 minutes and US$ 0.08 to 0.25 a run, 10 points below the same model at xhigh (633 against 643). It fixes the CSV formula injection in all three runs and the MD5 passwords in two (security 246, 250 and 228), fixes the file-handle leak without reporting it every time, and never names the nested ifs (clean code 0). The first run loses 60 points to two COMP-003: its CSV export now answers 403 to clients, and formatarStatus returns 'Desconhecido'; the other two lose 30 for the second.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 63310 of 13 flaws fixedno false positive2 business values changed22/22 checks7minUS$ 0.19Tokens: 74 in · 77,788 out · 2,737,517 cache · 2,815,379 totalScorecard
- Run 2 · 64410 of 13 flaws fixedno false positive1 business value changed22/22 checks9minUS$ 0.25Tokens: 70 in · 95,380 out · 2,932,651 cache · 3,028,101 totalScorecard
- Run 3 · 6229 of 13 flaws fixedno false positive1 business value changed22/22 checks6minUS$ 0.08Tokens: 60 in · 61,750 out · 1,948,974 cache · 2,010,784 totalScorecard
-
20
Grok 4.6 xAI · effort high 3 of 3 runs (633 · 640 · 620)633 of 1000 LEB Silver
- Security194/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility100/100
- Explanation35/50
Comment and details
Nearly Grok 4.7's score for about an eighth of the cost (US$ 0.28 to 0.31 a run), and steady: 620 to 640, 9 flaws found in each run, no contract broken. It never touched MD5 or the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6337 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.31Tokens: 54,906 in · 8,789 out · 158,208 cache · 234,078 totalScorecard
- Run 2 · 6408 of 13 flaws fixedno false positivecontract kept22/22 checks4minUS$ 0.31Tokens: 53,592 in · 8,053 out · 158,720 cache · 232,132 totalScorecard
- Run 3 · 6207 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.28Tokens: 38,168 in · 8,547 out · 145,536 cache · 205,923 totalScorecard
-
21
Grok 4.7 xAI · effort xhigh 3 of 3 runs (640 · 617 · 631)631 of 1000 LEB Silver
- Security185/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility100/100
- Explanation42/50
Comment and details
More effort scored slightly lower than Grok 4.7 at high, at a higher cost: it fixed 7 or 8 flaws a run and left MD5 and the hardcoded secrets in every one. Its reports, 42 to 44 of 50, are the best explanations from outside Anthropic and OpenAI.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6408 of 13 flaws fixedno false positivecontract kept22/22 checks37minUS$ 3.82Tokens: 630,356 in · 24,515 out · 3,767,424 cache · 4,510,681 totalScorecard
- Run 2 · 6177 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 3.59Tokens: 607,690 in · 20,802 out · 3,405,184 cache · 4,125,277 totalScorecard
- Run 3 · 6317 of 13 flaws fixedno false positivecontract kept22/22 checks33minUS$ 3.31Tokens: 579,862 in · 20,500 out · 2,982,528 cache · 3,672,680 totalScorecard
-
22
Kimi K3 Moonshot AI · effort high 3 of 3 runs (629 · 634 · 574)629 of 1000 LEB Silver
- Security198/250
- Architecture0/200
- Bugs150/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation36/50
Comment and details
The strongest Kimi by a wide margin, with bugs and performance at full marks in its official run, for US$ 0.41 to 1.02 a run. Every run changed one business value (−30), and none fixed the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Bugs, Performance
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6298 of 13 flaws fixedno false positive1 business value changed22/22 checks7minUS$ 0.41Tokens: 28,836 in · 11,322 out · 279,808 cache · 324,940 totalScorecard
- Run 2 · 6349 of 13 flaws fixedno false positive1 business value changed22/22 checks21minUS$ 1.02Tokens: 52,871 in · 21,600 out · 1,249,024 cache · 1,334,129 totalScorecard
- Run 3 · 5747 of 13 flaws fixedno false positive1 business value changed22/22 checks15minUS$ 0.81Tokens: 48,057 in · 18,001 out · 934,144 cache · 1,008,035 totalScorecard
-
23
GPT-6-astra OpenAI · effort xhigh 3 of 3 runs (661 · 596 · 628)628 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation41/50
Comment and details
Single-agent GPT-6-astra, 38 points below its ultra setting. Its runs disagree on what to fix: the first fixed the CSV formula injection and kept MD5, the official one migrated MD5 to password_hash and left the injection alone.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6619 of 13 flaws fixedno false positive1 business value changed22/22 checks17minTokens: 58,947 in · 31,166 out · 753,408 cache · 843,521 totalScorecard
- Run 2 · 5968 of 13 flaws fixedno false positive1 business value changed22/22 checks12minTokens: 55,599 in · 21,712 out · 549,120 cache · 626,431 totalScorecard
- Run 3 · 6289 of 13 flaws fixedno false positive1 business value changed22/22 checks14minTokens: 69,459 in · 26,327 out · 659,328 cache · 755,114 totalScorecard
-
24
GLM-5.3 Prime Z.AI · effort high 3 of 3 runs (635 · 628 · 541)628 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation38/50
Comment and details
The strongest GLM, at about US$ 2 a run. Its first run fixed all four flaws the probes cover, the CSV formula injection among them; the third changed two business values and fell to 541.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6359 of 13 flaws fixed1 false positivecontract kept22/22 checks14minUS$ 1.69Tokens: 66,011 in · 21,625 out · 2,077,696 cache · 2,182,652 totalScorecard
- Run 2 · 6288 of 13 flaws fixedno false positivecontract kept22/22 checks26minUS$ 2.13Tokens: 78,339 in · 24,496 out · 2,647,040 cache · 2,773,658 totalScorecard
- Run 3 · 5417 of 13 flaws fixedno false positive2 business values changed22/22 checks17minUS$ 1.77Tokens: 67,380 in · 20,363 out · 2,154,240 cache · 2,263,862 totalScorecard
-
25
GPT-5.6-terra OpenAI · effort xhigh 3 of 3 runs (625 · 611 · 645)625 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation35/50
Comment and details
Reports fewer flaws than it fixes: its first run listed 7 and fixed 9, the rest done in passing without a word in the report. Compatibility held in two of three runs, which keeps it 15th despite the short report.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6259 of 13 flaws fixedno false positivecontract kept22/22 checks8minTokens: 54,590 in · 25,866 out · 486,656 cache · 567,112 totalScorecard
- Run 2 · 61110 of 13 flaws fixedno false positive1 business value changed22/22 checks8minTokens: 63,042 in · 25,647 out · 822,016 cache · 910,705 totalScorecard
- Run 3 · 64510 of 13 flaws fixedno false positivecontract kept22/22 checks7minTokens: 52,130 in · 22,262 out · 444,928 cache · 519,320 totalScorecard
-
26
GLM-5.3-Flash Z.AI · effort high 1 of 3 runs624 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation34/50
Comment and details
One run so far: 624 for US$ 0.08, about a tenth of GLM-5.3's cost for a slightly higher score. It left the CSV formula injection and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6248 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 0.08Tokens: 68,681 in · 22,218 out · 1,549,184 cache · 1,655,227 totalScorecard
-
27
Kimi K3.1 Pro (highspeed) Moonshot AI · effort high (permissions auto-approved) 3 of 3 runs (648 · 345 · 622)622 of 1000 LEB Silver
- Security246/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation42/50
Comment and details
Three runs, 648, 345 and 622 (official 622, Silver; mean 538), in 5.7 to 11.4 minutes and US$ 0.81 to 1.46 at the client's list price (Kimi Code membership). The second run reported 10 of the 13 planted flaws but fixed none, which cost it the pass (Reprovada); the other two fixed 8 each, with full marks in performance and a contracted behavior changed in both (compatibility 70), and the third also lost 15 points for a new defect in its own fix. Architecture and clean code stayed at zero.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 64810 of 13 flaws fixedno false positive1 business value changed22/22 checks11minUS$ 1.46Tokens: 79,657 in · 23,167 out · 2,339,584 cache · 2,453,026 totalScorecard
- Run 2 · 3450 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.81Tokens: 43,577 in · 16,975 out · 595,361 cache · 664,708 totalScorecard
- Run 3 · 62210 of 13 flaws fixedno false positive1 business value changed22/22 checks6minUS$ 0.93Tokens: 52,272 in · 17,940 out · 567,552 cache · 650,440 totalScorecard
-
28
GLM-5.3 Z.AI · effort high 3 of 3 runs (629 · 621 · 604)621 of 1000 LEB Silver
- Security203/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation39/50
Comment and details
Steady (604 to 629) and cheap (US$ 0.47 to 0.73 a run), with compatibility at 100 in all three runs. It never fixed MD5 or the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6298 of 13 flaws fixedno false positivecontract kept22/22 checks17minUS$ 0.73Tokens: 65,504 in · 21,730 out · 1,778,176 cache · 1,883,245 totalScorecard
- Run 2 · 6217 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 0.51Tokens: 376,050 in · 21,002 out · 1,480,512 cache · 1,906,219 totalScorecard
- Run 3 · 6047 of 13 flaws fixedno false positivecontract kept22/22 checks13minUS$ 0.47Tokens: 63,302 in · 22,219 out · 1,457,920 cache · 1,561,006 totalScorecard
-
29
NVIDIA Nemotron 3 Ultra NVIDIA · effort high (permissions auto-approved) 1 of 3 runs618 of 1000 LEB Silver
- Security164/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code38/100
- Compatibility100/100
- Explanation37/50
Comment and details
One run so far: 618 (Silver) at no cost, through OpenRouter's free variant, in 4 minutes. It found 8 of the 13 planted flaws and fixed 7, with full marks in performance, 129 of 150 in bugs and 38 of 100 in clean code, and left architecture untouched; compatibility stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6187 of 13 flaws fixed2 false positivescontract kept22/22 checks4minUS$ 0.00Tokens: 87,335 in · 14,181 out · 429,632 cache · 533,888 totalScorecard
-
30
DeepSeek V4 Flash DeepSeek · effort xhigh 1 of 3 runs617 of 1000 LEB Silver
- Security207/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
One run so far, through Novita: 617 for US$ 0.18, against 282 for the same model at high. It fixed 7 flaws and left the CSV injection, MD5 and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6177 of 13 flaws fixedno false positivecontract kept22/22 checks28minUS$ 0.18Tokens: 163,821 in · 32,918 out · 4,833,280 cache · 5,068,978 totalScorecard
-
31
GPT-6.1-sol OpenAI · effort ultra 3 of 3 runs (597 · 656 · 616)616 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation39/50
Comment and details
Ultra effort scored 45 points below GPT-6.1-sol at xhigh, across runs from 597 to 656. With three subagents its first run kept MD5; with two, its second fixed 10 flaws, the CSV formula injection and MD5 among them.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5978 of 13 flaws fixedno false positive1 business value changed22/22 checks24minTokens: 215,626 in · 39,255 out · 3,416,064 cache · 3,670,945 totalScorecard
- Run 2 · 65610 of 13 flaws fixedno false positive1 business value changed22/22 checks13minTokens: 177,994 in · 38,220 out · 2,064,000 cache · 2,280,214 totalScorecard
- Run 3 · 6169 of 13 flaws fixedno false positive1 business value changed22/22 checks11minTokens: 193,606 in · 40,785 out · 3,059,328 cache · 3,293,719 totalScorecard
-
32
GPT-5.6-sol OpenAI · effort xhigh 3 of 3 runs (612 · 612 · 608)612 of 1000 LEB Silver
- Security246/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
Security at 246 of 250 and the CSV formula injection fixed, but the fix also turned the '-' of a ticket with no technician into "'-", a new bug, and every run changed one business value (−30). The most repeatable GPT: 608 to 612.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 61210 of 13 flaws fixedno false positive1 business value changed22/22 checks7minTokens: 52,386 in · 18,315 out · 1,163,520 cache · 1,234,221 totalScorecard
- Run 2 · 6129 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 67,246 in · 32,457 out · 905,216 cache · 1,004,919 totalScorecard
- Run 3 · 6089 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 63,699 in · 33,962 out · 1,569,920 cache · 1,667,581 totalScorecard
-
33
DeepSeek V4.1 Flash DeepSeek · effort high 3 of 3 runs (625 · 612 · 597)612 of 1000 LEB Silver
- Security203/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
The cheapest agent here, US$ 0.01 to 0.04 a run, two of them under two minutes, for 597 to 625. Compatibility held in all three; the lowest explanation score among the agents above 600 (30 of 50).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6259 of 13 flaws fixedno false positivecontract kept22/22 checks9minUS$ 0.04Tokens: 30,817 in · 24,049 out · 2,467,712 cache · 2,543,405 totalScorecard
- Run 2 · 6128 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.01Tokens: 20,053 in · 12,321 out · 357,504 cache · 397,812 totalScorecard
- Run 3 · 5977 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.02Tokens: 27,064 in · 13,385 out · 535,040 cache · 585,035 totalScorecard
-
34
GPT-5.6-luna OpenAI · effort xhigh 3 of 3 runs (599 · 601 · 624)601 of 1000 LEB Silver
- Security220/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
The smallest GPT-5.6, within 24 points of terra and sol across 599 to 624. It changed one business value in each run (−30).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5998 of 13 flaws fixedno false positive1 business value changed22/22 checks12minTokens: 74,979 in · 30,975 out · 886,784 cache · 992,738 totalScorecard
- Run 2 · 6019 of 13 flaws fixedno false positive1 business value changed22/22 checks10minTokens: 65,711 in · 31,899 out · 706,304 cache · 803,914 totalScorecard
- Run 3 · 62410 of 13 flaws fixedno false positive1 business value changed22/22 checks14minTokens: 146,803 in · 42,381 out · 1,674,752 cache · 1,863,936 totalScorecard
-
35
GLM-5.3-FlashX Z.AI · effort high 1 of 3 runs597 of 1000 LEB Bronze
- Security185/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation33/50
Comment and details
One run so far: 597 for US$ 0.10, 27 points below GLM-5.3-Flash. It fixed 7 flaws and left the CSV injection, MD5 and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5977 of 13 flaws fixedno false positivecontract kept22/22 checks8minUS$ 0.10Tokens: 48,070 in · 13,431 out · 702,784 cache · 771,911 totalScorecard
-
36
Gemini 3.8 Flash Google · effort high 3 of 3 runs (687 · 588 · 100)588 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation28/50
Comment and details
Three very different runs: 687 (it would have placed 6th), 588, and 100 from a run that stopped after ten minutes on a database server it started itself. The two complete runs kept MD5 and both secrets and left the CSV export serving every client.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 6878 of 13 flaws fixedno false positivecontract kept22/22 checks12minUS$ 1.00Tokens: 447,362 in · 30,814 out · 5,007,269 cache · 5,532,273 totalScorecard
- Run 2 · 5887 of 13 flaws fixedno false positivecontract kept22/22 checks25minUS$ 0.85Tokens: 466,673 in · 14,937 out · 3,198,620 cache · 3,735,479 totalScorecard
- Run 3 · 1000 of 13 flaws fixedno false positivecontract kept22/22 checks10minUS$ 0.28Tokens: 214,747 in · 1,444 out · 704,735 cache · 937,623 totalScorecard
-
37
Gemini 3.7 Flash Google · effort high 3 of 3 runs (587 · 587 · 556)587 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation27/50
Comment and details
The generation before Gemini 3.8 Flash, one point behind it and far steadier (556 to 587). Its first run is the only one outside Claude to flatten the nested ifs; its second searched the VM for the answer key, which is not there.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5877 of 13 flaws fixedno false positivecontract kept22/22 checks12minUS$ 0.72Tokens: 358,682 in · 22,820 out · 3,161,645 cache · 3,577,678 totalScorecard
- Run 2 · 5877 of 13 flaws fixedno false positivecontract kept22/22 checks16minUS$ 0.60Tokens: 294,559 in · 16,180 out · 2,248,786 cache · 2,599,093 totalScorecard
- Run 3 · 5567 of 13 flaws fixedno false positive1 business value changed22/22 checks7minUS$ 0.49Tokens: 315,568 in · 15,092 out · 1,053,498 cache · 1,416,483 totalScorecard
-
38
Crouton Alpha Stealth · effort max (permissions auto-approved) 1 of 3 runs580 of 1000 LEB Bronze
- Security220/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility40/100
- Explanation41/50
Comment and details
One run so far: 580 (Bronze) at no cost, through OpenRouter's free variant of a stealth model whose maker is withheld, in 30 minutes. Security 220 of 250, bugs 129 of 150 and full marks in performance; it found 8 of the 13 planted flaws, fixed 9 and reported 6 more real findings, but two contracted behaviors changed (compatibility 40) and architecture and clean code stayed at zero.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5809 of 13 flaws fixedno false positive2 business values changed22/22 checks30minUS$ 0.00Tokens: 30,356,555 in · 16,801 out · 0 cache · 30,374,669 totalScorecard
-
39
GPT-5.5 OpenAI · effort xhigh 3 of 3 runs (558 · 568 · 536)558 of 1000 LEB Bronze
- Security177/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
The previous GPT generation, 43 to 108 points below the GPT-5.6 and GPT-6 models: 8 flaws found and 7 fixed in every run, MD5 and the CSV formula injection always left in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 2 · 5587 of 13 flaws fixedno false positive1 business value changed22/22 checks6minTokens: 54,617 in · 18,745 out · 374,272 cache · 447,634 totalScorecard
- Run 3 · 5687 of 13 flaws fixedno false positivecontract kept22/22 checks5minTokens: 47,993 in · 16,748 out · 387,584 cache · 452,325 totalScorecard
- Run 4 · 5367 of 13 flaws fixedno false positive1 business value changed22/22 checks5minTokens: 50,817 in · 16,437 out · 435,456 cache · 502,710 totalScorecard
-
40
Kimi For Coding Moonshot AI · default effort (not configurable) (permissions auto-approved) 1 of 3 runs555 of 1000 LEB Bronze
- Security207/250
- Architecture0/200
- Bugs86/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation42/50
Comment and details
One run so far: 555 (Bronze) in 16 minutes, through the Kimi Code membership under its alias `kimi-for-coding`, which the API reports without the build behind it: on 2026-11-07, the day of the run, the lab was told it pointed to Kimi K2.8 Preview. It found 9 of the 13 planted flaws and fixed 7, with full marks in performance, 207 of 250 in security and 86 of 150 in bugs, and changed one contracted behavior (compatibility 70); architecture and clean code stayed at zero.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; File handle leaked on the error path; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5557 of 13 flaws fixedno false positive1 business value changed22/22 checks16minUS$ 0.00Tokens: 50,751 in · 20,022 out · 1,354,240 cache · 1,434,325 totalScorecard
-
41
Gemini 3.8 Flash Google · effort medium 1 of 3 runs550 of 1000 LEB Bronze
- Security147/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation24/50
Comment and details
One run so far, in 4min 34s for US$ 0.19: 550. It fixed 6 flaws and claimed SQL injection in two functions that only take integers.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5506 of 13 flaws fixed2 false positivescontract kept22/22 checks5minUS$ 0.19Tokens: 153,206 in · 11,176 out · 464,128 cache · 628,510 totalScorecard
-
42
Recursive Flint Recursive · default effort (not configurable) (permissions auto-approved) 1 of 3 runs526 of 1000 LEB Bronze
- Security142/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation35/50
Comment and details
One run so far: 526 (Bronze) at no cost, through OpenRouter's free variant at the model's default setting, in 9 minutes. It found 6 of the 13 planted flaws and fixed 6, with full marks in performance and 129 of 150 in bugs, but nothing in architecture or clean code, and one contracted behavior changed (compatibility 70).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5266 of 13 flaws fixedno false positive1 business value changed22/22 checks9minUS$ 0.00Tokens: 66,280 in · 11,100 out · 823,424 cache · 938,088 totalScorecard
-
43
Qwen3 Coder Next Alibaba (Qwen team) · default effort (not configurable) 1 of 3 runs507 of 1000 LEB Bronze
- Security125/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation18/50
Comment and details
The weakest explanation (18 of 50) and the worst calibration here: three flaws that cannot exist reported at confidence 100, and no tests run. It is, on the other hand, the only one of the bottom nine that fixed the N+1 query.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5075 of 13 flaws fixed3 false positivescontract kept22/22 checks16minUS$ 1.53Tokens: 7,393,841 in · 32,724 out · 0 cache · 7,426,565 totalScorecard
-
44
DeepSeek V4 Pro DeepSeek · effort high 3 of 3 runs (604 · 496 · 432)496 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance56/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
The larger DeepSeek: all three runs (604, 496 and 432) score below DeepSeek V4.1 Flash's official 612. The first rated SQL injection at confidence 100 in two functions that only take integers; the other two left the N+1 query in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6048 of 13 flaws fixed2 false positivescontract kept22/22 checks8minUS$ 0.03Tokens: 68,936 in · 13,219 out · 322,560 cache · 412,383 totalScorecard
- Run 2 · 4966 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 0.42Tokens: 28,203 in · 15,517 out · 1,835,520 cache · 1,904,131 totalScorecard
- Run 3 · 4326 of 13 flaws fixed1 false positivecontract kept22/22 checks12minUS$ 0.04Tokens: 103,562 in · 15,859 out · 580,864 cache · 708,898 totalScorecard
-
45
Kimi For Coding (highspeed) Moonshot AI · default effort (not configurable) (permissions auto-approved) 1 of 3 runs472 of 1000 LEB Bronze
- Security177/250
- Architecture0/200
- Bugs129/150
- Performance56/150
- Clean code0/100
- Compatibility70/100
- Explanation40/50
Comment and details
One run so far: 472 (Bronze) in under 3 minutes, through the Kimi Code membership under its alias `kimi-for-coding-highspeed`, which the API reports without the build behind it: on 2026-11-07, the day of the run, the lab was told it pointed to Kimi K2.7 Code Highspeed. It found 8 of the 13 planted flaws and fixed 5, with 177 of 250 in security, 129 of 150 in bugs and 56 of 150 in performance, and changed one contracted behavior (compatibility 70); architecture and clean code stayed at zero.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4726 of 13 flaws fixed2 false positives1 business value changed22/22 checks3minUS$ 0.00Tokens: 35,860 in · 11,255 out · 391,168 cache · 449,849 totalScorecard
-
46
MiniMax-M3 MiniMax · effort thinking 3 of 3 runs (462 · 616 · 434)462 of 1000 LEB Bronze
- Security220/250
- Architecture0/200
- Bugs150/150
- Performance0/150
- Clean code25/100
- Compatibility40/100
- Explanation27/50
Comment and details
Fixed 8 flaws in every run, with bugs at full marks, but changed business values (compatibility at 40 in its official run) and left the N+1 query. Its second run scored 616; its third broke four of the 22 characterization checks.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Bugs
- Zero
- Architecture, Performance
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 4628 of 13 flaws fixedno false positive1 business value changed22/22 checks4minUS$ 0.12Tokens: 55,144 in · 19,131 out · 993,940 cache · 1,085,756 totalScorecard
- Run 2 · 6168 of 13 flaws fixedno false positive1 business value changed22/22 checks6minUS$ 0.10Tokens: 69,677 in · 14,753 out · 601,806 cache · 707,582 totalScorecard
- Run 3 · 4348 of 13 flaws fixed1 false positive1 business value changed18/22 checks5minUS$ 0.13Tokens: 63,000 in · 22,731 out · 1,089,762 cache · 1,188,744 totalScorecard
-
47
NVIDIA Nemotron 3.5 Super VL NVIDIA · effort high (permissions auto-approved) 1 of 3 runs446 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation36/50
Comment and details
One run so far: 446 (Bronze) at no cost, through OpenRouter's free variant, in 4 minutes. It found 7 of the 13 planted flaws and fixed 6, with 181 of 250 in security and 129 of 150 in bugs, but nothing in architecture, performance or clean code; compatibility stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4466 of 13 flaws fixedno false positivecontract kept22/22 checks4minUS$ 0.00Tokens: 86,396 in · 10,525 out · 654,736 cache · 764,893 totalScorecard
-
48
Kimi K2.7 Code Moonshot AI · default effort (not configurable) 3 of 3 runs (415 · 415 · 512)415 of 1000 LEB Bronze
- Security203/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation26/50
Comment and details
Left the N+1 query in every run and reported flaws that do not exist in all three, SQL injection in functions that only take integers among them. Its third run, 512, fixed one flaw more than the other two.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4156 of 13 flaws fixed2 false positivescontract kept22/22 checks14minUS$ 0.48Tokens: 55,678 in · 20,809 out · 1,616,640 cache · 1,702,051 totalScorecard
- Run 2 · 4156 of 13 flaws fixed1 false positivecontract kept22/22 checks4minUS$ 0.25Tokens: 43,066 in · 10,167 out · 766,976 cache · 826,182 totalScorecard
- Run 3 · 5127 of 13 flaws fixed2 false positivescontract kept22/22 checks15minUS$ 0.81Tokens: 80,899 in · 20,334 out · 2,925,824 cache · 3,051,753 totalScorecard
-
49
Apodex 1.1 Mini Apodex · effort high (permissions auto-approved) 3 of 3 runs (406 · 250 · 468)406 of 1000 LEB Bronze
- Security168/250
- Architecture0/200
- Bugs75/150
- Performance0/150
- Clean code25/100
- Compatibility100/100
- Explanation38/50
Comment and details
Three runs, 406, 250 and 468, so the official score is the median, 406 (Bronze). Through Novita AI's own API, where the model is listed at no price; the runs took 44, 17 and 5 minutes. Security ran 138 to 203 of 250 and bugs 75 to 129 of 150, with nothing in architecture or performance in any run. The second run fell to 250: two contracted behaviors changed (compatibility 40) and it lost 35 points to penalties.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4065 of 13 flaws fixedno false positivecontract kept22/22 checks44minUS$ 0.00Tokens: 264,119 in · 162,368 out · 18,594,944 cache · 19,021,431 totalScorecard
- Run 2 · 2504 of 13 flaws fixed2 false positives2 business values changed21/22 checks17minUS$ 0.00Tokens: 140,711 in · 128,314 out · 10,914,176 cache · 11,183,201 totalScorecard
- Run 3 · 4687 of 13 flaws fixed1 false positivecontract kept22/22 checks5minUS$ 0.00Tokens: 124,909 in · 60,050 out · 772,864 cache · 957,823 totalScorecard
-
50
GPT-5.3-Codex OpenAI · effort xhigh 1 of 3 runs403 of 1000 LEB Bronze
- Security147/250
- Architecture0/200
- Bugs129/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation27/50
Comment and details
One run so far, through opencode: 403, the lowest GPT. It found 6 flaws and fixed 5, leaving the N+1 query, the session fixation at login and MD5 in place, with no false positive.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4035 of 13 flaws fixedno false positivecontract kept22/22 checks7minUS$ 0.54Tokens: 42,455 in · 7,374 out · 351,744 cache · 422,872 totalScorecard
-
51
Kimi K2.7 Code (highspeed) Moonshot AI · default effort (not configurable) 3 of 3 runs (402 · 363 · 402)402 of 1000 LEB Bronze
- Security155/250
- Architecture0/200
- Bugs129/150
- Performance0/150
- Clean code25/100
- Compatibility70/100
- Explanation23/50
Comment and details
The fast variant of Kimi K2.7 Code, 13 points below it: 5 flaws fixed in every run, two flaws that do not exist reported in each, and the N+1 query always left in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Performance
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4025 of 13 flaws fixed2 false positives1 business value changed22/22 checks2minUS$ 0.38Tokens: 31,371 in · 10,254 out · 506,880 cache · 554,104 totalScorecard
- Run 2 · 3635 of 13 flaws fixed2 false positives1 business value changed22/22 checks6minUS$ 0.68Tokens: 48,822 in · 12,842 out · 960,722 cache · 1,037,685 totalScorecard
- Run 3 · 4025 of 13 flaws fixed2 false positivescontract kept22/22 checks10minUS$ 1.16Tokens: 73,637 in · 20,792 out · 2,034,944 cache · 2,139,219 totalScorecard
-
52
GLM-5.2 Z.AI · effort high 1 of 3 runs388 of 1000 Failed
- Security172/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
One run so far: 388, below the pass line and 233 points under GLM-5.3. It left the N+1 query, the leaked file handle and the session fixation in place; what it did report was accurate, with no false positive.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3885 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.35Tokens: 52,899 in · 13,001 out · 716,992 cache · 790,478 totalScorecard
-
53
Nex N2.5 Pro Nex AGI · effort high 2 of 3 runs (317 · 428)317 of 1000 Failed
- Security194/250
- Architecture0/200
- Bugs107/150
- Performance75/150
- Clean code0/100
- Compatibility70/100
- Explanation31/50
Comment and details
Too slow to finish three runs: 10h 03min and 19h 35min, and a third voided after more than 21 hours when the operator's connection failed, so it stays at two and publishes the lower. Both runs were strong on security and both broke the contract: the first made every function return nothing without a logged-in user (8 checks, −160, 317), the second made the CSV export throw for a caller that has already printed output (3 checks, 428).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3179 of 13 flaws fixedno false positive1 business value changed14/22 checks10h 3minUS$ 2.80Tokens: 29,260,714 in · 117,982 out · 20,215,616 cache · 49,594,312 totalScorecard
- Run 2 · 42810 of 13 flaws fixedno false positive2 business values changed19/22 checks19h 35minUS$ 4.87Tokens: 50,033,318 in · 254,132 out · 33,604,416 cache · 83,891,866 totalScorecard
-
54
Claude Haiku 4.5 Anthropic · default effort (not configurable) 3 of 3 runs (317 · 369 · 232)317 of 1000 Failed
- Security138/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility70/100
- Explanation23/50
Comment and details
The small Claude, at US$ 0.21 to 0.40 a run, below the pass line in all three. Its first run's visibility fix hides a client's own tickets, comparing an integer with the string mysqli returns; its third broke three characterization checks.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3174 of 13 flaws fixed2 false positives1 business value changed22/22 checks2minUS$ 0.21Tokens: 1,062 in · 13,699 out · 819,787 cache · 834,548 totalScorecard
- Run 2 · 3695 of 13 flaws fixed2 false positivescontract kept22/22 checks7minUS$ 0.40Tokens: 1,198 in · 20,213 out · 2,111,822 cache · 2,133,233 totalScorecard
- Run 3 · 2325 of 13 flaws fixed1 false positive1 business value changed19/22 checks10minUS$ 0.27Tokens: 1,213 in · 17,741 out · 1,063,175 cache · 1,082,129 totalScorecard
-
55
DeepSeek V4 Flash DeepSeek · effort high 2 of 3 runs (612 · 282)282 of 1000 Failed
- Security99/250
- Architecture0/200
- Bugs96/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation22/50
Comment and details
Two runs that could hardly differ more: the first fixed 9 flaws for 612, the second's SQL-injection fix throws on every search and its client CSV comes out on one line, 282. It publishes the lower; which weights the hosts served under this name is not certain.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6129 of 13 flaws fixedno false positive1 business value changed22/22 checks9minUS$ 0.07Tokens: 85,100 in · 16,008 out · 1,518,080 cache · 1,646,803 totalScorecard
- Run 2 · 2823 of 13 flaws fixed1 false positivecontract kept21/22 checks7minUS$ 0.00Tokens: 43,443 in · 8,858 out · 445,696 cache · 510,802 totalScorecard
-
56
Cohere North Mini Code Cohere · effort high (permissions auto-approved) 1 of 3 runs237 of 1000 Failed
- Security56/250
- Architecture0/200
- Bugs75/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation26/50
Comment and details
One run so far: 237, below the pass line, at no cost, through OpenRouter's free variant, in 1 minute. It found 4 of the 13 planted flaws and fixed 2, and lost 20 points for a broken behavior no fix was blamed for; compatibility itself stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 2372 of 13 flaws fixed2 false positivescontract kept21/22 checks1minUS$ 0.00Tokens: 120,940 in · 6,047 out · 0 cache · 128,930 totalScorecard
-
57
NVIDIA Nemotron 3.5 Lightning NVIDIA · effort high (permissions auto-approved) 1 of 3 runs186 of 1000 Failed
- Security43/250
- Architecture0/200
- Bugs32/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
One run so far: 186, below the pass line, at no cost, through OpenRouter's free variant, in 5 minutes. It found 2 of the 13 planted flaws and fixed 2, and lost 20 points for a broken behavior no fix was blamed for; compatibility itself stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 1862 of 13 flaws fixed2 false positivescontract kept21/22 checks5minUS$ 0.00Tokens: 96,557 in · 13,687 out · 1,142,320 cache · 1,264,244 totalScorecard
-
58
NVIDIA Nemotron 3 Nano Omni NVIDIA · effort max (permissions auto-approved) 1 of 3 runs183 of 1000 Failed
- Security56/250
- Architecture0/200
- Bugs0/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation27/50
Comment and details
One run so far: 183, below the pass line, at no cost, through OpenRouter's free variant at max, the highest effort the client defines for it, in 3 minutes. It found 2 of the 13 planted flaws and fixed 1: security 56 of 250 and nothing in bugs, architecture, performance or clean code; compatibility stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Bugs, Performance, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); Division by zero in the SLA average; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 1831 of 13 flaws fixed2 false positivescontract kept22/22 checks3minUS$ 0.00Tokens: 34,130 in · 3,778 out · 274,736 cache · 328,022 totalScorecard
-
59
Liquid LFM 2.5 2.6B Liquid · default effort (not configurable) (permissions auto-approved) 1 of 3 runs164 of 1000 Failed
- Security39/250
- Architecture0/200
- Bugs0/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation25/50
Comment and details
One run so far: 164, below the pass line, at no cost, through OpenRouter's free variant at the model's default setting, in under a minute. It reported 2 of the 13 planted flaws and fixed none, so it scored only 39 of 250 in security and nothing in the other areas; compatibility stayed whole (100).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Bugs, Performance, Clean code
- Never fixed, in any run
- SQL injection in the search; Reflected XSS in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); Division by zero in the SLA average; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 1640 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.00Tokens: 19,142 in · 2,689 out · 153,148 cache · 176,902 totalScorecard
-
60
XingChen Xing4.0 29B A4B XingChen · effort high (permissions auto-approved) 1 of 3 runs138 of 1000 Failed
- Security73/250
- Architecture0/200
- Bugs43/150
- Performance56/150
- Clean code0/100
- Compatibility70/100
- Explanation31/50
Comment and details
One run so far: 138, below the pass line, at no cost, through OpenRouter's free variant, in 9 minutes. It found 5 of the 13 planted flaws and fixed 2, but lost 120 points for six broken behaviors no fix was blamed for and 15 for a defect its own change introduced, and one contracted behavior changed (compatibility 70).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Clean code
- Never fixed, in any run
- SQL injection in the search; Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 1382 of 13 flaws fixed1 false positive1 business value changed16/22 checks9minUS$ 0.00Tokens: 42,241 in · 9,597 out · 263,840 cache · 328,709 totalScorecard
-
61
dots.3 Note Preview dots.studio · effort high (permissions auto-approved) 1 of 3 runs86 of 1000 Failed
- Security52/250
- Architecture0/200
- Bugs43/150
- Performance0/150
- Clean code0/100
- Compatibility70/100
- Explanation1/50
Comment and details
One run so far: 86, below the pass line, at no cost, through OpenRouter's free variant, in 22 minutes. It reported none of the 13 planted flaws and its report is nearly empty (1 of 50); four of the flaws are fixed in the code it handed back, but the same changes broke 4 characterization tests (−80) and one contracted behavior (compatibility 70).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Any ticket readable by id (IDOR); File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 864 of 13 flaws fixedno false positive1 business value changed18/22 checks22minUS$ 0.00Tokens: 1,901,025 in · 19,280 out · 4,237,312 cache · 6,183,245 totalScorecard
Flaw by flaw
What each agent found and fixed among the planted flaws. The hard ones are flaws of absence — a missing authorization check, a session never regenerated, a file left open on the error path.
The table shows the top 10 of 61 agents. Every agent's result, flaw by flaw, is in its scorecard.
- fixed
- found, not fixed
- missed
| Flaw | Claude Fable 5.2 | Claude Sonnet 5.5 · xhigh | Claude Sonnet 5.5 · max | Claude Sonnet 5.5 · max (ultracode) | Claude Opus 5.5 · high | Claude Fable 5.1 | Claude Sonnet 5.5 · high | Claude Opus 5.5 · xhigh | Claude Sonnet 5.5 · medium | GPT-6-astra · ultra |
|---|---|---|---|---|---|---|---|---|---|---|
| SQL injection in the search | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed |
| Reflected XSS in the search | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| Formula injection in the CSV export | 6/6fixed | 6/6fixed | 6/6fixed | 6/6fixed | 2/6found, not fixed | 2/6found, not fixed | 6/6fixed | 2/6found, not fixed | 2/6found, not fixed | 6/6fixed |
| Session fixation at login | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| Unsalted MD5 passwords | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 3/8found, not fixed | 8/8fixed | 3/8found, not fixed | 3/8found, not fixed |
| Secrets hardcoded in the config | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| Any ticket readable by id (IDOR) | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 9/10fixed |
| Division by zero in the SLA average | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| File handle leaked on the error path | 6/6fixed | 5/6fixed | 6/6fixed | 4/6fixed | 5/6fixed | 5/6fixed | 5/6fixed | 5/6fixed | 5/6fixed | 4/6fixed |
| One query per ticket for the technician (N+1) | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| A dispatcher that does everything | 4/10found, not fixed | 2/10found, not fixed | 1/10found, not fixed | 0/10missed | 0/10missed | 0/10missed | 0/10missed | 4/10found, not fixed | 0/10missed | 2/10found, not fixed |
| Magic numbers for status and priority | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed |
| Four levels of nested ifs | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 0/8missed | 2/8found, not fixed | 2/8found, not fixed |
What stood out
- Six agents fixed the formula injection in the CSV (SEC-008) — Sonnet 5.5 in all three of its settings, GPT-6.1-sol, GPT-5.6-sol and Grok 4.7 — with the fix the answer key expects; Sonnet 5.5 is the only model with security at 250 of 250, in all three of its settings. GPT-5.6-sol also turned the
-of a ticket with no technician into'-, a new bug (−15). - Architecture was the weakest category for everyone: 50 of 200 for Opus 5.5, 25 for Sonnet 5.5 at xhigh and for GPT-6.1-sol, 12 for Sonnet 5.5 at max, and 0 for the other twenty-eight. Nobody split the dispatcher that does everything; those three named it and declined to restructure it.
- Two published runs broke the contract mechanically. All thirty-seven stayed on mysqli and reported no decoy, and thirty-five kept the 22 characterization checks green; DeepSeek V4 Flash publishes a run whose search throws, and Nex N2.5 Pro one whose functions return nothing without a logged-in user (8 checks broken, −160). Judgement is what separated the rest: twenty-four kept compatibility at 100, while the other thirteen each changed at least one business value (−30 each), most often the SLA average, scoped to each client.
- Twenty-six scores are official: Claude Sonnet 5.5 at xhigh, 809 (runs of 825, 809 and 724), Claude Sonnet 5.5 at max, 807 (807, 773 and 820), Claude Sonnet 5.5 at max in multi-agent mode, 774 (774, 820 and 759), Claude Opus 5.5, 717 (711, 717 and 805), GPT-6.1-sol, 661 (666, 653 and 661), GPT-6.1-sol at ultra, 616 (597, 656 and 616), Grok 4.7, 638 (638, 663 and 607), Grok 4.6, 633 (633, 640 and 620), Grok 4.7 at xhigh, 631 (640, 617 and 631), GPT-6-astra at ultra, 666 (668, 666 and 651), Kimi K3 at high, 629 (629, 634 and 574), GPT-6-astra, 628 (661, 596 and 628), GPT-5.6-terra, 625 (625, 611 and 645), GLM-5.3 Prime, 628 (635, 628 and 541), GLM-5.3, 621 (629, 621 and 604), DeepSeek V4.1 Flash, 612 (625, 612 and 597), GPT-5.6-sol, 612 (612, 612 and 608), GPT-5.6-luna, 601 (599, 601 and 624), Gemini 3.8 Flash at high, 588 (687, 588 and 100), GPT-5.5, 558 (558, 568 and 536), DeepSeek V4 Pro, 496 (604, 496 and 432), MiniMax-M3 with the thinking variant, 462 (462, 616 and 434), Kimi K2.7 Code, 415 (415, 415 and 512), Kimi K2.7 Code highspeed, 402 (402, 363 and 402), Gemini 3.7 Flash at high, 587 (587, 587 and 556), and Claude Haiku 4.5, 317 (317, 369 and 232), each the median of three runs. A single run can sit more than 100 points from the median: Opus's third scored 805, Sonnet's third 724, DeepSeek V4 Pro's first 604, Gemini 3.8 Flash's third, which stopped after ten minutes, 100, and astra's first, 661, had placed it 5th, from judgement calls such as which flaws to leave unfixed. Claude Fable 5.1 has two runs, 781 and 764, and publishes the lower, 4th
- Gemini 3.8 Flash at high is official at 588, 24th, Bronze, the median of three very different runs (687, 588 and 100). Gemini 3.7 Flash, the generation before, is official at 587 (587, 587 and 556), 25th. Its first run, 6th, fixed 8 of the 13 planted flaws and is the only one outside Claude to flatten the nested ifs (CLN-007); its second fixed 7, and searched the VM for the answer key, by name and by the hash quoted in the task. The key is not on the VM, and the search turned up nothing the run did not already have. Its third stopped after ten minutes, on a database server it started that kept its own shell command from ever returning: no report, the code untouched, scored as delivered. The two complete runs kept MD5 and both secrets and left the CSV export serving every client. At medium effort the same model scored 550 (27th) in 4min 34s: it fixed 6 flaws and claimed SQL injection in two functions that only take integers.
- GPT-6-astra at ultra is the strongest GPT agent here, official at 666 (runs of 668, 666 and 651), 6th, with two or three subagents in 10 to 15 minutes. GPT-6.1-sol follows, official at 661 (666, 653 and 661), 7th, and GPT-6.1-sol pro at 654, 8th, from one run. GPT-6-astra at xhigh, official at 628, is 13th. Its first run fixed the CSV injection and kept MD5; its official run did the opposite, migrating MD5 to
password_hashand leaving the CSV injection alone; GPT-6.1-sol's official run fixed both - Multi-agent work matched one agent at best, at many times the cost. Claude Sonnet 5.5 in Claude Code's multi-agent mode, at max effort, scored 774 in its first run (7 workflows, 68 subagents, 7h 55min, US$ 233), 820 in its second (2 workflows, 105 subagents, 2h, US$ 171) and 759 in its third (3 workflows, 145 subagents, 3h 34min, US$ 119). It is official at 774, 3rd, Gold. The same model at the same max effort as a single agent took about half an hour a run and is official at 807 (runs of 807, 773 and 820), 2nd; at xhigh it is official at 809. Its third report scored 47 of 50, the best explanation here, and no run touched the architecture. In each run a few subagents (nine, eight, then seven) fell back to an older Sonnet after a safety classifier stopped them; none of any delivery came from them.
- GPT-6.1-sol at ultra is official 45 points below itself at xhigh (616 against 661), the median of three runs spread over 59 points (597, 656 and 616). With three subagents, its first found 9 of the 13 planted flaws, kept MD5 and did not name the dispatcher; with two, its second fixed 10, the CSV injection and MD5 among them. Each took under half an hour, not the 8 hours of the multi-agent Sonnet.
- Grok 4.7 is the strongest model here from outside Anthropic and OpenAI (638, 9th, Silver, official over three runs of 638, 663 and 607), with an explanation at the level of the best GPT reports (42 of 50). In its first run it is also the only agent that used the web, to read the PHP manual page for fputcsv; the VM blocks GitHub, not the web. At xhigh, Grok 4.7 is official at 631 (640, 617 and 631), 11th, below its own high effort, with reports scored 42 to 44 of 50, the best explanations from outside Anthropic and OpenAI. Grok 4.6 follows, official at 633 (10th; runs of 633, 640 and 620), for about an eighth of the cost (US$ 0.31 against 2.43).
- GLM-5.3 Prime is the strongest GLM model (628, 14th, Silver), official over three runs through two hosts (635, 628 and 541): the first fixed all four probe-covered flaws, the CSV injection among them, for US$ 1.69; the second left the injection alone. GLM-5.3 is official at 621 (17th), the median of three runs (629, 621 and 604), the last two served by another host; GLM-5.3-Flash scores 624 for about a tenth of the cost, and GLM-5.3-FlashX 597
- DeepSeek V4.1 Flash is the cheapest agent here (US$ 0.01 to 0.04 a run; its last two took under two minutes each) and is official at 612 (runs of 625, 612 and 597), 21st. DeepSeek V4 Flash at xhigh, through Novita, scored 617 in its first run (18th). At high, run through two other hosts, it publishes 282, the lower of its two runs (612 and 282), last: the first fixed eight flaws fully, while the second's SQL-injection fix throws on every search and its client CSV comes out on one line. DeepSeek now routes the V4 Flash name to V4.1 on its own API, so which weights those hosts served is not certain. DeepSeek V4 Pro, the larger model, is official at 496, the median of three runs through two hosts (604, 496 and 432), all below DeepSeek V4.1 Flash's official 612: the first rates SQL injection at confidence 100 in two functions that only take integers, and the other two left the N+1 query in place.
- Twenty-four agents did not run at xhigh. Kimi K2.7 Code (in both of Moonshot's ids), Qwen3 Coder Next and Claude Haiku 4.5 have no effort setting and ran at their model's default; the two Grok models, the five GLM models, the three DeepSeek models and Gemini 3.8 and 3.7 Flash and Nex N2.5 Pro ran at high, in opencode, and Gemini 3.8 Flash also at medium; MiniMax-M3 ran in opencode with its thinking variant and Kimi K3 with its high variant; Sonnet 5.5 ran twice at max, once in multi-agent mode, and GPT-6.1-sol and GPT-6-astra once each at ultra. The rest ran at xhigh.
- Qwen3 Coder Next, DeepSeek V4 Pro, MiniMax-M3 with the thinking variant, Kimi K2.7 Code, GPT-5.3-Codex, Kimi K2.7 Code highspeed, GLM-5.2, Nex N2.5 Pro, Claude Haiku 4.5 and DeepSeek V4 Flash close the table (507, 496, 462, 415, 403, 402, 388, 317, 317 and 282; the last four are below the pass line). Nex N2.5 Pro is too slow to finish three runs: 10h 03min and 19h 35min, the longest single-agent runs here, and a third voided after more than 21 hours, so it stays at two; both fixed most of the security flaws, both broke the contract, and it publishes the lower, 317 (317 and 428). Claude Haiku 4.5 is official at 317 (runs of 317, 369 and 232); in the first, 2.5 minutes and US$ 0.21, its visibility fix hides a client's own tickets on the main page, comparing an integer with the string mysqli returns. All but Qwen3 Coder Next left the N+1 query in place. Qwen3 Coder Next has the weakest explanation (18 of 50) and the worst calibration (Brier 0.301): it reported three flaws that cannot exist at confidence 100, SQL injection in two functions that only take integers among them, as Kimi K2.7 Code did. It also ran no tests.
- Places 9 to 23 sit within 41 points (638 to 597), inside the noise of a single run. GPT-5.6-terra reported the fewest planted flaws and still ranks 15th, on compatibility and on what it did fix.
- The GPT models are among the best calibrated (Brier 0.015 or less): fewer findings, each stated with high confidence and nearly all real.
Finding is not fixing
Side by side, the three Claude models find nearly the same flaws and fix different numbers of them. One total hides which of the two it is measuring.
- They find nearly the same. In the run that counts for each, Sonnet 5.5 at xhigh reported 12 of the 13 planted flaws and Fable 5.1 and Opus 5.5 11; across their eight runs, each found 11 or 12
- Sonnet 5.5 fixes more, in the run that counts. Its official run fixed 11 of the 13, against 9 for Opus's official run and 10 for Fable's. Run by run it varies: Sonnet fixed 11, 11 and 8, Opus 9 in each of its three, Fable 9 and 10. Fable and Opus found and explained the formula injection in the CSV and left it in place, to protect the file's consumers; Sonnet fixed it in two of its three runs, prefixing only the cells that start a formula. Its 92-point lead over Opus's official 717 comes from clean code (+100), not from security (+17); its 45-point lead over Fable's 764 is security (+17) and architecture (+25)
- More compute did not change the pattern. The single-agent Claude runs took 16 to 27 minutes each, and the three at the same max effort scored 807, 773 and 820. Sonnet 5.5 in multi-agent mode took 7h 55min, 2h and 3h 34min and US$ 233, 171 and 119 for 774, 820 and 759, and is official at 774 against the single agent's 807: at best it tied the single-agent 820, with the same fixes and a deeper check of its own code, and in none of its three runs did the dispatcher reach its report.
- How to read it. On this task the three Claude models see the same problems; what separates them is how far they go in fixing them within the contract. In the median, Sonnet went further; a single run can reverse it, as Sonnet's third did. One task, two or three runs per model: a pattern worth testing, not a verdict
Read this before quoting a number
- Up to three runs per agent. An official LEB score is the median of three independent runs. Until an agent has three, its score is the lower of its totals so far, the totals are listed next to it, and everything shown with the score comes from that same run.
- The judge is an AI. Claude Opus 5.5 applied the published rubric to every delivery without knowing which model wrote it — each was anonymised — and the explanation was scored by a separate judge that saw neither the answer key nor the other scores.
- The judge is also a contestant. Claude Opus 5.5 is one of the agents evaluated, and six of the thirty-seven are Claude models, Sonnet 5.5 three times: they hold the top five places and the second to last. Anonymity limits that bias; it does not remove it, since a model can recognise its own style. Every verdict is published with its rationale, flaw by flaw, and the four verdicts changed in review say why; one of them adds 41 points to GPT-5.6-terra's total.
- Twenty-one attempts were left unscored, each before any judge saw it. Twice, Claude Fable 5.1's safeguards stopped one of its responses while it worked on the security flaws, and its client handed the rest of the run to Claude Opus 4.8, which wrote the whole report; a run must come from one model, so both are void, and Fable 5.1 shows its two complete runs. Four ended on the provider's side before the delivery was complete: GPT-5.6-sol pro when its gateway ran out of credit, and Gemini 3.8 Flash three times, on a gateway timeout and twice on a rate limit. Five were set up wrong: the first multi-agent Sonnet 5.5 run was stopped to give the machine more processors, the second found its client logged out and never reached the model, as did Claude Fable 5.1's third, Claude Haiku 4.5's first attempt had its client mode changed mid-run, and a Gemini attempt was started outside the task's folder. One finished, but GPT-5.6-terra ran in a different client from its first run, and was repeated in the same one. Six finished after their agent already had its three runs, two of GPT-6.1-sol at xhigh, three at ultra and one of Gemini 3.7 Flash; a fourth run would let the published score be chosen. Two finished, a Grok 4.7 run at xhigh and a Kimi K3 run at high, but their VMs were restored before the deliveries were copied off them, so nothing was left to score. One was cut off by the operator's own connection: Nex N2.5 Pro's third run, after more than 21 hours, which the operator chose not to repeat because the model is too slow. All twenty-one are kept in the repository.
- The answer key is public. The failure matrix of LEB-100-A has been in the public repository since 13 July 2026. The runs of 29 September had the VM's name block only, so no agent could fetch it by accident; the runs of 30 September also had GitHub's addresses blocked, and the session logs of every run that left one show no request to GitHub at all. What a model saw in training depends on how far its training data reaches: each run records the cutoff its provider publishes. Without one, the provider's release date bounds it, since a model cannot train on data from after its release. A model whose cutoff or release is later, or that has neither, is marked in the leaderboard.
- The benchmark's own tests were fixed. Scoring these runs exposed two defects in the evaluation tooling: an SQL loader that split a statement on a semicolon inside a comment, and CSV checks that read a temporary file the contract never promised. Both were fixed before scoring, the same way for every agent, and are recorded in the repository.
- The machine changed on 30 September. Until then the agents ran as the VM's administrator, with sudo, on a machine that kept earlier runs' leftovers. Their session logs show no agent reading another's work; the opencode agents shared a leftover test database that held only the seed rows. From Qwen3 Coder Next on, runs are made as an unprivileged user on a machine cleaned before each run.
- Not every run parameter was recorded. The exact model version and the temperature were not, and cost and model time only where the client kept them; the three runs that left no session log at all (GPT-5.5's first, MiniMax-M3's at its default and Kimi K3's) were withdrawn on 4 October 2026 and are not published. MiniMax-M3 at its default is gone from the table; Kimi K3 is back with a recorded run at high. Each run marks what is missing instead of guessing.
Audit it
Every delivery, mechanical report, verdict and scorecard is in the repository, next to the specification that produced them. The same data is here as two spreadsheets: one row per run, and one per run and planted flaw.
Results on GitHub Specification Runs (CSV) Flaw by flaw (CSV) What each column means
The numbers on this page are not typed by hand: they are rendered from the published results of the ai-benchmark repository, the same data samirhv.com.br shows.