LEB-300
AI Benchmark · LEB · exploratory pilot
The first level, LEB-100, is the reference instance: every agent evaluated so far has run on it, and every run is published.
The second level of LEB: an application of about 3,000 lines, in multiple files, whose flaws cross files. The instance is active, so only the aggregate of each agent's runs is published. Its answer key is public on GitHub with the instance, and the machines that run the agents cannot reach GitHub. How a run works, how it is scored and how this level differs from LEB-100 is on the Benchmark page.
Top 6 so far
LEB-300-A- 1Claude Fable 5.2947LEB Platinum
- 2Claude Sonnet 5.5 · max886LEB Gold
- 3Claude Opus 5.5 · high866LEB Gold
- 4Claude Sonnet 5.5 · max (ultracode)866LEB Gold
- 5Claude Sonnet 5.5 · xhigh860LEB Gold
- 6Claude Sonnet 5.5 · high855LEB Gold
Score out of 1000, of 37 agents. Not official until each has three runs. Full leaderboard
The results so far
One line per agent: the total, the grade, the score in each category, how many runs it rests on, and the cost and the time. Every agent solved the same package, byte for byte, so the numbers compare like for like.
LEB-300-A v1.0 · An application of about 3,000 lines
An exploratory pilot. An official score is the median of three runs of an agent. A line that rests on fewer runs is not official: it says how many it has, and is listed all the same, as on the LEB-100 page. The instance itself is still a pilot: its difficulty has not been homologated. Only the aggregate is published; each run's delivery, verdict and flaw-by-flaw results stay in a private archive. The answer key is public on GitHub with the instance, since 12 October 2026, after the first runs; the machines that ran the agents cannot reach GitHub, and every run records its model's training cutoff.
- mode A · 60 turns
- edition 2026
- key public
- matrix c42c82878d2c…
Showing 37 of 37 agents. Ranks stay those of the full leaderboard.
-
1
Claude Fable 5.2 Anthropic · effort xhigh 1 of 3 runs947 of 1000 LEB Platinum
- Security250/250
- Architecture183/200
- Bugs139/150
- Performance144/150
- Clean code86/100
- Compatibility100/100
- Explanation45/50
Comment and details
One run so far, so no official score yet: 947, Platinum. Security 250 of 250, architecture 183 of 200, bugs 139 of 150, performance 144 of 150 and clean code 86 of 100; compatibility 100, with no penalty. The report covered 33 of the 36 planted flaws and the code fixed 34, with no false positives and 16 extra findings; the report was rated 45 of 50. The session took 110 minutes for US$ 24.81 at the client's list prices: it ran on the owner's personal Claude account (a long-lived token), so that is not a charge.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security, Compatibility
Run by run
- Run 1 · 9471h 49minUS$ 24.81Tokens: 121,689 in · 570,724 out · 4,396,125 cache · 5,088,538 total
-
2
Claude Sonnet 5.5 Anthropic · effort max 3 of 3 runs (932 · 863 · 886)886 of 1000 LEB Gold
- Security250/250
- Architecture150/200
- Bugs137/150
- Performance134/150
- Clean code71/100
- Compatibility100/100
- Explanation44/50
Comment and details
Three runs (932, 863 and 886), so the published score is their median, 886, 69 points between the best and the worst, Platinum, Gold and Gold. Security 250, 232 and 250 of 250, architecture 178, 156 and 150 of 200, bugs 134, 139 and 137 of 150, performance 140, 134 and 134 of 150 and clean code 89, 82 and 71 of 100, in the order of the runs. Compatibility was 100, 75 and 100: run 2 lost 25 points (COMP-008, a change to the public contract). No penalty was charged. The reports covered 31, 30 and 27 of the 36 planted flaws and the code fixed 34, 31 and 32, with 1, 0 and 0 false positives and 22, 22 and 15 extra findings; the reports were rated 41, 45 and 44 of 50. The sessions took 93, 73 and 66 minutes for US$ 10.75, 16.57 and 13.10. For comparison, the same model at medium effort scored 717 (3 runs); the same model at high effort scored 855 (3 runs); the same model at xhigh effort scored 860 (2 runs).
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security, Compatibility
Run by run
- Run 1 · 9321h 32minUS$ 10.75Tokens: 1,390 in · 550,255 out · 15,124,592 cache · 15,676,237 total
- Run 2 · 8631h 14minUS$ 16.57Tokens: 144 in · 637,617 out · 25,936,320 cache · 26,574,081 total
- Run 3 · 8861h 6minUS$ 13.10Tokens: 122 in · 496,801 out · 25,293,464 cache · 25,790,387 total
-
3
Claude Opus 5.5 Anthropic · effort high 3 of 3 runs (848 · 866 · 866)866 of 1000 LEB Gold
- Security229/250
- Architecture153/200
- Bugs147/150
- Performance124/150
- Clean code71/100
- Compatibility100/100
- Explanation42/50
Comment and details
Three runs (848, 866 and 866), so the published score is their median, 866, 18 points between the best and the worst, all Gold. Security 232, 229 and 229 of 250, architecture 153, 153 and 150 of 200, bugs 137, 147 and 147 of 150, performance 134, 124 and 128 of 150 and clean code 71, 71 and 68 of 100, in the order of the runs. Compatibility was 75, 100 and 100: run 1 lost 25 points (COMP-008, a change to the public contract). No penalty was charged. The reports covered 28, 27 and 27 of the 36 planted flaws and the code fixed 32, 33 and 33, with 2, 0 and 0 false positives and 23, 26 and 18 extra findings; the reports were rated 46, 42 and 44 of 50. The sessions took 56, 50 and 43 minutes for US$ 8.77, 7.57 and 7.97. For comparison, the same model at xhigh effort scored 845 (3 runs).
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 84856minUS$ 8.77Tokens: 1,450 in · 148,633 out · 16,974,538 cache · 17,124,621 total
- Run 2 · 86650minUS$ 7.57Tokens: 1,832 in · 135,000 out · 13,004,371 cache · 13,141,203 total
- Run 3 · 86643minUS$ 7.97Tokens: 92 in · 130,598 out · 8,575,765 cache · 8,706,455 total
-
4
Claude Sonnet 5.5 Anthropic · effort max (ultracode) 3 of 3 runs (878 · 866 · 836)866 of 1000 LEB Gold
- Security250/250
- Architecture156/200
- Bugs139/150
- Performance134/150
- Clean code71/100
- Compatibility75/100
- Explanation41/50
Comment and details
Three runs (878, 866 and 836), so the published score is their median, 866, 42 points between the best and the worst, all Gold. Security 250, 250 and 243 of 250, architecture 164, 156 and 128 of 200, bugs 139 in all three of 150, performance 144, 134 and 134 of 150 and clean code 64, 71 and 71 of 100, in the order of the runs. Every run lost 25 compatibility points (75): a login lockout that refuses the right password for 15 minutes, answering 429 in run 2 and 401 in runs 1 and 3. No penalty was charged. The reports covered 28, 29 and 30 of the 36 planted flaws and the code fixed 34, 32 and 30, with 1, 0 and 0 false positives and 33, 35 and 37 extra findings; the reports were rated 42, 41 and 46 of 50. This is Claude Code's multi-agent mode (ultracode): 59, 10 and 13 subagents of the same model. The sessions took 108, 179 and 133 minutes; run 2 was stopped at the 180-minute pool during the second stage, so it is scored on the folder it left and has no cost recorded (US$ 30.42 and 27.82 for the others). The same model at max effort without the mode scored 886 in about 66 minutes.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security
Run by run
- Run 1 · 8781h 48minUS$ 30.42Tokens: 932 in · 1,078,570 out · 55,936,656 cache · 57,016,158 total
- Run 2 · 8662h 59minTokens: 618 in · 506,633 out · 67,698,918 cache · 68,206,169 total
- Run 3 · 8362h 14minUS$ 27.82Tokens: 656 in · 906,672 out · 59,993,597 cache · 60,900,925 total
-
5
Claude Sonnet 5.5 Anthropic · effort xhigh 3 of 3 runs (860 · 911 · 849)860 of 1000 LEB Gold
- Security250/250
- Architecture156/200
- Bugs139/150
- Performance140/150
- Clean code57/100
- Compatibility75/100
- Explanation43/50
Comment and details
Three runs of 860, 911 and 849, so the published score is the middle one, 860, and the agent is now official. The second run is still the highest single run of any agent (911, Platinum). The third fixed 34 of the 36 planted flaws but paid 50 compatibility points for two changes beyond what the manifest contracts, a login lockout that answers 429 and a narrower scope for the idempotency key, against one change in each of the first two (75). Security 232 to 250, architecture 156 to 183, explanation 41 to 43, US$ 6.12 to 8.14 per run. The same model has a median of 855 at high and 886 at max.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security
Run by run
- Run 1 · 86042minUS$ 6.58Tokens: 1,030 in · 244,526 out · 12,893,196 cache · 13,138,752 total
- Run 2 · 91143minUS$ 6.12Tokens: 3,602 in · 262,629 out · 9,421,853 cache · 9,688,084 total
- Run 3 · 84957minUS$ 8.14Tokens: 134 in · 258,037 out · 18,359,976 cache · 18,618,147 total
-
6
Claude Sonnet 5.5 Anthropic · effort high 3 of 3 runs (897 · 855 · 811)855 of 1000 LEB Gold
- Security243/250
- Architecture153/200
- Bugs145/150
- Performance115/150
- Clean code79/100
- Compatibility75/100
- Explanation45/50
Comment and details
Three runs (897, 855 and 811), so the published score is their median, 855, 86 points between the best and the worst, all Gold. Security 250, 243 and 229 of 250, architecture 178, 153 and 117 of 200, bugs 147, 145 and 145 of 150, performance 128, 115 and 121 of 150 and clean code 89, 79 and 54 of 100, in the order of the runs. Compatibility was 75, 75 and 100: run 1 lost 25 points (COMP-008, a change to the public contract); run 3 lost 25 points (COMP-008, a change to the public contract). Penalties: run 1 lost 15 points for a new defect introduced by its own change (PEN-001). The reports covered 32, 27 and 23 of the 36 planted flaws and the code fixed 34, 33 and 31, with 0, 0 and 0 false positives and 21, 18 and 12 extra findings; the reports were rated 45, 45 and 45 of 50. The sessions took 43, 43 and 41 minutes for US$ 5.18, 5.30 and 4.65. For comparison, the same model at medium effort scored 717 (3 runs); the same model at xhigh effort scored 860 (2 runs); the same model at max effort scored 886 (3 runs).
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 89743minUS$ 5.18Tokens: 1,918 in · 165,030 out · 11,795,003 cache · 11,961,951 total
- Run 3 · 85542minUS$ 5.30Tokens: 3,519 in · 174,950 out · 11,635,966 cache · 11,814,435 total
- Run 4 · 81141minUS$ 4.65Tokens: 120 in · 116,889 out · 9,329,232 cache · 9,446,241 total
-
7
Claude Opus 5.5 Anthropic · effort xhigh 3 of 3 runs (886 · 822 · 845)845 of 1000 LEB Gold
- Security232/250
- Architecture153/200
- Bugs139/150
- Performance131/150
- Clean code71/100
- Compatibility75/100
- Explanation44/50
Comment and details
Three runs (886, 822 and 845), so the published score is their median, 845. The shape is the same in all three: 229 to 246 of 250 in security, 144 to 164 of 200 in architecture, 139 to 147 of 150 in bugs, 131 to 144 of 150 in performance and 64 to 71 of 100 in clean code. The first two took the same 30-point compatibility deduction (COMP-003): the idempotency fingerprint now includes the user, so a repeated key from another user of the same company no longer returns the original response, which the manifest requires. The third did not repeat that, but lost 25 points (COMP-008) for a login lockout that answers 429, a status the manifest does not list, even to the right password. It fixed 32 of the 36 planted flaws and reported 28, with no false positive, and its report was rated 44 of 50. The sessions took 82, 48 and 74 minutes for US$ 13.03, US$ 6.77 and US$ 13.69.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 8861h 22minUS$ 13.03Tokens: 1,104 in · 247,375 out · 23,558,840 cache · 23,807,319 total
- Run 2 · 82248minUS$ 6.77Tokens: 1,294 in · 143,108 out · 8,673,335 cache · 8,817,737 total
- Run 3 · 8451h 14minUS$ 13.69Tokens: 1,384 in · 340,120 out · 14,941,510 cache · 15,283,014 total
-
8
Claude Fable 5.1 Anthropic · effort high 1 of 3 runs836 of 1000 LEB Gold
- Security215/250
- Architecture156/200
- Bugs147/150
- Performance128/150
- Clean code71/100
- Compatibility75/100
- Explanation44/50
Comment and details
On a single run: security 215 of 250, architecture 156 of 200, bugs 147 of 150, performance 128 of 150 and 71 of 100 in clean code, with no penalty. It fixed 33 of the 36 planted flaws and its report was rated 44 of 50. It lost 25 compatibility points for one security change that the manifest does not allow: after 10 failed logins for an e-mail, the login answers 429 even to the right password, a status the manifest does not list, and anyone who knows an e-mail can lock that user out. The run cost US$ 15.81 and took 52 minutes of the agent's own time, without the 18 minutes the operator took between the two stages. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 83652minUS$ 15.81Tokens: 2,426 in · 152,548 out · 8,470,393 cache · 8,625,367 total
-
9
GPT-6.1-sol OpenAI · effort xhigh 3 of 3 runs (800 · 830 · 789)800 of 1000 LEB Gold
- Security232/250
- Architecture142/200
- Bugs147/150
- Performance105/150
- Clean code29/100
- Compatibility100/100
- Explanation45/50
Comment and details
Three runs (800, 830 and 789), so the published score is their median, 800, 41 points between the best and the worst, all Gold. Security 232, 232 and 232 of 250, architecture 142, 142 and 125 of 200, bugs 147, 147 and 147 of 150, performance 105, 105 and 93 of 150 and clean code 29, 64 and 64 of 100, in the order of the runs. Compatibility was 100 in every run. Penalties: run 3 lost 15 points for a new defect introduced by its own change (PEN-001). The reports covered 22, 22 and 22 of the 36 planted flaws and the code fixed 31, 33 and 31, with 1, 0 and 0 false positives and 17, 15 and 15 extra findings; the reports were rated 45, 40 and 43 of 50. The sessions took 50, 49 and 43 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 80050minTokens: 310,106 in · 100,213 out · 6,995,200 cache · 7,405,519 total
- Run 2 · 83049minTokens: 217,725 in · 77,782 out · 4,814,592 cache · 5,110,099 total
- Run 3 · 78943minTokens: 190,110 in · 72,326 out · 6,388,864 cache · 6,651,300 total
-
10
GPT-6.1-sol OpenAI · effort ultra 3 of 3 runs (797 · 702 · 791)791 of 1000 LEB Gold
- Security236/250
- Architecture111/200
- Bugs147/150
- Performance109/150
- Clean code46/100
- Compatibility100/100
- Explanation42/50
Comment and details
Three runs (797, 702 and 791), so the published score is their median, 791, 95 points between the best and the worst, Gold, Silver and Gold. Security 232, 194 and 236 of 250, architecture 111, 142 and 111 of 200, bugs 145, 121 and 147 of 150, performance 121, 77 and 109 of 150 and clean code 46, 46 and 46 of 100, in the order of the runs. Compatibility was 100 in every run. Penalties: run 2 lost 20 points for a broken characterization test (PEN-002). The reports covered 25, 23 and 24 of the 36 planted flaws and the code fixed 29, 30 and 29, with 0, 1 and 0 false positives and 24, 18 and 28 extra findings; the reports were rated 42, 42 and 42 of 50. The sessions took 55, 29 and 38 minutes, without the operator's wait between the two stages. Codex keeps no cost. At ultra effort the agent spreads the work over subagent threads of its own; the token totals count every thread together.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 79755minTokens: 712,563 in · 166,543 out · 27,510,912 cache · 28,390,018 total
- Run 2 · 70229minTokens: 431,390 in · 115,151 out · 13,985,920 cache · 14,532,461 total
- Run 4 · 79138minTokens: 445,258 in · 152,169 out · 23,857,152 cache · 24,454,579 total
-
11
GPT-6.1-sol OpenAI · effort high 3 of 3 runs (779 · 682 · 793)779 of 1000 LEB Gold
- Security232/250
- Architecture142/200
- Bugs147/150
- Performance70/150
- Clean code46/100
- Compatibility100/100
- Explanation42/50
Comment and details
Three runs (779, 682 and 793), so the published score is their median, 779, 111 points between the best and the worst, Gold, Silver and Gold. Security 232, 194 and 232 of 250, architecture 142, 142 and 125 of 200, bugs 147, 121 and 147 of 150, performance 70, 86 and 96 of 150 and clean code 46, 32 and 50 of 100, in the order of the runs. Compatibility was 100 in every run. Penalties: run 3 lost 35 points for a broken characterization test (PEN-002); a new defect introduced by its own change (PEN-001). The reports covered 21, 21 and 21 of the 36 planted flaws and the code fixed 30, 31 and 31, with 0, 0 and 0 false positives and 19, 15 and 16 extra findings; the reports were rated 42, 42 and 43 of 50. The sessions took 33, 34 and 44 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 77933minTokens: 354,573 in · 63,518 out · 6,653,440 cache · 7,071,531 total
- Run 3 · 68234minTokens: 165,843 in · 78,006 out · 7,595,520 cache · 7,839,369 total
- Run 4 · 79344minTokens: 176,960 in · 78,060 out · 8,600,064 cache · 8,855,084 total
-
12
Claude Haiku 5.5 Anthropic · effort high 3 of 3 runs (777 · 770 · 713)770 of 1000 LEB Gold
- Security236/250
- Architecture119/200
- Bugs121/150
- Performance115/150
- Clean code64/100
- Compatibility75/100
- Explanation40/50
Comment and details
Three runs (777, 770 and 713), so the published score is their median, 770, 64 points between the best and the worst, Gold, Gold and Silver. Security 236, 236 and 246 of 250, architecture 125, 119 and 86 of 200, bugs 121, 121 and 121 of 150, performance 134, 115 and 128 of 150 and clean code 21, 64 and 14 of 100, in the order of the runs. Compatibility was 100, 75 and 75: run 2 lost 25 points (COMP-008, a change to the public contract); run 3 lost 25 points (COMP-008, a change to the public contract). No penalty was charged. The reports covered 27, 26 and 23 of the 36 planted flaws and the code fixed 26, 28 and 26, with 1, 0 and 1 false positives and 20, 19 and 21 extra findings; the reports were rated 40, 40 and 43 of 50. The sessions took 40, 35 and 36 minutes for US$ 2.07, 1.80 and 1.56. For comparison, the same model at xhigh effort scored 732 (3 runs). Claude Haiku 4.5 scored 242 on the same task.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 77740minUS$ 2.07Tokens: 126 in · 312,281 out · 21,117,926 cache · 21,430,333 total
- Run 2 · 77035minUS$ 1.80Tokens: 146 in · 241,442 out · 17,252,949 cache · 17,494,537 total
- Run 3 · 71336minUS$ 1.56Tokens: 156 in · 239,411 out · 16,719,134 cache · 16,958,701 total
-
13
Claude Haiku 5.5 Anthropic · effort xhigh 3 of 3 runs (708 · 732 · 824)732 of 1000 LEB Silver
- Security250/250
- Architecture100/200
- Bugs121/150
- Performance102/150
- Clean code43/100
- Compatibility75/100
- Explanation41/50
Comment and details
Three runs (708, 732 and 824), so the published score is their median, 732. The third is the best, and the one that differs most: architecture 172 of 200 against 92 and 100, bugs 137 of 150 against 121 and 121, performance 115 of 150 against 96 and 102; security (243 of 250) and clean code (43 of 100) are as in the first run. In every run one security change altered a behaviour the manifest contracts (compatibility 70, 75 and 75). The third run fixed 30 of the 36 planted flaws and reported 30, against 25 fixed in both earlier ones, with no false positive. It used one subagent and took 99 minutes for US$ 5.16, where the first two took 36 and 39 minutes for US$ 1.72 and US$ 1.61. Claude Haiku 4.5 scored 242 on the same task.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security
Run by run
- Run 1 · 70836minUS$ 1.72Tokens: 1,438 in · 252,651 out · 16,402,385 cache · 16,656,474 total
- Run 2 · 73239minUS$ 1.61Tokens: 1,378 in · 326,646 out · 9,253,810 cache · 9,581,834 total
- Run 3 · 8241h 40minUS$ 5.16Tokens: 69,441 in · 578,399 out · 62,347,246 cache · 62,995,086 total
-
14
GPT-6-astra OpenAI · effort ultra 3 of 3 runs (715 · 781 · 721)721 of 1000 LEB Silver
- Security194/250
- Architecture133/200
- Bugs124/150
- Performance102/150
- Clean code46/100
- Compatibility100/100
- Explanation42/50
Comment and details
Three runs (715, 781 and 721), so the published score is their median, 721, 66 points between the best and the worst, Silver, Gold and Silver. Security 194, 232 and 194 of 250, architecture 131, 100 and 133 of 200, bugs 124, 150 and 124 of 150, performance 93, 128 and 102 of 150 and clean code 50, 29 and 46 of 100, in the order of the runs. Compatibility was 100 in every run. Penalties: run 1 lost 20 points for a broken characterization test (PEN-002); run 4 lost 20 points for a broken characterization test (PEN-002). The reports covered 25, 25 and 24 of the 36 planted flaws and the code fixed 29, 28 and 31, with 0, 0 and 1 false positives and 21, 22 and 18 extra findings; the reports were rated 43, 42 and 42 of 50. The sessions took 25, 33 and 27 minutes, without the operator's wait between the two stages. Codex keeps no cost. At ultra effort the agent spreads the work over subagent threads of its own; the token totals count every thread together.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 71525minTokens: 397,360 in · 97,352 out · 12,956,160 cache · 13,450,872 total
- Run 2 · 78133minTokens: 558,132 in · 121,261 out · 18,943,872 cache · 19,623,265 total
- Run 4 · 72127minTokens: 306,601 in · 98,096 out · 13,530,112 cache · 13,934,809 total
-
15
Claude Sonnet 5.5 Anthropic · effort medium 3 of 3 runs (659 · 717 · 734)717 of 1000 LEB Silver
- Security236/250
- Architecture100/200
- Bugs121/150
- Performance99/150
- Clean code50/100
- Compatibility70/100
- Explanation41/50
Comment and details
Three runs (659, 717 and 734), so the published score is their median, 717. Security stayed between 222 and 236 of 250 and bugs between 121 and 137 of 150; performance rose from 77 to 99 to 115 of 150, architecture was 61, 100 and 94 of 200 and clean code 21, 50 and 21 of 100. The runs fixed 23, 28 and 27 of the 36 planted flaws and reported 21, 22 and 25, none with a false positive, and the reports were rated 41, 41 and 42 of 50. In the second run, the median one, a change went beyond what the manifest contracts (the idempotency key now includes the user, so another user of the same company repeating the request gets a 422 instead of the original answer) and cost 30 compatibility points (70, against 100 in the other two). The sessions took 14, 29 and 21 minutes, without the wait between the two stages, for US$ 1.98, US$ 2.96 and US$ 2.25. The same model at high effort has a median of 855 and at xhigh effort a published score of 860 over two runs.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 65914minUS$ 1.98Tokens: 1,364 in · 34,340 out · 3,425,232 cache · 3,460,936 total
- Run 2 · 71729minUS$ 2.96Tokens: 1,794 in · 59,634 out · 5,841,812 cache · 5,903,240 total
- Run 3 · 73421minUS$ 2.25Tokens: 1,272 in · 61,955 out · 4,759,376 cache · 4,822,603 total
-
16
GPT-6-astra OpenAI · effort xhigh 3 of 3 runs (717 · 729 · 680)717 of 1000 LEB Silver
- Security194/250
- Architecture142/200
- Bugs124/150
- Performance86/150
- Clean code50/100
- Compatibility100/100
- Explanation41/50
Comment and details
Three runs (717, 729 and 680), so the published score is their median, 717, 49 points between the best and the worst, all Silver. Security 194, 194 and 194 of 250, architecture 142, 136 and 142 of 200, bugs 124, 124 and 124 of 150, performance 86, 89 and 77 of 150 and clean code 50, 64 and 46 of 100, in the order of the runs. Compatibility was 100, 100 and 75: run 3 lost 25 points (COMP-008, a change to the public contract). Penalties: run 1 lost 20 points for a broken characterization test (PEN-002); run 2 lost 20 points for a broken characterization test (PEN-002); run 3 lost 20 points for a broken characterization test (PEN-002). The reports covered 22, 23 and 22 of the 36 planted flaws and the code fixed 32, 33 and 31, with 0, 0 and 0 false positives and 13, 21 and 18 extra findings; the reports were rated 41, 42 and 42 of 50. The sessions took 28, 42 and 56 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 71728minTokens: 213,017 in · 62,221 out · 6,991,616 cache · 7,266,854 total
- Run 2 · 72942minTokens: 284,263 in · 74,645 out · 7,340,800 cache · 7,699,708 total
- Run 3 · 68056minTokens: 214,526 in · 73,060 out · 6,658,816 cache · 6,946,402 total
-
17
Kimi K3 Moonshot AI · effort high 3 of 3 runs (737 · 712 · 652)712 of 1000 LEB Silver
- Security229/250
- Architecture92/200
- Bugs137/150
- Performance96/150
- Clean code18/100
- Compatibility100/100
- Explanation40/50
Comment and details
Three runs, 737, 712 and 652, so the official score is the median, 712. The first and third went through Novita AI and the second through Moonshot's own API, as in LEB-100; the third is the first made by the unattended lab, with no operator pause between the stages. Security stayed between 218 and 232 of 250 and the contract untouched (compatibility 100); bugs ran 132, 137 and 147 of 150. The spread is architecture and performance: the third run scored 47 of 200 in architecture (92 in both earlier runs) and 77 of 150 in performance (96 before). Clean code ran 43, 18 and 21 of 100. In the first run one finding described a conflict the code cannot produce. Sessions took 69, 71 and 60 minutes (the first without the 10 the operator took between the stages) and cost US$ 4.86, 4.67 and 4.18.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 7371h 9minUS$ 4.86Tokens: 331,676 in · 56,450 out · 7,277,568 cache · 7,721,456 total
- Run 2 · 7121h 11minUS$ 4.67Tokens: 206,664 in · 57,396 out · 8,800,256 cache · 9,100,849 total
- Run 3 · 6521hUS$ 4.18Tokens: 193,532 in · 61,730 out · 7,657,728 cache · 7,937,937 total
-
18
GLM-5.3-Flash Z.AI · effort max 1 of 3 runs693 of 1000 LEB Silver
- Security211/250
- Architecture106/200
- Bugs147/150
- Performance89/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security 211 of 250, bugs 147 of 150, architecture 106 of 200, performance 89 of 150 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 25 of the 36 planted flaws and reported 23, with no false positive; the report was rated 40 of 50. The same model at high effort scored 568 on a run through Z.ai's own API; this one ran at max effort through Novita AI, so the gap of 125 points mixes the effort and the host. The session took 53 minutes, without the 13 minutes the operator took between the two stages, for US$ 0.54 and about 13.4 million tokens. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 69353minUS$ 0.54Tokens: 622,450 in · 63,956 out · 12,599,424 cache · 13,354,809 total
-
19
GLM-5.3 Z.AI · effort high 1 of 3 runs688 of 1000 LEB Silver
- Security222/250
- Architecture69/200
- Bugs147/150
- Performance96/150
- Clean code14/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run, level with the best in bugs (147 of 150): security 222 of 250, performance 96 of 150, architecture 69 of 200 and 14 of 100 in clean code, with the contract untouched. Several of its security and architecture findings were filed as plain bugs or as quality issues, and each costs half of its credit. It ran through OpenRouter, in a 75-minute session for US$ 1.62. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 6881h 15minUS$ 1.62Tokens: 127,024 in · 66,405 out · 17,751,296 cache · 17,994,214 total
-
20
DeepSeek V4.1 Flash DeepSeek · effort high 1 of 3 runs649 of 1000 LEB Silver
- Security229/250
- Architecture47/200
- Bugs137/150
- Performance96/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security at 229 of 250 and bugs at 137 of 150, with the contract untouched (compatibility 100). It left architecture largely alone (47 of 200) and earned nothing in clean code, which is where its points ran out. One of its findings described a problem the code cannot have. Both stages ran in one OpenCode session of 36 minutes for US$ 0.12, by far the cheapest line here. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 64936minUS$ 0.12Tokens: 123,372 in · 47,279 out · 14,006,016 cache · 14,226,586 total
-
21
GLM-5.3-FlashX Z.AI · effort high 1 of 3 runs648 of 1000 LEB Silver
- Security187/250
- Architecture92/200
- Bugs145/150
- Performance77/150
- Clean code32/100
- Compatibility75/100
- Explanation40/50
Comment and details
On a single run: bugs at 145 of 150, security 187 of 250, architecture 92 of 200, performance 77 of 150 and clean code 32 of 100. It added a login lockout (10 failed attempts for an e-mail in 15 minutes, from anyone) that answers 401 even to the right password, which the judge charged as a change of contracted behaviour (compatibility 75 of 100). It ran through Z.ai's own API. The session took 47 minutes and cost US$ 0.96. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 64847minUS$ 0.96Tokens: 198,120 in · 54,593 out · 10,246,592 cache · 10,540,100 total
-
22
GPT-5.6-terra OpenAI · effort xhigh 3 of 3 runs (625 · 574 · 657)625 of 1000 LEB Silver
- Security225/250
- Architecture47/200
- Bugs134/150
- Performance96/150
- Clean code0/100
- Compatibility100/100
- Explanation38/50
Comment and details
Three runs from 574 to 657, and the official 625 is the middle one. Security was 225, 225 and 246 of 250, architecture 47 of 200 in all three, clean code 0, 0 and 18 of 100, bugs 134, 137 and 142 of 150 and performance 96, 41 and 64 of 150, with the contract untouched in every run (compatibility 100). The first two took the 15-point penalty for a fix that introduced a new bug; the third took none. The sessions took 32, 28 and 32 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 62532minTokens: 229,711 in · 55,406 out · 8,839,936 cache · 9,125,053 total
- Run 2 · 57428minTokens: 238,473 in · 55,402 out · 7,062,528 cache · 7,356,403 total
- Run 3 · 65732minTokens: 214,606 in · 52,932 out · 7,931,904 cache · 8,199,442 total
-
23
GPT-5.6-luna OpenAI · effort xhigh 3 of 3 runs (638 · 512 · 618)618 of 1000 LEB Silver
- Security211/250
- Architecture47/200
- Bugs142/150
- Performance80/150
- Clean code14/100
- Compatibility100/100
- Explanation39/50
Comment and details
Three runs (638, 512 and 618), so the published score is their median, 618, 126 points between the best and the worst, Silver, Bronze and Silver. Security 211, 173 and 211 of 250, architecture 47, 67 and 47 of 200, bugs 147, 121 and 142 of 150, performance 77, 57 and 80 of 150 and clean code 18, 0 and 14 of 100, in the order of the runs. Compatibility was 100, 75 and 100: run 2 lost 25 points (COMP-008, a change to the public contract). Penalties: run 2 lost 20 points for a broken characterization test (PEN-002); run 3 lost 15 points for a new defect introduced by its own change (PEN-001). The reports covered 20, 18 and 18 of the 36 planted flaws and the code fixed 22, 23 and 23, with 0, 1 and 0 false positives and 9, 6 and 10 extra findings; the reports were rated 38, 39 and 39 of 50. The sessions took 42, 38 and 34 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 63842minTokens: 449,393 in · 80,165 out · 19,209,472 cache · 19,739,030 total
- Run 2 · 51238minTokens: 392,754 in · 73,502 out · 13,548,032 cache · 14,014,288 total
- Run 3 · 61834minTokens: 530,870 in · 78,034 out · 22,673,536 cache · 23,282,440 total
-
24
GPT-6-luna OpenAI · effort xhigh 3 of 3 runs (471 · 612 · 630)612 of 1000 LEB Silver
- Security236/250
- Architecture47/200
- Bugs137/150
- Performance51/150
- Clean code0/100
- Compatibility100/100
- Explanation41/50
Comment and details
Three runs (471, 612 and 630), so the published score is their median, 612, 159 points between the best and the worst, Bronze, Silver and Silver. Security 180, 236 and 243 of 250, architecture 47, 47 and 47 of 200, bugs 124, 137 and 111 of 150, performance 70, 51 and 89 of 150 and clean code 0, 0 and 0 of 100, in the order of the runs. Compatibility was 45, 100 and 100: run 1 lost 55 points (COMP-008, a change to the public contract, COMP-003, ). Penalties: run 1 lost 35 points for a broken characterization test (PEN-002); a new defect introduced by its own change (PEN-001). The reports covered 21, 20 and 20 of the 36 planted flaws and the code fixed 22, 19 and 21, with 2, 0 and 0 false positives and 9, 7 and 10 extra findings; the reports were rated 40, 41 and 40 of 50. The sessions took 34, 17 and 31 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 47134minTokens: 325,721 in · 99,556 out · 8,990,976 cache · 9,416,253 total
- Run 2 · 61217minTokens: 288,934 in · 92,450 out · 5,842,048 cache · 6,223,432 total
- Run 3 · 63031minTokens: 334,449 in · 118,381 out · 7,602,944 cache · 8,055,774 total
-
25
GLM-5.3-Flash Z.AI · effort high 1 of 3 runs568 of 1000 LEB Bronze
- Security204/250
- Architecture47/200
- Bugs126/150
- Performance51/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security 204 of 250, bugs 126 of 150, performance 51 of 150, architecture 47 of 200 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 17 of the 36 planted flaws and reported 20, with no false positive; the report was rated 40 of 50, weakest on prioritisation, because the order of the findings contradicts its own severity labels. It ran through Z.ai's own API; the session took 46 minutes, without the nearly two hours the operator took between the two stages, for US$ 0.31 and about 7.7 million tokens. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 56846minUS$ 0.31Tokens: 333,659 in · 52,805 out · 7,231,040 cache · 7,652,156 total
-
26
MiniMax-M3 MiniMax · effort thinking 3 of 3 runs (388 · 398 · 810)398 of 1000 Failed
- Security169/250
- Architecture25/200
- Bugs42/150
- Performance26/150
- Clean code0/100
- Compatibility100/100
- Explanation36/50
Comment and details
Three runs, and the third is far from the other two: 388 and 398 (Reprovada) against 810 (Gold), so the official score, the median, is 398 and the configuration is not steady. Compatibility was whole in all three. In the first two, the first-stage report filed several security and architecture problems as plain bugs and never moved them (security 113 of 250, bugs 47 of 150); in the third it detected 23 of the 36 planted flaws and fixed 13, with security 225, architecture 136, bugs 145, performance 112 and clean code 50, and a report rated 42 of 50. The runs took 42, 53 and 105 minutes, at US$ 1.21, 0.81 and 4.12.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 38842minUS$ 1.21Tokens: 128,455 in · 48,678 out · 15,344,274 cache · 15,575,334 total
- Run 2 · 39853minUS$ 0.81Tokens: 282,942 in · 41,745 out · 10,646,261 cache · 11,004,186 total
- Run 3 · 8101h 45minUS$ 4.12Tokens: 591,588 in · 121,599 out · 35,526,597 cache · 36,384,563 total
-
27
GLM-5.2 Z.AI · effort high 1 of 3 runs390 of 1000 Failed
- Security151/250
- Architecture39/200
- Bugs63/150
- Performance51/150
- Clean code0/100
- Compatibility45/100
- Explanation41/50
Comment and details
Explanation scored 41 of 50, but the code lost ground: security 151 of 250, bugs 63 of 150, performance 51 of 150 and nothing in clean code. Two of its changes broke contracts of the manifest, which cost 55 of the 100 compatibility points, the lowest on the board. It ran through Novita AI, in a 21-minute session for US$ 3.14. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Clean code
Run by run
- Run 1 · 39021minUS$ 3.14Tokens: 625,652 in · 33,738 out · 7,828,480 cache · 8,505,204 total
-
28
NVIDIA Nemotron 3.5 Super VL NVIDIA · effort high (permissions auto-approved) 1 of 3 runs367 of 1000 Failed
- Security120/250
- Architecture19/200
- Bugs105/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation38/50
Comment and details
On a single run: compatibility whole (100), security 120 of 250, bugs 105 of 150 and architecture 19 of 200, nothing in performance or clean code, and 15 points lost for a new defect in its own fix. It detected 12 of the 36 planted flaws and fixed 6; it reported 2 more real findings outside the planted set, and its confidence was well calibrated (Brier 0.08). The report was rated 38 of 50. It ran through OpenRouter's free variant, at no cost and in 20 minutes, at high, the highest effort the client defines for it. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Performance, Clean code
Run by run
- Run 1 · 36720minUS$ 0.00Tokens: 581,974 in · 28,073 out · 6,144,112 cache · 6,781,516 total
-
29
NVIDIA Nemotron 3 Ultra NVIDIA · effort high (permissions auto-approved) 1 of 3 runs326 of 1000 Failed
- Security74/250
- Architecture47/200
- Bugs24/150
- Performance70/150
- Clean code0/100
- Compatibility75/100
- Explanation36/50
Comment and details
On a single run: compatibility 75 of 100 (one login rule of the contract changed), security 74 of 250, architecture 47 of 200, bugs 24 of 150 and performance 70 of 150, nothing in clean code. It detected 9 of the 36 planted flaws and fixed 4; it also reported 6 real findings outside the planted set, and its confidence was well calibrated (Brier 0.07). The report was rated 36 of 50. It ran through OpenRouter's free variant, at no cost and in 28 minutes, at high, the highest effort the client defines for it. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Clean code
Run by run
- Run 1 · 32628minUS$ 0.00Tokens: 760,068 in · 52,222 out · 12,511,936 cache · 13,331,009 total
-
30
MiniMax-M2.7-highspeed MiniMax · default effort (not configurable) 1 of 3 runs291 of 1000 Failed
- Security77/250
- Architecture22/200
- Bugs11/150
- Performance51/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
On a single run: security 77 of 250, bugs 11 of 150, performance 51 of 150, architecture 22 of 200 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It made a few small, targeted edits and fixed 7 of the 36 planted flaws; it reported 8 of them (one of those left unfixed) and one false positive, a counting bug the code does not have. The report was rated 30 of 50, weakest on prioritisation and trade-offs: its two critical findings come 7th and 8th in the list, and its table of decisions rarely compares alternatives. MiniMax-M2.7-highspeed is the M2.7 series served faster (MiniMax: "same performance, faster"), and OpenCode offers no effort setting for it. The agent wrote RELATORIO.md and achados.json inside code/ instead of the project root, and the delivery was accepted as made. Both stages ran in one OpenCode session of 10 minutes, without the 7 minutes the operator took between the two stages, through MiniMax's own API, for US$ 0.35. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 29110minUS$ 0.35Tokens: 59,631 in · 15,697 out · 3,198,034 cache · 3,280,847 total
-
31
Crouton Alpha Stealth · effort max (permissions auto-approved) 1 of 3 runs275 of 1000 Failed
- Security67/250
- Architecture0/200
- Bugs100/150
- Performance0/150
- Clean code0/100
- Compatibility70/100
- Explanation38/50
Comment and details
On a single run: security 67 of 250 and bugs 100 of 150, nothing in architecture, performance or clean code, and compatibility 70 of 100 (one rule of the contract changed without being reported). It detected 11 of the 36 planted flaws and fixed 7; no decoy was reported as a flaw, one finding of its own was real but outside the planted set, and its confidence was well calibrated. The report was rated 38 of 50, weakest on prioritisation. It ran through OpenRouter's free variant of a stealth model (its maker is withheld), at no cost and in 21 minutes; a first attempt ended on a provider error after 146 minutes and is withdrawn. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Architecture, Performance, Clean code
Run by run
- Run 2 · 27521minUS$ 0.00Tokens: 23,998,526 in · 14,387 out · 0 cache · 24,015,200 total
-
32
Claude Haiku 4.5 Anthropic · default effort (not configurable) 3 of 3 runs (242 · 218 · 272)242 of 1000 Failed
- Security56/250
- Architecture25/200
- Bugs29/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation32/50
Comment and details
Three runs from 218 to 272, and the published 242 is the middle one. Two of the three changed behaviour that the manifest contracts and lost 30 compatibility points; the published run did not. Its security score was 56 of 250 in the published run and above 100 in the other two, while performance and clean code stayed near zero throughout. The published run was the fastest line here, 12 minutes; the other two took 28 and 44, and the second cost US$ 2.22. All three ran in Claude Code.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Performance, Clean code
Run by run
- Run 1 · 24212minUS$ 0.96Tokens: 1,476 in · 32,117 out · 6,064,686 cache · 6,098,279 total
- Run 2 · 21844minUS$ 2.22Tokens: 16,438 in · 88,702 out · 13,017,862 cache · 13,123,002 total
- Run 3 · 27228minUS$ 1.36Tokens: 2,073 in · 43,806 out · 9,106,328 cache · 9,152,207 total
-
33
Gemma 4 26B A4B Google DeepMind · effort high 1 of 3 runs232 of 1000 Failed
- Security63/250
- Architecture14/200
- Bugs8/150
- Performance16/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
On a single run: the contract stayed whole (compatibility 100), but little else moved: security 63 of 250, architecture 14 of 200, bugs 8 of 150, performance 16 of 150 and nothing in clean code. It fixed 3 of the 36 planted flaws and reported 5, all of them real and none a false positive, each with full confidence; the report was rated 31 of 50, weakest on prioritisation and on trade-offs. The run cost US$ 0.36 and took 14 minutes of the agent's own time, without the 7 minutes the operator took between the two stages. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 23214minUS$ 0.36Tokens: 2,731,666 in · 20,301 out · 0 cache · 2,752,166 total
-
34
MiniMax-M2.7 MiniMax · default effort (not configurable) 1 of 3 runs202 of 1000 Failed
- Security85/250
- Architecture0/200
- Bugs8/150
- Performance0/150
- Clean code0/100
- Compatibility75/100
- Explanation34/50
Comment and details
On a single run: security 85 of 250, bugs 8 of 150, and nothing in architecture, performance or clean code, with compatibility at 75 of 100 for one violation kept as judged: the login rate limiter it added answers 401 even to the right password after five attempts in 15 minutes from the same non-local address (COMP-008). It fixed 3 of the 36 planted flaws and reported 5 (two of those left unfixed), with one false positive, a counting bug the code does not have, and one extra finding. The report was rated 34 of 50, weakest on precision, because it contradicts itself in places (a token valid "until expiry" and then "indefinitely") and overstates how long a brute-force attack would take. MiniMax-M2.7 is the standard-speed model of the series, and OpenCode offers no effort setting for it. Both stages ran in one OpenCode session of 15 minutes, without the 7 minutes the operator took between the two stages, through MiniMax's own API, for US$ 0.44. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Architecture, Performance, Clean code
Run by run
- Run 1 · 20215minUS$ 0.44Tokens: 76,199 in · 15,764 out · 5,219,356 cache · 5,319,203 total
-
35
NVIDIA Nemotron 3.5 Lightning NVIDIA · effort high (permissions auto-approved) 1 of 3 runs175 of 1000 Failed
- Security39/250
- Architecture0/200
- Bugs5/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
On a single run: compatibility whole (100), security 39 of 250 and bugs 5 of 150, nothing else. It detected 3 of the 36 planted flaws and fixed none; its three findings were all real and its confidence was well calibrated (Brier 0.001), but it did little of the work. The report was rated 31 of 50. It ran through OpenRouter's free variant, at no cost and in 14 minutes, at high, the highest effort the client defines for it. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
Run by run
- Run 1 · 17514minUS$ 0.00Tokens: 285,470 in · 11,957 out · 4,397,408 cache · 4,715,841 total
-
36
NVIDIA Nemotron 3 Nano Omni NVIDIA · effort max (permissions auto-approved) 1 of 3 runs148 of 1000 Failed
- Security18/250
- Architecture0/200
- Bugs0/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
On a single run: the contract stayed whole (compatibility 100), but almost nothing else moved: security 18 of 250 and nothing in architecture, bugs, performance or clean code. It detected 1 of the 36 planted flaws and fixed it; it reported 3 findings, each with high confidence, and two of them were not real flaws (no decoy among them), so its confidence was poorly calibrated. The report was rated 30 of 50, weakest on trade-offs. It ran through OpenRouter's free variant, at no cost and in 9 minutes, at max, the highest effort the client defines for it. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Architecture, Bugs, Performance, Clean code
Run by run
- Run 1 · 1489minUS$ 0.00Tokens: 91,719 in · 5,148 out · 1,502,704 cache · 1,620,918 total
-
37
Cohere North Mini Code Cohere · effort high (permissions auto-approved) 1 of 3 runs0 of 1000 Failed
- Security25/250
- Architecture0/200
- Bugs0/150
- Performance10/150
- Clean code0/100
- Compatibility100/100
- Explanation19/50
Comment and details
On a single run, the lowest possible score: 0. Its changes broke 70 of the characterization tests that guard the legacy behavior (70 × −20 = −1,400 points, which the score floors at zero), although the contract itself counted as whole (compatibility 100). It detected 3 of the 36 planted flaws and fixed none; it reported 9 findings and its confidence was poorly calibrated (Brier 0.52, three of four high-confidence findings were not real). Security 25 of 250 and performance 10 of 150, nothing else; the report was rated 19 of 50. It ran through OpenRouter's free variant, at no cost and in 4 minutes, at high, the highest effort the client defines for it. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Architecture, Bugs, Clean code
Run by run
- Run 1 · 04minUS$ 0.00Tokens: 2,843,166 in · 20,581 out · 0 cache · 2,871,222 total
What the record does not have
- No checkpoint was taken between the two stages of the task in any of the runs, so it is not on record that the model stayed the same through the first stage. The transcripts show a single model.
- The task text the agent read names the previous scoring-matrix hash. The matrix this run is scored against differs from it only by a header field that marks the instance as active. The scoring is the same.
- The agents did not all run in the same client, Claude Code or OpenCode, each with its own tools. The client of every run is in its record.
What is published
This page shows one line per agent: the total, the grade, the score in each category, how many runs it rests on, and the cost and the time. It never shows a run's delivery, its verdict or its flaw-by-flaw results, which stay in a private archive. The instance and its answer key are public on GitHub; the machines that run the agents cannot reach GitHub.
The instance is in an exploratory pilot. The results above are the ones published so far, and other agents follow. Its first runs also served to check the procedure.
The protocol, the scoring and the tools are public in the ai-benchmark repository. How a run works and how the two levels differ is on the Benchmark page, and the results of the first level are on the LEB-100 page.