LEB Default

One run of a case of five scored 880 (Gold). It completed inside the 90-minute budget, at about US$ 27.33 and 70 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (209 of 213), then performance (208 of 213), clean code (116 of 141) and architecture (202 of 283). The report was rated 45 of 50. It ran on the owner's personal Claude account (a long-lived token), so its cost is the client's list price, not a charge. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 880.0 (880.0 to 880.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 813 and 858, mean 835 with a 95% interval of 822 to 849, all Gold. All five completed inside the 90-minute budget, at about US$ 7.58 and 32 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (206 of 213 on average), then performance (192 of 213), architecture (200 of 283) and clean code (95 of 141). The reports were rated 43 to 45 of 50. Penalties: runs 2 and 3 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Runs 3, 4, 5 ran on an Anthropic API key, billed to API credit, the others on a subscription token; the model and the client are the same.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 835.2 (822.2 to 849.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 785 and 874, mean 826 with a 95% interval of 797 to 855, all Gold. All five completed inside the 90-minute budget, at about US$ 23.32 and 71 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (193 of 213), architecture (190 of 283) and clean code (89 of 141). The reports were rated 44 to 45 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Runs 2, 3, 4, 5 ran on an Anthropic API key, billed to API credit, the other on a subscription token; the model and the client are the same.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 825.6 (796.6 to 854.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs of a case of five, between 544 and 815. The headline, 815 (Gold), is the one run that finished inside the 90-minute budget (61 minutes, US$ 31.01). The other four were stopped at the budget and are scored on the folder each left: 573, 544, 812 and 555, and only the third delivered a report, three minutes before the cut. So one attempt in five completed, and the expected score per scheduled attempt is 660, with an interval of 554 to 765. This is Claude Code's multi-agent mode (ultracode), switched on with /effort ultracode at the start of the session: the runs used 16, 26, 31, 13 and 126 subagents of the same model, in the order of the runs. No cost is recorded for the four stopped runs. Runs 2 and 3 ran on an Anthropic API key, the others on a Claude subscription token. It is one system, so the interval only reflects the variation among the runs on it. The blind judges agreed on almost every item, and a person settled the rest.

• 1 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 0 of 1.
• Expected score per scheduled attempt: 659.8 (554.2 to 765.4).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 4. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 752 and 836, mean 785 with a 95% interval of 764 to 812, all Gold. All five completed inside the 90-minute budget, at about US$ 5.78 and 35 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (202 of 213 on average), then performance (176 of 213), clean code (88 of 141) and architecture (176 of 283). The reports were rated 45 of 50. Penalties: run 3 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 785.2 (764.0 to 812.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 625 and 812, mean 732 with a 95% interval of 677 to 789, two Gold and three Silver. All five completed inside the 90-minute budget, at about US$ 3.97 and 22 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (173 of 213 on average), then performance (173 of 213), architecture (166 of 283) and clean code (79 of 141). The reports were rated 43 to 44 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 731.6 (676.8 to 789.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 672 and 776, mean 726 with a 95% interval of 698 to 754, one Gold and four Silver. All five completed inside the 90-minute budget, about 4.6 million tokens and 25 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (183 of 213), architecture (150 of 283) and clean code (46 of 141). The reports were rated 42 to 43 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change; runs 4 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 725.6 (698.0 to 753.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 682 and 748, mean 718 with a 95% interval of 695 to 740, all Silver. All five completed inside the 90-minute budget, at about US$ 2.79 (the client's own estimate) and 40 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (185 of 213 on average), then performance (157 of 213), architecture (192 of 283) and clean code (46 of 141). The reports were rated 38 to 41 of 50. Penalties: run 7 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 718.2 (694.8 to 740.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 20.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 694 and 734, mean 713 with a 95% interval of 700 to 728, all Silver. All five completed inside the 90-minute budget, at about US$ 2.29 and 20 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (199 of 213 on average), then performance (173 of 213), architecture (138 of 283) and clean code (66 of 141). The reports were rated 40 to 43 of 50. Penalties: runs 1, 2, 3 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 713.4 (699.8 to 727.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 629 and 749, mean 698 with a 95% interval of 662 to 734, all Silver. All five completed inside the 90-minute budget, about 11.1 million tokens and 16 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (194 of 213), architecture (142 of 283) and clean code (20 of 141). The reports were rated 41 to 45 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change; runs 1, 2, 3, 4 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes). Ultra is the client's multi-agent effort: a main agent and subagents on the same model, and the tokens counted are those of all of them.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 698.4 (662.2 to 734.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 650 and 718, mean 684 with a 95% interval of 664 to 704, all Silver. All five completed inside the 90-minute budget, about 6.0 million tokens and 24 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (184 of 213), architecture (130 of 283) and clean code (29 of 141). The reports were rated 41 to 43 of 50. Penalties: runs 1 and 4 lost 15 points each for a new defect introduced by its own change; runs 1, 2, 3, 4 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 683.6 (664.0 to 704.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 607 and 694, mean 661 with a 95% interval of 631 to 689, all Silver. All five completed inside the 90-minute budget, about 5.3 million tokens and 17 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (203 of 213 on average), then performance (181 of 213), architecture (128 of 283) and clean code (16 of 141). The reports were rated 41 to 44 of 50. Penalties: runs 1 and 2 lost 15 points each for a new defect introduced by its own change; runs 2, 3 and 4 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 661.4 (631.2 to 688.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 622 and 648, mean 638 with a 95% interval of 630 to 646, all Silver. All five completed inside the 90-minute budget, about 15.3 million tokens and 17 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (189 of 213), architecture (91 of 283) and clean code (12 of 141). The reports were rated 41 to 44 of 50. Penalties: runs 1, 2, 3, 4 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes). Ultra is the client's multi-agent effort: a main agent and subagents on the same model, and the tokens counted are those of all of them.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 638.2 (629.6 to 646.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 579 and 627, mean 605 with a 95% interval of 588 to 622, three Silver and two Bronze. All five completed inside the 90-minute budget, at about US$ 7.41 (the client's own estimate) and 39 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (206 of 213 on average), then performance (157 of 213), architecture (101 of 283) and clean code (0 of 141). The reports were rated 40 to 44 of 50. Penalties: runs 2 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, whose only upstream for this model is xAI's own.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 605.2 (588.2 to 622.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 526 and 696, mean 602 with a 95% interval of 553 to 650, two Silver and three Bronze. All five completed inside the 90-minute budget, at about US$ 2.55 and 26 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (186 of 213 on average), then performance (142 of 213), architecture (131 of 283) and clean code (2 of 141). The reports were rated 43 to 45 of 50. Penalties: run 4 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Runs 2, 3, 4, 5 ran on an Anthropic API key, billed to API credit, the other on a subscription token; the model and the client are the same.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 601.6 (553.2 to 650.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 541 and 654, mean 598 (the interval, 541 to 654, is the range of the two), one Silver and one Bronze. Both completed inside the 90-minute budget, at about US$ 3.01 (the client's own estimate) and 39 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Performance was the strongest area (170 of 213 on average), then bug fixing (159 of 213), architecture (109 of 283) and clean code (25 of 141). The reports were rated 42 to 43 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 597.5 (541.0 to 654.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 556 and 596, mean 576 (the interval, 556 to 596, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 8.54 (the client's own estimate) and 40 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (200 of 213 on average), then performance (145 of 213), architecture (105 of 283) and clean code (0 of 141). The reports were rated 40 to 42 of 50. Penalties: runs 1 and 2 lost 15 points each for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 576.0 (556.0 to 596.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 525 and 614, mean 560 with a 95% interval of 532 to 593, one Silver and four Bronze. All five completed inside the 90-minute budget, at about US$ 1.21 and 17 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (183 of 213 on average), then performance (119 of 213), architecture (85 of 283) and clean code (38 of 141). The reports were rated 39 to 42 of 50. Penalties: runs 1, 2, 3 and 4 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Every run ran on an Anthropic API key, billed to API credit, where the first Claude configurations ran on a subscription token; the model and the client are the same.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 560.4 (531.6 to 593.4).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 523 and 588, mean 556 (the interval, 523 to 588, is the range of the two), all Bronze. Both completed inside the 90-minute budget, about 8.2 million tokens and 13 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (200 of 213 on average), then performance (158 of 213), architecture (61 of 283) and clean code (0 of 141). The reports were rated 40 to 41 of 50. Penalties: runs 1 and 2 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 555.5 (523.0 to 588.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 487 and 597, mean 549 with a 95% interval of 511 to 588, all Bronze. All five completed inside the 90-minute budget, at about US$ 0.14 (the client's own estimate) and 44 minutes each, and compatibility was between 75 and 100. Bug fixing was the strongest area (178 of 213 on average), then performance (147 of 213), architecture (91 of 283) and clean code (0 of 141). The reports were rated 39 to 44 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 549.4 (510.6 to 588.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 524 and 580, mean 547 with a 95% interval of 530 to 565, all Bronze. All five completed inside the 90-minute budget, at about US$ 1.13 and 27 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (173 of 213 on average), then performance (125 of 213) and architecture (96 of 283); clean code scored 10 of 141 on average, with two runs at 0. The reports were rated 41 to 43 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 546.8 (530.0 to 565.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 71dfeb7c3f82c75a96f3134e4e050b4476436116e14402d483e9c5f5510955e4.

One run of a case of five scored 547 (Bronze). It completed inside the 90-minute budget, at about US$ 0.08 (the client's own estimate) and 60 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (159 of 213), then performance (104 of 213), clean code (60 of 141) and architecture (81 of 283). The report was rated 43 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, pinned to Xiaomi's own endpoint (fp8 quantization), because the router otherwise spreads the model over other hosts; the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 547.0 (547.0 to 547.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 519 and 606, mean 540 with a 95% interval of 521 to 574, one Silver and four Bronze. All five completed inside the 90-minute budget, about 6.4 million tokens and 16 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (202 of 213 on average), then performance (142 of 213), architecture (54 of 283) and clean code (0 of 141). The reports were rated 40 to 44 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 540.2 (521.0 to 573.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 519 and 566, mean 538 with a 95% interval of 526 to 553, all Bronze. All five completed inside the 90-minute budget, at about US$ 0.15 (the client's own estimate) and 49 minutes each, and compatibility was between 75 and 100. Bug fixing was the strongest area (188 of 213 on average), then performance (144 of 213), architecture (70 of 283) and clean code (0 of 141). The reports were rated 40 to 43 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). DeepSeek's API reports only the alias deepseek-flash: the 1017 build is what the lab was told it pointed to.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 538.0 (525.6 to 553.4).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 511 and 536, mean 521 with a 95% interval of 513 to 530, all Bronze. All five completed inside the 90-minute budget, at about US$ 2.45 (the client's own estimate) and 37 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (154 of 213 on average), then performance (154 of 213), architecture (68 of 283) and clean code (6 of 141). The reports were rated 38 to 39 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 521.0 (512.6 to 530.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 422 and 575, mean 519 with a 95% interval of 465 to 567, all Bronze. All five completed inside the 90-minute budget, at about US$ 0.09 (the client's own estimate) and 30 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (176 of 213 on average), then performance (131 of 213), architecture (69 of 283) and clean code (2 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Some runs ran with the client's default permissions and some with its auto-approval flag (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 519.0 (464.8 to 566.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 499 and 526, mean 512 (the interval, 499 to 526, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 4.09 (the client's own estimate) and 52 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (166 of 213 on average), then performance (104 of 213), architecture (101 of 283) and clean code (0 of 141). The reports were rated 41 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 512.5 (499.0 to 526.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Three runs of a case of five, between 442 and 562, mean 511 (the interval, 442 to 562, is the range of the three), all Bronze. All three completed inside the 90-minute budget, at about US$ 2.90 (the client's own estimate) and 52 minutes each, and compatibility was between 85 and 100. Bug fixing was the strongest area (188 of 213 on average), then performance (133 of 213), architecture (61 of 283) and clean code (0 of 141). The reports were rated 40 to 43 of 50. Penalties: run 3 lost 15 points for a new defect introduced by its own change; run 3 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other two are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It ran through the Kimi Code membership, where the client's cost is the list price, not a charge.

• 3 of 3 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 511.3 (442.0 to 562.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 3.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 509 (Bronze). It completed inside the 90-minute budget, at about US$ 1.56 (the client's own estimate) and 29 minutes, and it did not break a behavior the public tests protect (compatibility 100). Performance was the strongest area (159 of 213), then bug fixing (137 of 213), architecture (69 of 283) and clean code (0 of 141). The report was rated 44 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, pinned to Xiaomi's own endpoint (fp8 quantization), the only upstream of this model; the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 509.0 (509.0 to 509.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 451 and 578, mean 504 with a 95% interval of 462 to 547, all Bronze. All five completed inside the 90-minute budget, at about US$ 2.60 (the client's own estimate) and 40 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (161 of 213 on average), then performance (104 of 213), architecture (92 of 283) and clean code (4 of 141). The reports were rated 42 to 44 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). Runs 3 and 4 went through Moonshot's platform API and runs 5, 7 and 8 through the Kimi Code membership, another route to the same model; the owner kept them in one configuration, with the route recorded in each run.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 504.2 (461.8 to 546.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 3.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 462 and 494, mean 478 (the interval, 462 to 494, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 0.55 (the client's own estimate) and 46 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (134 of 213 on average), then performance (126 of 213), architecture (77 of 283) and clean code (0 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was served by Novita AI, the host the first series used for this model, in its 0731 version; the client names no xhigh for it, and DeepSeek treats xhigh as max.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 478.0 (462.0 to 494.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 427 and 543, mean 477 with a 95% interval of 441 to 515, all Bronze. All five completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 45 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (133 of 213 on average), then performance (127 of 213), architecture (69 of 283) and clean code (6 of 141). The reports were rated 41 to 44 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant of the model, whose only endpoint is MiniMax's own, so the runs cost nothing; that variant no longer exists.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 476.8 (441.0 to 514.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 4.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 423 and 523, mean 473 with a 95% interval of 441 to 505, all Bronze. All five completed inside the 90-minute budget, at about US$ 0.34 (the client's own estimate) and 32 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (150 of 213 on average), then performance (109 of 213), architecture (73 of 283) and clean code (0 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 472.8 (440.6 to 505.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 423 and 510, mean 466 (the interval, 423 to 510, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 49 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (148 of 213 on average), then performance (124 of 213), architecture (54 of 283) and clean code (0 of 141). The reports were rated 39 to 40 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). In both runs the client ran its own compaction agent (an automatic summary), which the audit had marked as an operator error; the owner decided it is normal client behavior, so the runs count. The lab defines the model itself, since the client's catalog no longer lists it.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 466.5 (423.0 to 510.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Three scheduled attempts of a case of five: two completed, 467 and 445 (both Bronze), and one ended on the 90-minute budget with the work folder identical to the package, which scores 100 (Reprovada) as the design asks of every scheduled attempt. The quality of the evaluable deliveries is 456. Across the three, a run took about 57 minutes and US$ 2.25 (the client's own estimate: it ran through the Kimi Code membership, so that is the list price, not a charge), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (130 of 213 on average), then performance (45 of 213), architecture (34 of 283) and clean code (0 of 141). The reports were rated 0 to 43 of 50, the budget-exhausted attempt having none. Seven more attempts were lost to the lab, the membership's quota among the causes. The blind judges agreed on every item. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 3 scheduled attempts completed with the requested model. Systems on which every attempt completed: 0 of 1.
• Expected score per scheduled attempt: 337.3 (100.0 to 467.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 1. Switched: 0. Lost to the lab or the operator, rerun: 7.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 408 and 499, mean 455 with a 95% interval of 427 to 482, all Bronze. All five completed inside the 90-minute budget, about 2.4 million tokens and 7 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (181 of 213 on average), then performance (81 of 213), architecture (57 of 283) and clean code (0 of 141). The reports were rated 37 to 41 of 50. Penalties: runs 2, 3 and 4 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 455.0 (426.8 to 481.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 438 and 465, mean 451 with a 95% interval of 442 to 461, all Bronze. All five completed inside the 90-minute budget, at about US$ 1.92 (the client's own estimate) and 34 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (132 of 213 on average), then performance (122 of 213), architecture (60 of 283) and clean code (0 of 141). The reports were rated 36 to 39 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, whose only upstreams for this model are Google's own.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 451.4 (442.0 to 460.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 434 (Bronze). It completed inside the 90-minute budget, at about US$ 0.37 (the client's own estimate) and 29 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (144 of 213), then performance (77 of 213), architecture (69 of 283) and clean code (0 of 141). The report was rated 44 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, whose only upstream for this model is Tencent's own (fp4 quantization); the client defines the efforts none, low and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 434.0 (434.0 to 434.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 291 and 587, mean 433 with a 95% interval of 332 to 534, three Bronze and two Reprovada. All five completed inside the 90-minute budget, at about US$ 1.52 (the client's own estimate) and 60 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (127 of 213 on average), then performance (99 of 213), architecture (59 of 283) and clean code (15 of 141). The reports were rated 35 to 44 of 50. Penalties: runs 3 and 4 lost 15 points each for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Some runs ran with the client's default permissions and some with its auto-approval flag (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 433.0 (332.4 to 533.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 2.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 384 and 457, mean 421 with a 95% interval of 398 to 442, four Bronze and one Reprovada. All five completed inside the 90-minute budget, about 3.8 million tokens and 15 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (162 of 213 on average), then performance (67 of 213), architecture (53 of 283) and clean code (0 of 141). The reports were rated 39 to 41 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 421.2 (397.6 to 442.4).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 421 (Bronze). It completed inside the 90-minute budget, at about US$ 0.17 (the client's own estimate) and 26 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (137 of 213), then performance (77 of 213), architecture (69 of 283) and clean code (0 of 141). The report was rated 43 of 50. Penalties: run 1 lost 5 points for reporting something that is not a flaw. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, pinned to Xiaomi's own endpoint (fp8 quantization), because the router otherwise spreads the model over other hosts; the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 421.0 (421.0 to 421.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 394 and 425, mean 410 (the interval, 394 to 425, is the range of the two), one Bronze and one Reprovada. Both completed inside the 90-minute budget, at about US$ 1.50 (the client's own estimate) and 33 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (152 of 213 on average), then performance (86 of 213), architecture (32 of 283) and clean code (0 of 141). The reports were rated 38 to 41 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 409.5 (394.0 to 425.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 323 and 485, mean 404 (the interval, 323 to 485, is the range of the two), one Bronze and one Reprovada. Both completed inside the 90-minute budget, at about US$ 3.98 (the client's own estimate) and 42 minutes each, and compatibility was between 70 and 100. Performance was the strongest area (127 of 213 on average), then bug fixing (108 of 213), architecture (42 of 283) and clean code (0 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 404.0 (323.0 to 485.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 356 and 440, mean 400 with a 95% interval of 371 to 427, three Bronze and two Reprovada. All five completed inside the 90-minute budget, about 8.7 million tokens and 14 minutes each (the log keeps no cost), and compatibility was between 70 and 100. Bug fixing was the strongest area (159 of 213 on average), then performance (68 of 213), architecture (46 of 283) and clean code (0 of 141). The reports were rated 38 to 40 of 50. Penalties: run 3 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 400.0 (371.0 to 426.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 357 and 387, mean 372 (the interval, 357 to 387, is the range of the two), all Reprovada. Both completed inside the 90-minute budget, at about US$ 0.31 (the client's own estimate) and 25 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (119 of 213 on average), then performance (100 of 213), architecture (12 of 283) and clean code (0 of 141). The reports were rated 41 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was served by Novita AI, the host the first series used for this model, in its 0731 version.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 372.0 (357.0 to 387.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 321 and 364, mean 343 with a 95% interval of 328 to 357, all Reprovada. All five completed inside the 90-minute budget, at about US$ 0.23 (the client's own estimate) and 25 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (123 of 213 on average), then performance (46 of 213), architecture (32 of 283) and clean code (0 of 141). The reports were rated 41 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 342.6 (328.2 to 357.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 335 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 31 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (94 of 213), then performance (68 of 213), architecture (53 of 283) and clean code (0 of 141). The report was rated 35 of 50. Penalties: run 1 lost 15 points for a new defect introduced by its own change. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was Recursive's); the client defines no effort for it, so it ran at the model's default.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 335.0 (335.0 to 335.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 327 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 13 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (137 of 213), then performance (45 of 213), architecture (4 of 283) and clean code (0 of 141). The report was rated 41 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was NVIDIA's); the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 327.0 (327.0 to 327.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 284 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 10 minutes, and it did not break a behavior the public tests protect (compatibility 100). Performance was the strongest area (95 of 213), then bug fixing (51 of 213), architecture (0 of 283) and clean code (0 of 141). The report was rated 38 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was NVIDIA's); the client defines the efforts none, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 284.0 (284.0 to 284.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 192 and 252, mean 223 with a 95% interval of 202 to 244, all Reprovada. All five completed inside the 90-minute budget, at about US$ 0.05 (the client's own estimate) and 14 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (53 of 213 on average), then architecture (23 of 283), performance (13 of 213) and clean code (0 of 141). The reports were rated 33 to 35 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 223.2 (202.4 to 244.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Four runs of a case of five, between 157 and 298, mean 220 with a 95% interval of 160 to 280, all Reprovada. All four completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 56 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (43 of 213 on average), then performance (34 of 213), architecture (11 of 283) and clean code (2 of 141). The reports were rated 0 to 40 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The fifth run is judged but not settled yet, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through Novita AI's own API, where the model is listed at no price; the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 4 of 4 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 220.0 (159.5 to 280.5).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 179 and 261, mean 209 with a 95% interval of 180 to 241, all Reprovada. All five completed inside the 90-minute budget, at about US$ 0.82 and 16 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (34 of 213 on average), then performance (34 of 213), architecture (10 of 283) and clean code (0 of 141). The reports were rated 29 to 33 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. Every run ran on an Anthropic API key, billed to API credit, where the first Claude configurations ran on a subscription token; the model and the client are the same. Every run used the client's acceptEdits permission mode with only the Bash tool allowed besides the edits, because Haiku 4.5 has no auto mode: the only configuration of the series with another permission setting.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 209.0 (180.4 to 240.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 118 and 164, mean 152 with a 95% interval of 134 to 163, all Reprovada. All five completed inside the 90-minute budget, at about US$ 0.12 (the client's own estimate) and 16 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (27 of 213 on average), then performance (0 of 213), architecture (0 of 283) and clean code (0 of 141). The reports were rated 0 to 34 of 50. Penalties: run 4 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 151.6 (134.2 to 162.6).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 113 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 7 minutes, and it did not break a behavior the public tests protect (compatibility 100). No area scored: bug fixing (0 of 213), performance (0 of 213), architecture (0 of 283) and clean code (0 of 141). The report was rated 33 of 50. Penalties: run 2 lost 20 points for a broken behavior no fix was blamed for. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was NVIDIA's); the client defines the efforts high and max for it, so max is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 113.0 (113.0 to 113.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 108 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 21 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (0 of 213), then performance (0 of 213), architecture (0 of 283) and clean code (0 of 141). The report was rated 23 of 50. Penalties: run 1 lost 15 points for a new defect introduced by its own change. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was NVIDIA's); the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 108.0 (108.0 to 108.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 97 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.00 (the client's own estimate) and 5 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (0 of 213), then performance (0 of 213), architecture (0 of 283) and clean code (0 of 141). The report was rated 17 of 50. Penalties: run 1 lost 15 points for a new defect introduced by its own change; run 1 lost 5 points for reporting something that is not a flaw. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter's free variant, with no provider pinned (the endpoint that answered the lab's probe was Cohere's); the client defines the efforts low, medium and high for it, so high is the most it could ask for.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 97.0 (97.0 to 97.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.