On the v1.0 leaderboard, the best agent, GPT-5.6 Sol with Codex CLI, resolved 333 of the 417 tasks. Agents released since then solve more, and when we looked at the tasks they still failed, the test was often the one at fault. A correct patch could fail because it was written differently from the reference fix, and a patch that left the bug in place could pass because the test never looked at the behaviour in question.
After v1.0 we went through every task that at least one of six frontier agents had failed, 148 in total, and decided for each whether the patch or the test was wrong. For the 269 tasks that all six passed, we checked that the patches really fixed the bug, that the test could fail at all, and that it observed the behaviour the task is about. In the end 208 tasks changed. Every revised test passes the reference fix and fails the original buggy code, and each task keeps its repository, base commit, Docker image and fail-to-pass tests.
| Repository | Tasks | Revised | Test script | Statement | Reference fix |
|---|---|---|---|---|---|
| OpenTitan | 245 | 157 | 145 | 35 | – |
| Ibex | 35 | 12 | 9 | 4 | – |
| CVA6 | 35 | 4 | 4 | – | – |
| Caliptra | 16 | 2 | 2 | – | – |
| XiangShan | 54 | 17 | 12 | 9 | 1 |
| Rocket Chip | 32 | 16 | 16 | 2 | – |
| Total | 417 | 208 | 188 | 50 | 1 |
Tests that own testbench files
Many hidden tests bring their own testbench code: a DV top such as OpenTitan's hw/ip/csrng/dv/tb.sv, a Verilator harness, or a small unit test. Agents often edit the same files while they debug. In v1.0 the hidden test then ran on the agent's copy, so a leftover testbench edit could decide whether the task passed.
- Each of these 91 tests lists the files it owns and restores them from the base commit before it runs.
- Changes outside those files, including every design change, are tested as the agent made them.
Tests tied to the reference fix
Some tests depended on how the reference fix was written. They referred to internal signals by name, matched exact source text, or replaced design macros with simpler stubs. A correct fix written another way then failed to compile or elaborate.
In OpenTitan #12505, for example, the test replaced the ASSERT macro with a stub that took two arguments. GPT-6.1 Sol and GPT-6 Astra both fixed the bug and added an assertion that passes a clock argument, which the real macro accepts, and VCS stopped with a macro argument error. In OpenTitan #10732 the test switched off an assertion by its hierarchical name. Claude Opus 5.5's fix no longer declared that assertion, so the name did not resolve and VCS reported a cross-module reference error.
- These 60 tests check the behaviour the task asks for and no longer depend on the reference fix's signal names, source text or macro use.
- The macro stub in #12505 takes the same optional clock and reset arguments as the real one, and #10732 switches the assertion off only when the patched design still declares it.
Stronger checks
Other tests were too weak. They passed patches that did not fix the bug, or never exercised the behaviour the task describes. For the tasks all six agents had passed, we read each patch against the task, and where one looked wrong we added a check for the described behaviour and reran every patch. In 29 tasks at least one patch failed the new check.
- These 33 tests now observe the described behaviour directly, so a patch that leaves the bug in place fails.
Build environment
A few tasks failed for reasons unrelated to the patch. Two OpenTitan builds broke with newer releases of setuptools_scm, and five Rocket Chip tests ran out of time while sbt was still downloading dependencies.
- OpenTitan #14681 and #16176 pin
setuptools_scm6.4.2. - Rocket Chip #177, #387, #1493, #1761 and #2167 raise the test's time limit from 1200 s to 3600 s.
- The evaluator gives seven tasks a 4500 s container limit instead of the default 1800 s: Rocket Chip #177, #387, #1493, #1761, #2036 and #2167, and OpenTitan #14681.
Problem statements and one reference fix
For 50 tasks, the v1.0 statement left open an interface or behaviour that the hidden test checks, or described it differently. These statements now specify it.
The reference fix of XiangShan #2781 used to change the fudian floating-point library. It now carries the rounding mode through XiangShan's own bundles, and the statement notes that changes under fudian/ are not evaluated.
The new leaderboard
We evaluated seven agents on v1.1, six of them at max reasoning effort. Claude Opus 5.5 with Claude Code is #1 at 99.0% (413/417), followed by GPT-6.1 Sol and GPT-6 Astra with Codex CLI at 97.1%. Kimi K3.1 Pro and DeepSeek V4.1 Flash 1017 lead the open-source agents at 92.6%.
Precision and cost are computed as in v1.0. The leaderboard now also plots each agent's cost against its resolved rate: GPT-6.1 Sol resolves 405 tasks for $197, and DeepSeek V4.1 Flash 1017 resolves 386 for $113.
View the v1.1 leaderboard · Compare with v1.0 →
Run v1.1 yourself
The Hugging Face dataset serves v1.1 from its main branch, with the same files and fields as before. Regenerate the Harbor task directories from the v1.1 files, since they carry the problem statements, and use the evaluator from the current repository, which includes the 4500 s limit above. To reproduce v1.0 results, including those in the paper, download the v1.0 tag:
hf download henryen/hwe-bench --repo-type dataset --revision v1.0 --local-dir datasets-v1.0/