Introducing OWE-Bench for Long-Horizon Engineering Projects

Today we're releasing OpenWorldEngineer (OWE-Bench), a frontier coding benchmark of 26 long-horizon engineering and research tasks across software engineering, physical engineering, machine learning, and scientific computing.

We built OWE-Bench to test whether AI agents can make real progress on long-horizon engineering projects, ranging from GPU kernels to aircraft design. Every task is graded on unseen cases and scored on a continuous scale. This means that an agent that speeds up a program halfway to the ceiling earns 50% rather than a pass-or-fail zero.

Claude Fable 5.2 tops the current leaderboard with a score of 54.6%, using an average of 211 million tokens per task attempt. GPT-6 Astra follows with a score of 44.2% using 89 million tokens, and Claude Opus 5.5 at 41.6% using 253 million tokens. No model is close to the benchmark ceiling. Fable 5.2 leads all four domains, with Astra within a few points in physical engineering and scientific computing. The benchmark also successfully differentiates between small and large models, with Inkling and Beam only scoring 1.1% and 3.3% respectively.

OWE-Bench can be downloaded here and will grow with the community. The next version will be open to outside contributors.

Engineers work in the open world

Engineering requires more than recall. Engineers read papers, check specifications, and adapt open-source implementations. Much of the knowledge an engineer needs to complete their work cannot be found in the project brief alone.

Tasks for OWE-Bench come in two shapes. In scaffolded tasks, the agent starts from a working codebase and pushes a measurable target such as kernel speed, latency, or error. In end-to-end tasks, it builds the whole artifact from a bare environment: a trained model, a data pipeline, a mechanical design. Depending on the task, agents get 4 to 18 hours per attempt (median 8), for 257 agent-hours across one pass over all tasks, on hardware matched to the task, such as a CPU sandbox, NVIDIA H100 or B300. Seven tasks use a GPU.

The 26 tasks span four domains:

• Software engineering (8 tasks): databases, data pipelines, interpreters, and layout, such as exactly-once streaming payments ETL into Iceberg tables with alerts under fault injection
• Physical engineering (5 tasks): CAD reconstruction, structural design, and aircraft, such as a parametric Design/Build/Fly electric aircraft scored over 34 hidden flights spanning three missions
• Machine learning (7 tasks): GPU kernels, training, and inference, such as post-training Qwen3-8B on chess tactics through a metered training service, scored on 1,000 private puzzles
• Scientific computing (6 tasks): chemistry and earth-science modeling, such as relocating an earthquake sequence from supplied seismic observations

Tasks built by practitioners

Each task starts with a domain expert suggesting a project derived from their real-world work. Domain experts recruited through Mercor write a proposal that describes the relevant public work, the deliverable, and how it will be verified. After maintainers review the proposal, the expert builds the task environment and its automated verifier.

Each task goes through layered review, and can only be accepted if approved by three human reviewers. Automated static checks and an LM-assisted rubric grade for structure, reproducibility, instructions, score design and budgets. We red teamed the quality of the task grading adversarially to find problems that need fixing.

Graded on what the agent ships

Evaluation begins with agents submitting an artifact, like a model checkpoint, a data pipeline, a kernel package, a CAD program, or an aircraft design. They are then scored offline by an independent verifier operating without network access, testing the deliverable on CPUs and GPUs using unexposed, held-out inputs. Performance is measured strictly through execution, simulation, or geometric analysis rather than model judgments.

Tasks report scores in their own units (e.g., speed in milliseconds, accuracy). We put them on one scale so the untouched baseline scores 0% and the task's ceiling scores 100%. A model's task score is the mean of its eight attempts, and its leaderboard score is the mean over all 26 tasks. We audited every trial for reward hacking. Exploit attempts appear in 39 of 2,912 trials (1.3%). In our grading we include a check for reward hacking, and exploits that increase the score are set to zero.

What the results show

Fable 5.2 leads in every domain. It scores 62.2% in software engineering against 49.7% for Claude Opus 5.5 (+12.2pp), and 54.6% in machine learning against 47.8% for Claude Sonnet 5.5 (+6.8pp). The gap with the next best model is far smaller in physical engineering (59.6% vs. 57.3%) and scientific computing (40.2% vs. 39.4%), where GPT-6 Astra is within the margin of error.

We read the trajectories for the 8 runs on the 26 tasks from Claude Opus 5.5, Gemini 4 Argon and GPT-6.1 Sol (i.e., 624 trajectories total) to understand where they fail. Most failed runs shared one problem: the model checked its code on simpler tests than the actual ones in the tasks. Agents validated on small inputs, familiar data or their own simulators, then shipped work that broke at the scale, temperature or input shapes the tasks actually required. GPT-6.1 Sol and Gemini 4 Argon also lose points by overfitting to the practice data, for example by adding shortcuts that only work on public samples.

Cost varies sharply. GPT-6 Astra scores 44.2% while spending $234.63 on tokens per attempt; Claude Fable 5.2 averages 54.6% while spending $105.08; GPT-6.1 Sol averages 41.2% while spending $52.12, and Claude Sonnet 5.5 averages 39.0%, spending $38.65.

Get involved

All 26 tasks in OWE-Bench are public, with their instructions, starting code, development cases and scoring anchors. For 16 tasks we also release the same grader used in the leaderboard, but with a public test instead of the hidden version. The other ten graders stay private because they cannot be separated from the hidden data. Hidden test sets, their generators and seeds stay private for every task.

We’re planning to open OWE-2.0 to the community and are accepting submissions for additional tasks here. We'll share more details soon and all contributors will be fully acknowledged in future releases. OWE-Bench was built by Mercor’s Code Data and Research teams. We’re hiring, so come join us.

Acknowledgments: Thanks to Chirag Mahapatra for backing the project from the start, to Victor Barres for research review and guidance, to Brad Johnson for helping with marketing, to Harish Shankar for helping with analysis, to the Mercor experts who proposed, built, reviewed, and audited the tasks, and to the broader Code Data and Research teams.