OpenWorldEngineer (OWE-Bench)

The OpenWorldEngineer leaderboard

We built OpenWorldEngineer to evaluate whether agents can make substantial contributions to full engineering projects. Every task requires significant work beyond the best public methods: agents must adapt existing work, solve the remaining challenges, and turn their decisions into measurable improvements.

Scores reflect how far an agent advances the project toward a defined objective. Each attempt is rescaled so the untouched baseline scores 0% and the task's ceiling scores 100%, so both incremental gains and major breakthroughs count. Final evaluation runs on held-out cases the agent never sees.

There are 26 tasks across 4 domains: science, software, ML, and engineering. Every task is built by a domain expert with a master's or PhD in their field, 10+ years of experience, or both. The benchmark is open source.

Mean@8 averages each task's attempts, then the tasks, with a 95% interval across tasks. Best@8 averages each task's best attempt. Elo rates the models head to head on every task, with the task baseline anchored at 1000.