The AI Productivity Index for SWEs
In partnership with
The AI Productivity Index for Software Engineers (APEX-SWE) measures whether frontier AI systems can execute economically valuable software engineering work. It covers Integration and Observability tasks.
The APEX-SWE leaderboard
We created APEX-SWE to evaluate the real day-to-day work of software engineers, unlike unit-level and single-repository bug-fix benchmarks. It comprises n=200 cases and spans two complementary settings that mirror professional SWE work: (1) Integration tasks that require end-to-end system construction and deployment across heterogeneous services and (2) Observability tasks that require debugging with production-style telemetry.
Each task includes a human-authored rubric that grades agent outputs for functional requirements, robustness, and code style, alongside unit tests.
To support open research, we have open-sourced n=50 cases that are in-distribution of APEX-SWE on Hugging Face with all metadata labels. We have also shared our eval harness for reproducibility.
Evaluates a model's ability to orchestrate end-to-end workflows and synchronize data across heterogeneous services.
View more
Gemini 3.6 FlashGemini 3.6 Flash
64.2%
64.2%
Opus 5Opus 5
64.0%
64.0%
Fable 5Fable 5
63.5%
63.5%
Evaluates a model's ability to diagnose and remediate real-world software engineering production failures.
View more
Opus 5Opus 5
63.5%
63.5%
Fable 5Fable 5
54.2%
54.2%
Grok 4.6Grok 4.6
50.0%
50.0%