The AI Productivity Index for SWEs
The AI Productivity Index for Software Engineers (APEX-SWE) measures whether frontier AI systems can execute economically valuable software engineering work. It covers Integration and Observability tasks.
The APEX-SWE leaderboard
We created APEX-SWE to evaluate the real day-to-day work of software engineers, unlike unit-level and single-repository bug-fix benchmarks. It comprises n=200 cases and spans two complementary settings that mirror professional SWE work: (1) Integration tasks that require end-to-end system construction and deployment across heterogeneous services and (2) Observability tasks that require debugging with production-style telemetry.
Each task includes a human-authored rubric that grades agent outputs for functional requirements, robustness, and code style, alongside unit tests.
To support open research, we have open-sourced n=50 cases that are in-distribution of APEX-SWE on Hugging Face with all metadata labels. We have also shared our eval harness for reproducibility.
Evaluates a model's ability to orchestrate end-to-end workflows and synchronize data across heterogeneous services.
View more
Opus 5Opus 5
65.3%
65.3%
Grok 4.5Grok 4.5
65.0%
65.0%
Gemini 3.6 FlashGemini 3.6 Flash
61.0%
61.0%
Evaluates a model's ability to diagnose and remediate real-world software engineering production failures.
View more
Fable 5Fable 5
50.3%
50.3%
Opus 5Opus 5
44.0%
44.0%
Opus 4.8Opus 4.8
43.3%
43.3%