Built in Partnership with Ramp
We're releasing APEX-Accounting, built with Ramp to test whether AI agents can complete real accounting work to professional standards. The benchmark includes 160 tasks across 10 simulated companies. Claude Fable 5 leads at 56.4%, but the headline result is not which model wins. It's how rarely any model succeeds consistently.
APEX-Accounting
Measuring frontier AI's ability to do accounting work
Fable 5 tops the leaderboard
Claude Fable 5 leads with 56.4% of expert grading criteria passed on a typical attempt, ahead of Meta's Muse Spark 1.1 at 52.6% and GPT-5.6 Sol at 51.5%. The best models manage to complete just over half of the work that a professional would.
Consistency is the real gap
Accounting work has to be repeatedly correct, not correct once, so we ran every model on every task eight separate times.
Partial credit was common: at least one model earned some credit on more than 95% of tasks. Full solutions were much harder.
Here's how often models fully solved a task:
- Pass@8, correct at least once in eight tries: 21.5% (Muse Spark 1.1), just ahead of Fable 5 at 20.1%
- Pass^8, correct on all eight tries: 2.6% (GPT-5.6 Sol)
- Tasks never fully solved by any model, on any run: 58%
Performance vs. cost
We tested every model at budgets of $1, $5, $10, and $50 per task. Fable 5 is extremely budget sensitive, scoring 11.8% at $1 and 56.4% at $50. Muse Spark 1.1 is the opposite: already strong on a tight budget and barely improved by more. At the $50 cap Fable 5 spends about $32 per run while Muse Spark 1.1 spends about $5, yet their scores land within 4 percentage points. On average, models use just 64.7% of the budget available to them at $50.
Harness comparison
The canonical leaderboard runs every model in the Loop Harness, Mercor's standard model-and-tools architecture. We separately compared it against a purpose-built Ramp Harness that mirrors parts of a specialized accounting-agent architecture, which improves Mean Criteria@3 by 1.2 percentage points on average. That comparison is a harness ablation, not an evaluation of the complete Ramp Stack product.
Why models fail
Working with accounting experts, we categorized every failed run into four types of mistake:
- Reasoning
- Information gathering
- Instruction following
- Planning and reflection
Roughly seven in ten failures by the top three models are reasoning failures, not an inability to find the right information. A model might correctly identify a discrepancy early in the workflow, then omit or contradict that finding in its final journal entry. Better retrieval alone will not solve this.
How we built it
More than 40 accounting professionals, median 11 years of experience and over half from Big Four firms, authored and solved every task and wrote its rubric, averaging 13.7 criteria each. An open-source AI judge does the grading and agrees with expert graders 97% of the time. Every task runs inside a self-contained world: a fictional company frozen at month-end close, with its own accounts, records, business history, accounting software, spreadsheets, and PDFs.