The AI Productivity Index for Accounting
The AI Productivity Index for Accounting (APEX-Accounting) measures whether frontier AI agents can execute long-horizon, cross-application tasks in professional accounting.
The APEX-Accounting leaderboard
We created APEX-Accounting to evaluate agents on the real day-to-day work of accountants closing the books each month. The tasks require agents to reason over incomplete records, apply company-specific context, use multiple applications, and carry figures accurately through a close.
APEX-Accounting was built in three steps. First, accountants created 10 data-rich worlds, each a business frozen at month-end close. Second, they created challenging period-close tasks using the files from within each world, then solved them to produce the rubrics. Third, we gave agents access to execute the tasks with the software a human accountant would use.
There are 10 worlds in APEX-Accounting, comprising 160 tasks and grading rubrics. An eleventh world is open-source, along with Archipelago, our infra service for executing and evaluating agent trajectories.
Key takeaways
- 1
Frontier agents cannot yet reliably close the books
The best model, Claude Fable 5, meets 56.4% of grading criteria. 58% of the benchmark tasks are never fully solved by any model in any of its eight attempts.
- 2
Capability ≠ consistency
Models occasionally solve a task but almost never solve it on every attempt.
- 3
Spending more does not buy reliability
One model spending the highest dollar amount per run finishes in a statistical tie with one of the leanest models.
- 4
The bottleneck is accounting judgement, not retrieval and tooling
Multi-step reasoning errors account for over half of the top performing model's errors.
| Model | Score | $/run |
|---|---|---|
| Claude Fable 5 | 56.4% | $36.00 |
| Muse Spark 1.1Top quadrant | 52.6% | $4.24 |
| GPT-5.6 Sol | 51.5% | $33.20 |
| Claude Opus 4.8 | 48.0% | $15.82 |
| GLM 5.2Top quadrant | 42.7% | $3.75 |
| Grok 4.5 | 40.8% | $3.77 |
| Kimi K2.7 Code | 37.0% | $1.72 |
| Gemini 3.1 Pro | 32.4% | $8.75 |
| Qwen3.5-397B | 24.4% | $0.68 |
Sample task
Law firmTask prompt
StandardReview the December client cost advance reconciliation. Reconcile the client cost advance ledger to the QBO/GL balance for client cost advances. Using the cost advance ledger methodology and provided support, verify whether hard costs that should be treated as client cost advances were capitalized and whether soft costs were expensed. Identify any discrepancies in cost classification or capitalization treatment that need to be resolved. Round all amounts to two decimal places. Do not round intermediary values. Provide your answer to the console.
Grading criteria
| Criterion | Score |
|---|---|
| Reports the client cost advance ledger ending balance for Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — Yes | Yes |
| Reports the QBO GL ending balance for Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — Yes | Yes |
| Reports the difference between the client cost advance ledger and QBO Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — No | No |
+ 21 more grading criteria
Environment
Project scenario
Sterling, Marsh & Associates LLP is a Philadelphia-based boutique law firm completing its December 2024 month-end close under US GAAP accrual accounting. The world focuses on law-firm-specific close workflows around Clio-based billable hours and WIP, hourly, flat-fee, and contingency revenue recognition, IOLTA trust accounting, retainer deposits and earned-fee transfers, client cost advances, realization adjustments, partner guaranteed payments and profit allocations, contract attorney accruals, and matter-level profitability.
- QuickBooks
- Sheets
- PDFs
- Docs
- Code
90 files, 3.6 MB
ap_aging_2024_12_31.xlsx10.9 KBattorney_staff_roster.xlsx6.7 KBclient_matter_list.xlsx9.0 KBclient_retainer_schedule.xlsx17.8 KBclio_time_entries_2024.csv1.6 MBengagement_letter_summary_2024.xlsx8.7 KBgusto_employee_earnings_summary_2024.xlsx12.8 KBgusto_payroll_register_2024.csv100.1 KBiolta_interest_log.xlsx6.9 KBopening_ar_collections_2024.xlsx10.1 KBpartner_compensation_schedule.xlsx23.1 KBqbo_ar_aging_summary_2024_12_31.xlsx7.1 KBqbo_balance_sheet_2024_12_31.xlsx5.9 KBqbo_closing_trial_balance_2024_12_31.xlsx8.1 KBqbo_general_ledger_detail_2024.xlsx266.8 KBqbo_opening_trial_balance_2024_01_01.xlsx5.4 KBqbo_profit_and_loss_2024.xlsx6.4 KBqbo_profit_and_loss_by_class_2024.xlsx7.4 KBqbo_reports_index.xlsx10.8 KBrate_card.xlsx6.0 KB
sterling_malpractice_policy_2024.pdf7.2 KBworkpaper_annual_budget_model_2024.xlsx13.2 KBworkpaper_ar_aging_2024_12_31.xlsx12.7 KBworkpaper_bad_debt_reserve_2024.xlsx10.1 KBworkpaper_client_cost_advance_ledger_2024.xlsx15.2 KBworkpaper_contingency_case_status_2024.xlsx7.9 KBworkpaper_iolta_bank_reconciliation_2024_11_30.xlsx11.5 KBworkpaper_iolta_bank_reconciliation_2024_12_31.xlsx366.5 KBworkpaper_operating_bank_reconciliation_2024_12_31.xlsx8.6 KBworkpaper_partner_capital_rollforward_2024.xlsx24.1 KBworkpaper_payroll_accrual_2024_12_31.xlsx8.1 KBworkpaper_prepaid_insurance_2024_12_31.xlsx7.8 KBworkpaper_pto_accrual_2024_12_31.xlsx8.7 KBworkpaper_realization_rate_analysis_2024.xlsx12.7 KBworkpaper_trust_reconciliation_2024_12_31.xlsx8.8 KBworkpaper_unbilled_wip_2024_12_31.xlsx12.2 KBworkpaper_whitfield_settlement_summary.xlsx7.8 KBworkpaper_year_end_AJEs_2024.xlsx9.9 KBLoss analysis
Discover where models failed on APEX-Accounting tasks.
Hover, focus, or tap a slice to inspect it
Methodology
Each model runs a month-end close inside one of 10 real businesses, with its accounts, records, and accounting software. More than 40 professionals, median 11 years' experience and half from Big Four firms, built the tasks and rubrics. Agents get a short request: no file list, no method. They must find their own evidence, apply U.S. GAAP accrual accounting, and produce an answer graded criterion by criterion. Our LM judge matches expert graders 97% of the time.
Harness
Leaderboard results use the Loop Harness, the canonical while-loop-with-tools agent architecture.
Grading
LM judge scores each rubric criterion independently as Met or Not Met, seeing the task prompt, the criterion, and the model’s final answer.
Runs
Every model is run 8 times to get Pass@1, Pass@8, and Pass^8. The canonical leaderboard metric is Mean Criteria % Passed (k=3).
Model config
Nine frontier models at highest available reasoning, called through LiteLLM with exponential backoff, and up to three attempts per request.
Cost limits
Leaderboard runs use a 5-million-token cap and 500 step limit.
Error bars / CI
All intervals are 95% task-level bootstraps over 10,000 resamples.
Dataset
160 tasks across 10 held-out worlds, with 2,186 grading criteria, and a mean of 7.5 required input files per task.
Frequently Asked Questions
The Mercor AI Productivity Index (APEX) is a family of benchmarks that measure how effectively AI models and agents perform economically valuable tasks. APEX-Accounting evaluates how well AI models complete realistic month-end close and bookkeeping tasks across simulated companies. It was developed in collaboration with Ramp.
Rankings can change whenever a frontier model is released. New models are evaluated on APEX-Agents, APEX-SWE, APEX-Accounting and APEX-1 when they ship.
Practicing professionals from leading firms that the tasks simulate, including experts from Deloitte, PwC, EY, and KPMG.
APEX-Accounting evaluates the quality of completed work. Model outputs are graded using expert-authored rubrics with an LM judge.
Mean Score measures the average percentage of criteria that are passed for all tasks in the benchmark. It is the primary metric used to rank the APEX-Accounting leaderboard. Pass@8 measures the percentage of tasks that an agent successfully completes on at least one of eight attempts (i.e., scores 100% on the rubric). Pass^8 measures the percentage of tasks that an agent successfully completes on all eight attempts. Together, they capture models’ overall capability and end-to-end task reliability.
Yes, on the open subset. The eval harness is published on GitHub and sample tasks are on HuggingFace, so you can run the same scoring pipeline against your own model. The full task set stays private so that models can't be trained on it.
Yes. Mercor licenses off-the-shelf datasets built by the same expert network. 50,000+ tasks across 30+ domains, with samples available the same day. Learn more about off-the-shelf data.