The AI Productivity Index for Accounting
The AI Productivity Index for Accounting (APEX-Accounting) measures whether frontier AI agents can execute long-horizon, cross-application tasks in professional accounting.
The APEX-Accounting leaderboard
We created APEX-Accounting to evaluate agents on the real day-to-day work of accountants closing the books each month. The tasks require agents to reason over incomplete records, apply company-specific context, use multiple applications, and carry figures accurately through a close.
APEX-Accounting was built in three steps. First, accountants created 10 data-rich worlds, each a business frozen at month-end close. Second, they created challenging period-close tasks using the files from within each world, then solved them to produce the rubrics. Third, we gave agents access to execute the tasks with the software a human accountant would use.
There are 10 worlds in APEX-Accounting, comprising 100 tasks and grading rubrics. An eleventh world is open-source, along with Archipelago, our infra service for executing and evaluating agent trajectories.
Key takeaways
- 1
Frontier agents cannot yet reliably close the books
The best models meet most grading criteria, but fully complete only a small share of tasks on the first try.
- 2
Capability ≠ consistency
More attempts mean more tasks solved at least once but fewer solved every time. The gap widens with each try.
- 3
Higher cost does not mean higher scores
The top scorer is not the most expensive model. The costliest model scores no better than models that cost far less per run.
- 4
No single model wins every task
Even the top model fails tasks that lower-ranked models solve. Each model has different blind spots.
Results
Explore model performance by cost, token usage, and consistency. Hover for additional details or filter by model.
Mean criteria score (%)
Average cost per run ($)
Sample task
Law firmTask prompt
StandardReview the December client cost advance reconciliation. Reconcile the client cost advance ledger to the QBO/GL balance for client cost advances. Using the cost advance ledger methodology and provided support, verify whether hard costs that should be treated as client cost advances were capitalized and whether soft costs were expensed. Identify any discrepancies in cost classification or capitalization treatment that need to be resolved. Round all amounts to two decimal places. Do not round intermediary values. Provide your answer to the console.
Grading criteria
| Criterion | Score |
|---|---|
| Reports the client cost advance ledger ending balance for Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — Yes | Yes |
| Reports the QBO GL ending balance for Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — Yes | Yes |
| Reports the difference between the client cost advance ledger and QBO Client Cost Advances Hard Costs (or Account 1300) as $0.00 (acceptable value is $0.00). — No | No |
+ 21 more grading criteria
Environment
Project scenario
Sterling, Marsh & Associates LLP is a Philadelphia-based boutique law firm completing its December 2024 month-end close under US GAAP accrual accounting. The world focuses on law-firm-specific close workflows around Clio-based billable hours and WIP, hourly, flat-fee, and contingency revenue recognition, IOLTA trust accounting, retainer deposits and earned-fee transfers, client cost advances, realization adjustments, partner guaranteed payments and profit allocations, contract attorney accruals, and matter-level profitability.
- QuickBooks
- Sheets
- PDFs
- Docs
- Code
90 files, 3.6 MB
ap_aging_2024_12_31.xlsx10.9 KBattorney_staff_roster.xlsx6.7 KBclient_matter_list.xlsx9.0 KBclient_retainer_schedule.xlsx17.8 KBclio_time_entries_2024.csv1.6 MBengagement_letter_summary_2024.xlsx8.7 KBgusto_employee_earnings_summary_2024.xlsx12.8 KBgusto_payroll_register_2024.csv100.1 KBiolta_interest_log.xlsx6.9 KBopening_ar_collections_2024.xlsx10.1 KBpartner_compensation_schedule.xlsx23.1 KBqbo_ar_aging_summary_2024_12_31.xlsx7.1 KBqbo_balance_sheet_2024_12_31.xlsx5.9 KBqbo_closing_trial_balance_2024_12_31.xlsx8.1 KBqbo_general_ledger_detail_2024.xlsx266.8 KBqbo_opening_trial_balance_2024_01_01.xlsx5.4 KBqbo_profit_and_loss_2024.xlsx6.4 KBqbo_profit_and_loss_by_class_2024.xlsx7.4 KBqbo_reports_index.xlsx10.8 KBrate_card.xlsx6.0 KB
sterling_malpractice_policy_2024.pdf7.2 KBworkpaper_annual_budget_model_2024.xlsx13.2 KBworkpaper_ar_aging_2024_12_31.xlsx12.7 KBworkpaper_bad_debt_reserve_2024.xlsx10.1 KBworkpaper_client_cost_advance_ledger_2024.xlsx15.2 KBworkpaper_contingency_case_status_2024.xlsx7.9 KBworkpaper_iolta_bank_reconciliation_2024_11_30.xlsx11.5 KBworkpaper_iolta_bank_reconciliation_2024_12_31.xlsx366.5 KBworkpaper_operating_bank_reconciliation_2024_12_31.xlsx8.6 KBworkpaper_partner_capital_rollforward_2024.xlsx24.1 KBworkpaper_payroll_accrual_2024_12_31.xlsx8.1 KBworkpaper_prepaid_insurance_2024_12_31.xlsx7.8 KBworkpaper_pto_accrual_2024_12_31.xlsx8.7 KBworkpaper_realization_rate_analysis_2024.xlsx12.7 KBworkpaper_trust_reconciliation_2024_12_31.xlsx8.8 KBworkpaper_unbilled_wip_2024_12_31.xlsx12.2 KBworkpaper_whitfield_settlement_summary.xlsx7.8 KBworkpaper_year_end_AJEs_2024.xlsx9.9 KBMethodology
Each model runs a month-end close inside one of 10 real businesses, with its accounts, records, and accounting software. More than 40 professionals, median 11 years' experience and half from Big Four firms, built the tasks and rubrics. Agents get a short request: no file list, no method. They must find their own evidence, apply U.S. GAAP accrual accounting, and produce an answer graded criterion by criterion. Our LM judge matches expert graders 97% of the time.
Harness
Leaderboard results use the Loop Harness, the canonical while-loop-with-tools agent architecture.
Grading
LM judge scores each rubric criterion independently as Met or Not Met, seeing the task prompt, the criterion, and the model’s final answer. The LM judge also penalizes scattergunning (hedging across several answers).
Runs
Every model is run 8 times to get Pass@1, Pass@8, and Pass^8. The canonical leaderboard metric is Mean Criteria % Passed (k=4).
Model config
Unless otherwise stated, models were run at highest available reasoning setting. Issues with responses on a given step are rerun until success. In cases of persistent system error or model failures, we rerun the trajectory up to three times.
Cost limits
Leaderboard runs use a 5-million-token cap and 500 step limit.
Error bars / CI
All intervals are 95% task-level bootstraps over 10,000 resamples.
Dataset
100 tasks across 10 held-out worlds, with 1,475 grading criteria, and a mean of 7.5 required input files per task.
Frequently Asked Questions
The Mercor AI Productivity Index (APEX) is a family of benchmarks that measure how effectively AI models and agents perform economically valuable tasks. APEX-Accounting evaluates how well AI models complete realistic month-end close and bookkeeping tasks across simulated companies. It was developed in collaboration with Ramp.
Rankings can change whenever a frontier model is released. New models are evaluated on APEX-Agents, APEX-SWE, APEX-Accounting and APEX-1 when they ship.
Practicing professionals from leading firms that the tasks simulate, including experts from Deloitte, PwC, EY, and KPMG.
APEX-Accounting evaluates the quality of completed work. Model outputs are graded using expert-authored rubrics with an LM judge.
Mean Score measures the average percentage of criteria that are passed for all tasks in the benchmark. It is the primary metric used to rank the APEX-Accounting leaderboard. Pass@8 measures the percentage of tasks that an agent successfully completes on at least one of eight attempts (i.e., scores 100% on the rubric). Pass^8 measures the percentage of tasks that an agent successfully completes on all eight attempts. Together, they capture models’ overall capability and end-to-end task reliability.
Yes, on the open subset. The eval harness is published on GitHub and sample tasks are on HuggingFace, so you can run the same scoring pipeline against your own model. The full task set stays private so that models can't be trained on it.
Yes. Mercor licenses off-the-shelf datasets built by the same expert network. 50,000+ tasks across 30+ domains, with samples available the same day. Learn more about off-the-shelf data.