Benchmarking methodology

Mercor

The AI Productivity Index assesses whether frontier models can perform economically valuable tasks across four jobs: investment banking associate, management consultant, big law associate, and primary care physician.

Source

Mercor

Tasks

300

Runs

k = 4

Metric

Mean Score

Modality

Text + reference files

Task format

Single-turn

Question format

Open-ended professional deliverables

Canonical harness

Single-shot agent

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

LLM judge over expert rubrics

Released

September 2025

Mercor

Assesses whether frontier models can produce the core deliverables of an accountant — reconciliations, financial statement preparation, audit workpapers, and tax treatment — graded against rubrics written by practising accountants.

Source

Mercor

Tasks

160

Runs

k = 4

Metric

Mean Score, Pass@1

Modality

Text + spreadsheets and filings

Task format

Single-turn

Question format

Open-ended accounting deliverables

Canonical harness

Single-shot agent

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

LLM judge over expert rubrics

Released

March 2026

OpenAI

Evaluates models on real-world, economically valuable tasks spanning most BLS work activities for 44 occupations across nine major GDP sectors.

Source

OpenAI

Tasks

213

Runs

k = 4

Metric

Mean Score

Modality

Text + reference files

Task format

Multi-step agentic

Question format

Open-ended real-world deliverables

Canonical harness

Stirrup

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3 Flash Preview

Released

October 2025

Harvey

Evaluates AI agents' abilities to perform real legal work across 24 practice areas, requiring agents to produce legal documents such as memos, redlines, contracts, and filings.

Source

Harvey

Tasks

1,749

Runs

k = 3

Metric

Pass@1

Modality

Text + reference files

Task format

Multi-step agentic

Question format

Open-ended legal deliverables

Canonical harness

Harvey LAB Agent

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

Gemini 3 Flash Preview

Released

May 2026

Mercor

Measures whether frontier AI agents can execute long-horizon, cross-application tasks across three jobs in professional services.

Source

Mercor

Tasks

480

Runs

k = 3

Metric

Pass@1, Mean Score

Modality

Text + cross-application environment

Task format

Multi-step agentic

Question format

Long-horizon cross-application tasks

Canonical harness

Loop (truncated tools)

Grading

Expert-authored rubrics over final deliverables and agent trajectories

Judge

LLM judge over expert rubrics

Released

January 2026

OpenAI

Questions requiring agents to persistently navigate the internet to locate hard-to-find, entangled information. Tests information-discovery competence through persistence and creative problem-solving.

Source

OpenAI

Tasks

130

Runs

k = 4

Metric

Pass@1

Modality

Text + live web browsing

Task format

Multi-step agentic

Question format

Short free-text answer

Canonical harness

Web research agent (100 max steps, 3 hour time limit)

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

April 2025

University of Science and Technology of China

Assesses whether web-research agents can produce long-form research reports from open-ended expert briefs. Each report is evaluated for information recall, analysis, and presentation.

Source

University of Science and Technology of China

Tasks

130

Runs

k = 1

Metric

Mean Score

Modality

Text + live web browsing

Task format

Multi-step agentic

Question format

Long-form research report generation

Canonical harness

Web research agent (100 max steps)

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

GPT 5.5

Released

January 2026

Mercor + Cognition

Measures whether frontier AI agents can execute high-value engineering work, split across observability and integration tasks. Created by Mercor in collaboration with Cognition.

Source

Mercor + Cognition

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Observability and integration engineering tasks

Canonical harness

Agentic coding harness

Grading

Expert-authored rubrics plus repository test suites

Judge

LLM judge + programmatic tests

Released

January 2026

University of Illinois Urbana-Champaign

Determines if models can write code to solve realistic scientific research problems across natural science domains like physics, chemistry, biology, and mathematics.

Source

University of Illinois Urbana-Champaign

Tasks

65

Runs

k = 3

Metric

Mean Score

Modality

Text (scientific code)

Task format

Chained single-shot

Question format

Scientific subproblem code implementation

Canonical harness

SciCode chain agent

Grading

Programmatic unit tests for each subproblem (all must pass)

Judge

None

Released

July 2024

Stanford; Harbor; Laude Institute

Evaluates AI agents on hard, realistic long-horizon command-line tasks: curated tasks with unique environments, human-written solutions, and verification tests.

Source

Stanford; Harbor; Laude Institute

Tasks

89

Runs

k = 3

Metric

Pass@1

Modality

Text (terminal, CLI)

Task format

Multi-step agentic

Question format

Command-line task completion

Canonical harness

Terminus 2 (1000 max steps, 3 hour time limit)

Grading

Programmatic unit tests (all must pass)

Judge

None

Released

January 2026

Scale AI

Benchmarks coding agents on codebase comprehension via under-specified, agentic Q&A tasks over real repositories, combining programmatic validation with rubric-based software-quality scoring.

Source

Scale AI

Tasks

124

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Under-specified agentic codebase Q&A

Canonical harness

mini-swe-agent (250 max steps, 1 hour time limit)

Grading

LLM-as-a-judge (LMAAJ) over software quality rubrics

Judge

Claude Opus 4.5

Released

May 2026

Princeton; OpenAI (Verified subset)

Tests whether models can resolve real-world GitHub issues by editing codebases, requiring multi-file, long-context changes verified by the repository’s test suite.

Source

Princeton; OpenAI (Verified subset)

Tasks

500

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Code patch resolving a GitHub issue

Canonical harness

mini-swe-agent (1000 max steps, 3 hour time limit)

Grading

Programmatic fail-to-pass and pass-to-pass unit tests (all must pass)

Judge

None

Released

October 2023

Datacurve

Gauges whether models can complete realistic, long-horizon software-engineering work inside real code repositories.

Source

Datacurve

Tasks

113

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Code patch resolving a real-world repository issue

Canonical harness

mini-swe-agent (500 max steps, 2 hour time limit)

Grading

Programmatic unit tests (all must pass)

Judge

None

Released

June 2026

Meta

Tests whether software-engineering agents can rebuild complete programs from scratch given only a program and its documentation, matching a reference executable via end-to-end testing — from CLI tools to FFmpeg, SQLite, and PHP.

Source

Meta

Tasks

100

Runs

k = 3

Metric

Pass@1

Modality

Text (executable + documentation)

Task format

Multi-step agentic

Question format

Executable reverse engineering

Canonical harness

mini-swe-agent (1000 max steps, 6 hour time limit)

Grading

Programmatic unit tests (all must pass)

Judge

None

Released

May 2026

University of Waterloo (TIGER-Lab)

An enhanced MMLU adding harder, reasoning-focused questions and expanding choices from 4 to 10 options. MMLU evaluates general knowledge and reasoning across 57 academic and professional subjects.

Source

University of Waterloo (TIGER-Lab)

Tasks

12,032

Runs

k = 1

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Multiple choice (10 options)

Canonical harness

Single-shot agent

Grading

Programmatic multiple-choice answer extraction

Judge

None

Released

June 2024

New York University

Graduate-level, “Google-proof” multiple-choice science questions written by domain experts; the Diamond split is the hardest, expert-validated subset.

Source

New York University

Tasks

198

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Multiple choice (4 options)

Canonical harness

Single-shot agent

Grading

Programmatic multiple-choice answer extraction

Judge

None

Released

November 2023

Mathematical Association of America

Competition-mathematics problems from the 2025 American Invitational Mathematics Examination, each with an integer answer; measures advanced multi-step mathematical reasoning.

Source

Mathematical Association of America

Tasks

30

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Integer answer (0–999)

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

2025

Scale AI + Center for AI Safety

Frontier, multi-disciplinary academic knowledge and reasoning on closed-ended questions requiring genuine subject mastery rather than retrieval.

Source

Scale AI + Center for AI Safety

Tasks

2,158

Runs

k = 3

Metric

Pass@1

Modality

Text + multimodal

Task format

Single-shot

Question format

Exact-match short answer (~76%); multiple choice, 5+ options (~24%)

Canonical harness

Single-shot multimodal

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

January 2025

Peking University

Challenges models to solve expert-curated chemistry problems, with tasks ranging from synthesis and structural analysis to experimental interpretation and physical-chemistry calculations.

Source

Peking University

Tasks

500

Runs

k = 3

Metric

Pass@1

Modality

Text + multimodal

Task format

Single-shot

Question format

Multiple choice (up to 26 options)

Canonical harness

Single-shot multimodal

Grading

Programmatic multiple-choice answer extraction

Judge

None

Released

December 2025

Carnegie Mellon University

A more robust multimodal benchmark that filters text-only-solvable questions, expands answer options, and adds vision-only inputs (text embedded in images) to test true joint visual and textual reasoning.

Source

Carnegie Mellon University

Tasks

1,730

Runs

k = 3

Metric

Pass@1

Modality

Multimodal (image required)

Task format

Single-shot

Question format

Multiple choice (up to 10 options)

Canonical harness

Single-shot multimodal

Grading

Programmatic multiple-choice answer extraction

Judge

None

Released

September 2024

Princeton University

Evaluates multimodal models on realistic chart understanding using hand-curated arXiv charts, with descriptive questions on basic elements and reasoning questions that synthesize across complex elements.

Source

Princeton University

Tasks

5,000

Runs

k = 3

Metric

Pass@1

Modality

Multimodal (charts, figures)

Task format

Single-shot

Question format

Short answer (descriptive + reasoning questions on charts)

Canonical harness

Single-shot multimodal

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

June 2024

Tsinghua University; Shanghai Artificial Intelligence Laboratory

Assesses medical reasoning on multimodal multiple-choice questions that require models to interpret clinical images and patient descriptions across 11 body systems.

Source

Tsinghua University; Shanghai Artificial Intelligence Laboratory

Tasks

200

Runs

k = 3

Metric

Pass@1

Modality

Multimodal (image required)

Task format

Single-shot

Question format

Multiple choice (5 options)

Canonical harness

Single-shot multimodal

Grading

Programmatic multiple-choice answer extraction

Judge

None

Released

January 2025

Artificial Analysis

Long Context Reasoning: 100 hard questions over 234 documents in 30 sets that require synthesizing information across multiple documents and complex reasoning rather than simple retrieval.

Source

Artificial Analysis

Tasks

100

Runs

k = 4

Metric

Pass@1

Modality

Text (~100k tokens per question)

Task format

Single-shot

Question format

Short free-text answer over multi-document sets

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

August 2025

Meta

Evaluates complex, multi-turn, and system-level instruction following via expert-curated rubrics over 1,600+ prompts.

Source

Meta

Tasks

1,645

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Open-ended prompts graded against expert-written rubrics

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ) over expert-written rubric criteria

Judge

Gemini 3.5 Flash

Released

November 2025