Benchmarking methodology

Mercor

The AI Productivity Index assesses whether frontier models can perform economically valuable tasks across four jobs: investment banking associate, management consultant, big law associate, and primary care physician.

Source

Mercor

Tasks

300

Runs

k = 4

Metric

Mean Score

Modality

Text + reference files

Task format

Single-turn

Question format

Open-ended professional deliverables

Canonical harness

Single-shot agent

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

LLM judge over expert rubrics

Released

September 2025

Mercor

Assesses whether frontier models can produce the core deliverables of an accountant — reconciliations, financial statement preparation, audit workpapers, and tax treatment — graded against rubrics written by practising accountants.

Source

Mercor

Tasks

160

Runs

k = 4

Metric

Mean Score, Pass@1, Pass@8, Pass^8

Modality

Text + spreadsheets and filings

Task format

Single-turn

Question format

Open-ended accounting deliverables

Canonical harness

Single-shot agent

Grading

Expert-authored rubrics, scored criterion by criterion

Judge

LLM judge over expert rubrics

Released

March 2026

Mercor

Measures whether frontier AI agents can execute long-horizon, cross-application tasks across three jobs in professional services.

Source

Mercor

Tasks

480

Runs

k = 3

Metric

Mean Score, Pass@1

Modality

Text + cross-application environment

Task format

Multi-step agentic

Question format

Long-horizon cross-application tasks

Canonical harness

Loop (truncated tools)

Grading

Expert-authored rubrics over final deliverables and agent trajectories

Judge

LLM judge over expert rubrics

Released

January 2026

OpenAI

Questions requiring agents to persistently navigate the internet to locate hard-to-find, entangled information. Tests information-discovery competence through persistence and creative problem-solving.

Source

OpenAI

Tasks

130

Runs

k = 4

Metric

Pass@1

Modality

Text + live web browsing

Task format

Multi-step agentic

Question format

Short free-text answer

Canonical harness

Web research agent (max_steps = 100, 3-hour timeout)

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

April 2025

Mercor + Cognition

Measures whether frontier AI agents can execute high-value engineering work, split across observability and integration tasks. Created by Mercor in collaboration with Cognition.

Source

Mercor + Cognition

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Observability and integration engineering tasks

Canonical harness

Agentic coding harness

Grading

Expert-authored rubrics plus repository test suites

Judge

LLM judge + programmatic tests

Released

January 2026

Harbor

Evaluates AI agents on hard, realistic long-horizon command-line tasks: curated tasks with unique environments, human-written solutions, and verification tests.

Source

Harbor

Tasks

89

Runs

k = 3

Metric

Pass@1

Modality

Text (terminal, CLI)

Task format

Multi-step agentic

Question format

Command-line task completion, verified by tests

Canonical harness

Lighthouse Harbor (Terminus)

Grading

Every verification test must pass — no LLM judge

Judge

None — programmatic unit tests

Released

January 2026

Scale

Benchmarks coding agents on codebase comprehension via under-specified, agentic Q&A tasks over real repositories, combining programmatic validation with rubric-based software-quality scoring.

Source

Scale

Tasks

124

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Under-specified agentic codebase Q&A

Canonical harness

host-mini-se

Grading

Programmatic validation plus rubric-based software-quality scoring

Judge

claude-opus-4.5

Released

May 2026

Princeton; OpenAI (Verified subset)

Tests whether models can resolve real-world GitHub issues by editing codebases, requiring multi-file, long-context changes verified by the repository’s test suite.

Source

Princeton; OpenAI (Verified subset)

Tasks

500

Runs

k = 3

Metric

Pass@1

Modality

Text (code repositories)

Task format

Multi-step agentic

Question format

Code patch resolving a GitHub issue, verified by tests

Canonical harness

Lighthouse Inspect

Grading

Fail-to-pass and pass-to-pass tests must pass — no LLM judge

Judge

None — programmatic unit tests

Released

October 2023

University of Waterloo (TIGER-Lab)

An enhanced MMLU adding harder, reasoning-focused questions and expanding choices from 4 to 10 options. MMLU evaluates general knowledge and reasoning across 57 academic and professional subjects.

Source

University of Waterloo (TIGER-Lab)

Tasks

12,032

Runs

k = 1

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Multiple choice (10 options)

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

June 2024

New York University

Graduate-level, “Google-proof” multiple-choice science questions written by domain experts; the Diamond split is the hardest, expert-validated subset.

Source

New York University

Tasks

198

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Multiple choice (4 options)

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

November 2023

Mathematical Association of America

Competition-mathematics problems from the 2025 American Invitational Mathematics Examination, each with an integer answer; measures advanced multi-step mathematical reasoning.

Source

Mathematical Association of America

Tasks

30

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Integer answer (0–999)

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

2025

Artificial Analysis

Long Context Reasoning: 100 hard questions over 234 documents in 30 sets that require synthesizing information across multiple documents and complex reasoning rather than simple retrieval.

Source

Artificial Analysis

Tasks

100

Runs

k = 4

Metric

Pass@1

Modality

Text (~100k tokens per question)

Task format

Single-shot

Question format

Short free-text answer over multi-document sets

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ)

Judge

Gemini 3.5 Flash

Released

August 2025

Meta

Evaluates complex, multi-turn, and system-level instruction following via expert-curated rubrics over 1,600+ prompts.

Source

Meta

Tasks

1,645

Runs

k = 4

Metric

Pass@1

Modality

Text

Task format

Single-shot

Question format

Open-ended prompts graded against expert-written rubrics

Canonical harness

Single-shot agent

Grading

LLM-as-a-judge (LMAAJ) over expert-written rubric criteria

Judge

Gemini 3.5 Flash

Released

November 2025