Introducing APEX-Agents 1.1

APEX-Agents tests a model’s ability to conduct real-world work across Investment Banking, Management Consulting, and Corporate Law. Models need to reason over task requirements scattered across files and communications, mirroring the environments faced by working professionals.

As AI models have advanced, so have their methods for solving APEX-Agents tasks. We’ve updated our benchmark with task specifications, tooling, and environments to ensure leaderboard accuracy. Most notably, APEX-Agents 1.1 no longer rewards noncommittal answers. This ensures that our tasks and grading align with how real-world professionals operate.

The Mercor Research team is confident that this revised benchmark better measures how models can conduct economically valuable work. Claude Fable 5.1 remains on top of the leaderboard with 68.6% on Pass@1, though rankings have shifted throughout the rest of the board.

If you’d like to run APEX-Agents 1.1, you can find our dataset on Hugging Face and Harbor Hub. We’ve also released our evaluation agent on GitHub.

Scattergunning

In early July, we discovered that models were increasing benchmark scores by aggressively hedging their answers. We call it scattergunning. During our updates, we removed scores from the APEX-Agents leaderboard, or did not publish them, for models that engaged in this behavior.

We define scattergunning as “a behavior where models provide multiple answers despite having a single answer that is well-specified by the environment.”

Figure 1. A sample scattergunning response on APEX-Agents task “World221_TR_10.”

Since APEX-Agents grades using binary rubric items that only check for inclusion of correct information, models that scattergun generally gain an advantage. Only one answer needs to be right, so hedging responses increases the chance of passing a given criterion.

In some cases, noncommittal behavior is arguably correct. When information is not specified, models scattergun answers to acknowledge gaps. In professional work, this is often suboptimal. For example, a banker doesn’t usually submit an LBO model with separate cases for calculations with the median or mean: they pick one option for their numbers.

On APEX-Agents 1.1, there is an unambiguous answer that models can and should give. That information is often hard to find, but it is present in conventions or communications.

Overall, we decreased the influence of scattergunning on leaderboard scores through three changes:

1. Three task audits with Mercor domain experts to ensure that our problems are complex and realistic, but well-specified enough to be solvable.
2. A new judge model that scores scattergunned rubric items as a zero.
3. Explicit instructions in the system prompt not to engage in scattergunning.

Collectively, these changes make the benchmark better at measuring a model’s ability to conduct economically valuable work.

Task updates

We’ve selected 80 tasks per domain as our refined benchmark. These tasks were chosen and refined through three audits with Mercor domain experts. Our first two audits focused on ensuring a single correct and well-specified answer exists. The last audit verified the fairness of information placed in the world files to clarify decisions.

Since the original APEX-Agents release, we’ve noticed areas where tasks could have been more specific. We’ve updated the world files to contain all information needed from the failure modes that we’ve discovered. One answer is always correct, but the clarifying information is not entirely in the system prompt. Working professionals typically don’t specify parameters ad nauseam, so, to preserve realism, neither do our tasks.

Instead, preferences are listed in communications, spreadsheets, and documentation. We wanted models to discover this information, much like a professional would search past documents to follow the analytical conventions used by their organization. Models were told in the system prompt that contextual information could be found in world files, including communications.

Figure 2. Sample task hardening used for APEX-Agents 1.1 via the Mail app.

Judge model changes

We’ve also changed our judge model to grade correctly in the presence of scattergunning. DeepSeek-v4-Flash-0731 (temperature = 0.1) provided the strongest baseline accuracy. We then conducted Genetic-Pareto (GEPA) optimization to refine the system prompt. The result was a judge that could accurately score scattergunned rubric items, but not full answers, as a zero.

To conduct the optimization, the Mercor Research team hand-labeled a set of 1,407 rubric items from 337 trajectories across APEX-Agents tasks. On a held-out set of 354 of the rubric items, we observed significant increases in accuracy and detection. This increased the false negative rate from 5.3% to 8.0% compared to the naïve grader, but we believe the massively improved detection rate for scattergunning and better accuracy than the human average for a panel of three independent graders justifies this change.

Figure 3. Scattergun detection rate for the naïve judge against the scattergun-aware judge as well as the overall average accuracy of independent human evaluators vs. scattergun-aware judge.

Grader variance comes from scattergunning being difficult to detect. The difference between a parenthetical note and an answer hedge is often quite subtle. To ensure model interpretations of scattergunning did not affect evaluation, we explicitly defined the term in the system prompt and informed models that scattergunned answers will receive a zero.

Reliability fixes

We also made performance and reliability improvements across our MCP tools. This includes near-complete dependency support for all observed packages invoked by the code execution tool.

We scanned 26,302 trajectories to determine an exhaustive list of all dependencies commonly used by the models. Any dependency that appeared at least 25 total times across the trajectories was included in our environment.

Performance

Our move towards better task specifications has resulted in higher scores in aggregate, but it has also created a much more discriminating and fair benchmark. In particular, we observe significantly more headroom between Pass@4 and Pass^4 scores than in APEX-Agents. This gap shows the percentage of tasks that a model solved at least once but could not solve consistently across four runs. In real-world professional work, the latter score is vital.

Figure 4. Pass@4 to Pass^4 gap comparison, APEX-Agents vs. APEX-Agents 1.1.

In some tasks, models would search for information and successfully detangle messy communications in one run, but not even look for parameters in another. Among top models, GPT-6 Astra was the most consistent. It had the highest Pass^4 score of any model at 56.3%, but had a lower Pass@4 score than other models with comparable Pass@1 scores.

Figure 5. Updated APEX-Agents 1.1 leaderboard ranked by Pass@1. See the full leaderboard.

Run score variance was affected by three main factors: propensity to seek clarifying information, ability to infer parameters from files, and scattergunning penalties. Gemini models searched communications most often, with Gemini 3.8 Flash searching on 93.1% of runs. Claude models searched slightly less, but were generally more effective at converting the information to pass scores with a 24.3% task-level uplift for Claude Fable 5.1 on Pass@1.

Interestingly, Kimi K3 and GPT-5.6 Sol searched communications on only about 15% of runs; but when they did search, they saw significant score uplift at the task-level. In fact, Kimi K3 received the largest task-level uplift of 29.7% on Pass@1. We believe this failure to search under ambiguity, despite prompting, lowered these models’ scores.

Models also saw score reductions for scattergunning. Compared to the naïve grader, some older models saw penalties over 20% on Pass@1 despite being explicitly warned by the system prompt. Newer frontier models generally received no scattergunning penalty with the new tasks, but some received minor deductions. This occasionally caused newer models to underperform their non-scattergun older counterparts.

Evaluations for the frontier

APEX-Agents 1.1’s rank-ordering lines up with other benchmarks; however, there are a few deviations from consensus metrics. We view this as strictly positive. If every benchmark told the same story, the strengths and weaknesses of models would not be surfaced.

It’s exciting when new models top benchmark leaderboards. We see that high score and want to instinctively declare a winner. This ignores the purpose of a benchmark: to show a snapshot of model capabilities. We’re open sourcing our evaluation suite to show the context behind the numbers, so users can see what tasks models failed on and why.

If we want our benchmarks to accurately measure model capabilities, then they must evolve as we push the frontier of AI. As the author of Terminal-Bench writes, “Benchmarks are not static artifacts. They are software and we should maintain them like software.” We couldn’t agree more and we are committed to upholding this standard.

We will continue to update and audit APEX benchmarks to ensure their reliability. We look forward to sharing learnings with the broader research community and to evaluating future models as we push the frontier of artificial intelligence.

See the full APEX-Agents 1.1 leaderboard for updated scores.

Acknowledgements

Mercor would like to thank all of the contributors to this project, including our Mercor domain experts for their guidance during task audits and independent evaluation of our grader model.