Mercor raises $750 million in Series D funding at a $20 billion valuation to expand AI evaluations, post-training, and enterprise agent infrastructure.

Organizing human intelligence for the entire economy 

LLM eval jobs spent 82% of their time waiting for a container. Mercor’s job-level compute scheduler got 6x to 20x more throughput from the same quota.

How we 20x LLM eval throughput

How Mercor built offline evaluations from real incidents to improve its SRE agent, compare agent performance, and reduce cost per alert by over 3x.

Offline Evaluations in Practice: Improving Mercor’s SRE Agent

Introducing Physical-Engineering’s Last Exam (PLE) with Google Cloud AI Research

By Derek Shimozawa, Payments Platform Engineering
“There’s someone you should meet.”
It was June 2025, and that was my recruiter’s entire message. The person she wanted me to meet was Adarsh, Mercor’s cofounder and co-CEO. I had spent the last three years leading Airbnb’s payments product organization and knew almost nothing about Mercor. After that first call with Adarsh, I went to the office to meet Sundeep, Mercor’s President and former CPO at Uber.
What struck me first was the extraordinary concentration of talent and the speed of execution. The office was buzzing: everyone was aligned on the problem, the vision, and the fastest path to decision. Mercor was building the core data infrastructure for frontier AI and reinventing how experts work worldwide. Its ambition was enormous, and the team was uniquely equipped to deliver on it.
By the time I left the office, I knew I had found what I was looking for. To explain why, I first need to explain where I had been.
<h1>The role I was leaving</h1>
At Airbnb, I had the kind of role that looks hard to leave. Under co-founder Nate Blecharczyk, I led the product organization behind every dollar that moved through Airbnb, more than $100 billion a year. Over five years, my scope grew from leading a small team of PMs to multiple groups across five time zones. Together, we built the systems that moved money globally, from payouts and local payment methods to affordability, credit products, billing, ledgering, gift cards, and cross-border settlement.
Before Airbnb’s IPO, our teams established new entities in the UK, India, Luxembourg, and Brazil and shifted critical payments volume to new processors, bringing host payout reliability to near-perfect levels. We built the controls and cross-platform ledger that enabled money to move securely at a massive scale. After the IPO, payments became a strategic growth lever for the company: new local payment methods, lower processing costs, and embedded finance solutions improved guest conversion and affordability. My teams worked on problems that touched millions of guests and hosts and hundreds of billions of dollars.
As my portfolio grew, however, I moved further from the hands-on work that had drawn me to product management. My days increasingly centered on reviewing others’ roadmaps and designs, while even strong ideas could take quarters to reach customers. I remained accountable for the overall direction, but I missed immersing myself in a difficult problem, designing the solution, and shipping the fix.
AI made that distance from hands-on work hard to ignore. These tools are rapidly expanding what a focused builder can accomplish and raising the standard for leadership. Earlier in my career, while working in ordering and payments for Amazon’s Retail/Core Shopping organization, I learned what that bar demands. One of Amazon’s core leadership principles says leaders “Are Right, A Lot.” To me, that doesn’t mean having ready-made answers. It means staying close enough to complex problems to ask the right questions, uncover hidden constraints, earn genuine technical credibility with your teams, and persist through delivery. In a field moving as quickly as AI, product intuition fades fast. I wanted to work somewhere mine would be continually tested and sharpened, not merely applied.
<h1>I’m in</h1>
On our first conversation, Adarsh got straight to the challenge: Mercor was scaling quickly, its financial and payments backbone needed to keep pace, and the team needed leaders who had built through this kind of massive growth before. I knew that kind of problem well. What pulled me in was the chance to solve it at the frontier of an entirely new category of work.
At the time, I had offers to lead larger, more established organizations in SVP and VP roles - the conventional next career step, at least on paper. But as I weighed them against Mercor, I realized I wasn’t seeking a bigger title. I wanted to be closer to the work again: tackling hard problems, navigating ambiguity, and measuring my impact by what I shipped rather than the size of the organization around me.
Sundeep and I spoke candidly about Mercor’s ambition, and the sheer speed and momentum required to achieve it. Transitioning away from a mature organization was a major shift; he encouraged me to step back, reflect on the decision, and follow up. But my mind was made up before I walked out the door: I had to join this rocket ship. That night, I replied, “I’m in.”
<h1>Why the opportunity is so large</h1>
AI advances along three axes: compute, algorithmic innovation, and data. Compute and models dominate the conversation, but data is not merely a supporting input. It determines what a model can learn and how it behaves. For frontier labs and enterprises alike, that makes data a strategic differentiator. The most valuable data is not neatly stored in a database or captured in a training manual. It lives in expert knowledge, complex workflows, bespoke applications, edge cases, and the decisions people make when there isn’t an obvious answer.
This is the opportunity Mercor is built to pursue. We partner with leading AI labs to develop the data and environments their models need, bringing together highly specialized experts, synthetic data, quality control, and rigorous evaluations in a continuous improvement cycle. We bring the same flywheel beyond the frontier, helping enterprises incorporate proprietary workflows, policies, and edge cases into AI systems that improve over time.
Making all of that real requires financial, payments, and contractual systems: paying companies for the data we use, compensating experts, putting the right agreements in place, and invoicing customers. That is where my team sits. It is the operating foundation that converts Mercor’s work across data acquisition, enterprise programs, evaluations, synthetic data, and physical world robotics into a scalable business.
<h1>A smaller team, greater leverage</h1>
At Mercor, I chose to become an individual contributor again. My engineering team is a fraction of the organization I had at Airbnb, but I have more direct ownership of the outcome. A lean team can quickly take a problem from first principles to production without layers of handoffs or approvals. I help shape the global expert experience, the financial infrastructure that supports it, and the product decisions connecting the two.
Since joining, I’ve helped rebuild Mercor’s core transactional and financial architecture. Weekly disbursements that once took more than ten hours at modest volume now run in minutes at <a href="https://www.mercor.com/blog/when-you-go-from-2-million-a-month-to-2-million-a-day/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">30 times the scale</a>. New earnings interfaces give experts full visibility into upcoming disbursements, sharply reducing support inquiries. We automated client invoicing and moved most revenue reconciliation to auditable, real-time streams. These improvements bring immediate value to our community and customers, creating the scalable operational foundation needed for growth.
Decisions that might take weeks in a larger organization turn into action almost instantly here. At Airbnb and Amazon, I learned to lead large organizations and operate at global scale. At Mercor, I apply that experience hands-on at the very frontier of AI.
<h1>Come build with us</h1>
We are hiring people who want to work close to difficult problems: payments infrastructure, contracting systems, data systems, product, operations, evaluations, and expert tooling. You will have real responsibility, direct feedback, and the chance to build before the playbook exists. If that sounds like you, I would love to talk. <a href="http://mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Learn more at mercor.com/careers</a>.

Mercor’s payments platform team pays experts and invoices customers. Derek Shimozawa left Airbnb to build the systems that move millions of dollars a week.

Why I joined Mercor – Derek Shimozawa

AI startup Mercor now valued at $10 billion with new $350 million funding round

AI's next job? Recruiting people to train more AI

AI is learning to do the jobs of doctors, lawyers, and consultants

24 AI startups to watch in 2026

Mercor in the news

With Braden Holstege, Han Pei, Linshu Li, Isaac Chung, Mia Watanabe, Johnny Kim, Brandon Lei
AI Due Diligence Series
This article is Part 3 of a three-part series.
<ul><li>Part 1: <a href="https://www.mercor.com/blog/ai-due-diligence-private-equity/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Why AI Due Diligence Matters</a></li><li>Part 2: <a href="https://www.mercor.com/blog/ai-due-diligence-framework/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">The Mercor AI Due Diligence Framework</a></li><li>Part 3: When AI Strengthens the Investment Thesis (current article)</li></ul>
<h1>When AI Strengthens the Investment Thesis</h1>
In the <a href="https://www.mercor.com/blog/ai-due-diligence-framework/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">previous article</a>, we introduced a two-layer framework for AI diligence. The distinction becomes much clearer when applied to a real investment decision.
During a pre-acquisition diligence engagement, a leading global investment firm asked us to evaluate a market intelligence company serving the hardware devices industry. Frontier models were rapidly improving their ability to search, synthesize, and explain highly technical information, forcing investors to reconsider what customers were actually paying for. If AI could perform an increasing share of the work required to produce specialized research, would customers continue paying for premium market intelligence, or would the economics of the industry fundamentally change?
That question framed the engagement, but it did not determine the investment outcome.
<h2>Looking Beyond the AI Roadmap</h2>
Answering that question required understanding how the business created value long before AI entered the conversation.
Rather than starting with the product roadmap or AI strategy, we started by understanding how the company actually delivered value to customers. We mapped the end-to-end workflow, from collecting proprietary hardware data through reverse engineering, expert analysis, report generation, and customer delivery. At each stage, we asked two questions: what are customers truly paying for, and how would advances in frontier models change that work?
Looking at the business this way revealed that generating technical summaries represented only a small portion of what customers were actually buying. The company’s differentiation came from the combination of proprietary data collection, reverse engineering, laboratory infrastructure, expert interpretation, specialized workflows, and years of accumulated operational knowledge. AI could accelerate portions of that system, but it could not easily replicate the system itself.
This distinction informed the investment thesis. The objective of the diligence was not to determine whether AI posed a threat to the business, but to identify the highest-value opportunities to deploy AI across the company and the investments required to realize them.
<h2>Building the Value Creation Plan</h2>
<h3>Identifying value creation opportunities</h3>
Once we understood how the company created value, the next step was identifying where AI could materially improve the business. Rather than generating a long list of ideas, the goal was to isolate the handful of initiatives capable of meaningfully increasing enterprise value.
Several opportunities emerged quickly. Report generation contained repetitive drafting and formatting work that could be automated while preserving expert review. Market monitoring could be accelerated by continuously scanning new information rather than relying on periodic manual updates. Internal research workflows could be streamlined by automating data extraction, synthesis, and quality checks, allowing specialists to spend more time on analysis and custom advisory work.
<h3>Assessing readiness</h3>
Each opportunity was then evaluated for technical feasibility and organizational readiness. AI researchers assessed whether frontier models could reliably perform the required work today or were likely to within the investment horizon. Forward Deployed Engineers estimated the engineering effort, infrastructure changes, and integration work required to deploy each capability in production.
<h3>Prioritizing implementation</h3>
Finally, the initiatives were prioritized based on expected enterprise value, implementation effort, strategic importance, and the pace of frontier model improvement. Some were ready for immediate execution, while others depended on model capabilities that were expected to mature over the coming years. The result was a sequenced implementation roadmap that balanced financial impact with execution feasibility.
<h2>Investment Outcome</h2>
The diligence ultimately identified approximately $4 million of net value that could be realized within six months. Looking further ahead, the same initiatives represented an estimated $20 million in annual run-rate impact by 2031.
Those figures quantified the opportunity, but they were not the most important outcome of the engagement. The greater value came from separating market exposure from company capability. AI was clearly changing the economics of the market, but that did not mean every participant would be affected in the same way.
The result was not simply greater confidence in the investment. It was a clearer understanding of why the company was positioned to outperform as AI reshaped the industry and a practical roadmap for creating additional enterprise value throughout the investment period.
Read the full series:
<ul><li>Part 1: <a href="https://www.mercor.com/blog/ai-due-diligence-private-equity/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Why AI Due Diligence Matters</a></li><li>Part 2: <a href="https://www.mercor.com/blog/ai-due-diligence-framework/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">The Mercor AI Due Diligence Framework</a></li><li>Part 3: When AI Strengthens the Investment Thesis (current article)</li></ul>

Mercor applied AI due diligence framework in private equity diligence engagement to identify value creation opportunities and strengthen the investment thesis.

AI Due Diligence Case Study: Strengthening the Investment Thesis | Mercor

Part 3: When AI Strengthens the Investment Thesis

With Brandon Lei, Tannvi Banerjee, Daniel He, Vidit Agrawal, Johnny Kim
Two things are going wrong in agent development. Teams are not building credible evals, and they are not thinking holistically about the levers available to improve an agent once they have them.
Your engineering or product team builds an agent that reaches roughly 90% accuracy on its eval suite (or worse, they aren’t using an eval suite at all), but it still does not feel ready for production. It hallucinates in cases nobody predicted. It still needs a person watching it, correcting it, and cleaning up after it. So the team rewrites the prompt again, but the agent still never reaches meaningful production volume.
We see this across our customer engagements, and it almost always comes down to two failure modes.
First, the agent evals don’t credibly define what good looks like. They were generated by an engineer, or by an AI model an engineer prompted, without enough input from the domain experts or end users who actually do the work. A 90% on an eval suite like that does not tell you the agent is 90% good. It tells you the agent agrees with a rough guess at what good looks like. That is why the score reads 90% and your gut says otherwise.
Second, the team is only optimizing the prompt. Even with a credible suite, hill climbing on one dimension of a system that has many will stall. Everything in the agent is tunable: the model, the skills, the context, the harness. In practice, teams often spend far more time iterating on the prompt than on the rest of the system.
<h2>Credible evals come from people who do the work</h2>
Reliable software development depends on tests. The analog for agents is evals. Test-driven development becomes eval-driven development.
The difference is where the spec comes from. When an engineer writes a unit test, they already know the right answer, because they decided what the function was supposed to do. When an engineer writes an agent eval, the right answer lives in a domain they may not practice.
The engineer building a Telco internet customer service agent knows Python. They do not necessarily know what separates a good customer service call from a bad one.
Take that internet customer service agent. An expert will tell you:
<ul><li>Call verify_identity before any tool that reads account data</li><li>Call check_outage before troubleshooting the individual line</li><li>Run get_modem_diagnostics and try reset_modem before asking the customer to unplug anything</li></ul>
The engineer may not have thought to check any of that. Those checks become part of the eval suite. The eval suite is the set of tasks, environments, verifiers, and criteria used to measure the agent. The broader platform is what turns that measurement into an optimization loop e.g., running the evals, identifying where failures originate, and routing that feedback to the part of the agent stack that needs to change.
Most engineers building the agent have experienced customer support as users, not operators. The gap is even wider in domains like finance, auditing, or underwriting.
The hard part of building a credible eval suite is knowing what to test. That judgment comes from people who understand the work.
The experts who will build your agent evals come from one of two places:
<ol><li>Your own employees: The right choice when the agent runs an internal process specific to how your company works. These employees understand the workflow, company-specific policies, and edge cases the agent needs to handle. These employees should be well versed in how to write agent evaluations and prepared to dedicate significant time to writing them.</li><li>External practitioners: The right choice when the agent is customer or product-facing, like a shopping agent or a support agent, or when the work takes a general professional skill rather than company-specific knowledge e.g., performing an audit, or underwriting a risk, or reviewing a contract.</li></ol>
For the second case, Mercor sources domain experts from its 7M+ contractor network who have done the specific work the agent is being built to perform. Their judgment helps define the criteria, edge cases, and failure modes the evaluation system needs to capture.
<h2>Everything in the agent is hill climbable, not just the prompt</h2>
Ask most teams what they can change about an agent and you get prompts and instructions. That is a fraction of what is available.
Everything in the agent configuration is hill climbable:
<ul><li>The prompt</li><li>The skills</li><li>The context</li><li>The tool definitions</li><li>The model</li><li>The harness</li><li>The deterministic logic</li></ul>
Each can be tuned against the same evaluation standard.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/90d07687abee09790ebd7e6c905081053126cac3-1820x640.png" alt="The levers of AI agent optimization, including prompts, skills, context, tools, models, harnesses, and deterministic logic." style="width: 100%; border-radius: 10px;" />
Fig 1: The levers of agent optimization
Take the example mentioned earlier: a Telco internet customer service agent.
The eval says the agent should try reset_modem before asking the customer to touch anything. Instead the production agent spends six minutes walking them through unplugging the router, and never calls the tool. There are at least five ways to fix that, and one of them is the prompt.
<table class="table-cell-border" style="width: 100%; border-collapse: collapse; margin: 1em 0;"><thead style="text-align: left;"><tr><th>Lever</th><th>What you would change</th></tr></thead><tbody style="text-align: left;"><tr><td>Prompt instructions</td><td>Add a rule to try the remote reset first. This may work, but it adds another instruction the model has to balance against the rest of the prompt.</td></tr><tr><td>Tool definitions</td><td>reset_modem is described as &quot;resets customer equipment.&quot; Say instead that it is the first-line remedy and takes thirty seconds.</td></tr><tr><td>Harness</td><td>Run check_outage and get_modem_diagnostics concurrently as soon as the account is verified, so the results are available before troubleshooting begins.</td></tr><tr><td>Model selection</td><td>Move to a model that holds a multi-step procedure over a long call, and let the eval tell you whether it was worth the cost.</td></tr><tr><td>Deterministic logic</td><td>Run reset_modem on any no-sync fault before the call reaches troubleshooting. </td></tr></tbody></table>
A change that improves one behavior can degrade another, so each meaningful change should be evaluated against the full suite before it ships.
This is the development loop we build at Mercor. Domain experts help define the criteria, edge cases, and failure modes that matter. Those standards become the eval suite. Our platform then runs and audits those evals, surfaces where the agent is failing, and routes that feedback to the relevant layer of the agent stack. Our forward-deployed engineers can use that signal to hill climb across the prompt, model, skills, context, tools, harness, and deterministic logic, evaluating each change against the same standard.
<h2>Build the eval suite alongside the agent</h2>
The eval suite should not be a one-time test added at the end of development. It should evolve alongside the agent.
For a new agent, that means defining what good looks like early and expanding the suite as new edge cases and failure modes emerge. For an agent already in production or stuck in POC, it is not too late. Production failures are often some of the best inputs for strengthening the eval suite. Capture what went wrong, turn it into a repeatable test, and use it to prevent the same failure from recurring.
The eval suite sets the bar. The platform turns that bar into a development loop. As the agent encounters new failure modes, the suite gets stronger; as the suite gets stronger, teams get better signals on what to improve across the prompt, model, tools, skills, context, harness, and deterministic logic.
Without a credible standard, optimization is guesswork. With one, teams can identify what is failing, change the right part of the system, and measure whether the agent actually got better.

Why AI agents get stuck in POC, and how credible evals and optimization across models, prompts, tools, and harnesses help teams reach production.

Why AI Agents Get Stuck in POC | Mercor

Why AI Agents Get Stuck in POC

With Brandon Lei, Braden Holstege, Joe Kaplan, Alex Gonzalez, and Imani Parker-Fong
<h2>In one paragraph</h2>
Your submission tool went live and cycle time on the small commercial book improved. Twelve months later, it is performing much as it did at launch.
Underwriters improve through regular file reviews and earn greater authority as their judgment proves reliable, but most AI systems have no equivalent process. Insurance already has a model for managing this risk. Some underwriting decisions can be made independently within defined limits, while others require referral or specialist review. AI touches 17% of underwriting tasks today, and executives expect that to reach 75% within three years.¹
As AI takes on more of the underwriting workflow, carriers can apply the same model by turning existing file reviews into continuous evaluations and using the results to determine what AI can handle independently, what requires review and where an underwriter still needs to make the decision.
<h2>1. AI needs the equivalent of a file review</h2>
The challenge is that underwriting quality can be difficult to evaluate quickly. A case file or quote letter is rarely simply right or wrong, and much of the underlying submission data still arrives in documents.
In some cases, the ultimate test may not come until a claim arrives. By then, the model, data feed and underwriting guidelines may all have changed, making it difficult to trace the original decision back to what the system saw and produced at the time.
Reviewing every file manually is not a practical answer either. The underwriters best equipped to assess the work are already constrained for time. Underwriters spend 41 to 43% of their time on administrative work,² and 64% of underwriting executives cite talent as one of the external forces they expect to have the greatest impact on their business.¹
<h2>2. Turn the file review into an eval</h2>
Most carriers already use peer review or underwriting audits to assess the quality of underwriting decisions. A senior underwriter reviews a sample of bound files against the carrier’s guidelines, checking whether the exposure was captured correctly, the classification was appropriate, the loss history was reconciled, the right referrals were made and the rate was justified. The limitation is that these reviews typically happen after the fact and cover only a fraction of the book.
For AI, carriers can take the same criteria and turn them into a scorecard that evaluates every file the system produces before a quote reaches the broker. In technical terms, this is an eval.³ Mercor uses this approach to evaluate AI on complex professional work, and in underwriting the evaluation can be built around the carrier’s own files, guidelines and standards.
<h2>3. Build the eval from real underwriting work</h2>
An eval has five components that together define the task, what good work looks like and how the AI’s performance is measured.³
The file. A sandboxed copy of what the AI sees including the broker thread, the statement of values, five years of loss runs, survey and guideline in force.
The task. One defined deliverable at a time so the submission-to-quote process can be evaluated as distinct pieces of work.
The benchmark. You do not need to start from scratch. Real files that experienced underwriters have already reviewed provide the benchmark, along with the source material and context needed to establish what good work looks like.
The scorecard. Weighted criteria with partial credit that preserve the complexity of a real submission. If the valuation on location five is stale, the AI needs to flag it.
The marker. The automated reviewer that applies the scorecard, calibrated against expert judgment until it reliably reflects the underwriting team’s quality bar.
Benchmarks also need to reflect the context in which an underwriting decision was made. A file written in a hard market can penalize a correct decision in a softening one, so each benchmark carries an effective date, guideline version and rate level, and is updated as those conditions change.³
<h2>4. Authority should follow performance</h2>
Once the quality of the AI’s work can be measured, carriers can use that evidence to determine where the system operates independently and where an underwriter remains in the loop. The existing tiers of underwriting authority provide the structure.⁶
Straight through. The AI can act within agreed limits when its work clears the quality threshold, the risk falls within defined appetite and the underlying data is reliable. The system should also preserve a clear record of the information and logic behind each decision so the work can be audited and explained.
Prepare and refer. The AI assembles the file and recommends an action, but an underwriter makes the decision.
Specialist review. Complex risks, incomplete data and work below the quality threshold stay with a specialist.
The practical place to start is with work an underwriter can verify quickly. Zurich reports up to two hours saved per submission on U.S. Middle Market underwriting narratives, and a Sixfold case study of Zurich North America reports that 80% of submissions were processed in the initial phase while the underwriter retained the risk decision.⁴ Accenture reports that QBE can process 100% of broker submissions in the product lines where its AI underwriting solutions are in production.⁷
As the carrier collects more evidence, the level of authority can change. If the AI consistently clears the required quality threshold for a particular class of risk, its authority can expand within defined limits. If performance declines after a model, guideline or data change, that authority can contract. If a class continues to perform poorly, the referral stays.
The referral path also preserves an important part of underwriter development. Junior underwriters continue to see the facts, the recommendation and the expert judgment together rather than having the work that builds judgment disappear entirely into automation. Accenture’s survey points in the same direction, with 81% of respondents expecting AI to create new roles and 76% expecting easier knowledge transfer.¹ 
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f5be73b161ca8ae360cfaf097e6a39fd4f6891be-1101x621.png" alt="Diagram showing how AI underwriting files are routed to straight-through processing, underwriter referral, or specialist review based on evaluation results." style="width: 100%; border-radius: 10px;" />
Figure 1. How each file is routed. The AI case manager assembles and prices the file, and each file then lands in one of three levels of authority. The automated file review runs before and after the quote, so underwriter overrides and claims outcomes feed back into the scorecard. Structure adapted from note 6.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/49a945877fe64d9cd2b9e5974a36fe732612c4ef-1101x401.png" alt="AI underwriting workflow from submission receipt through evaluation, referral, underwriting decision, and quote." style="width: 100%; border-radius: 10px;" />
Figure 2. One submission, end to end. The checks between steps are automated markings, so a file below the threshold is held, reworked or referred before a broker sees it. Process adapted from note 6.
<h2>5. Use overrides and outcomes to improve the eval</h2>
Every reviewed file creates a record of what the system saw, what it produced, how it scored and where an underwriter disagreed. Repeated overrides can reveal gaps in the scorecard, while claims outcomes can show which criteria actually matter to underwriting performance.³
The same feedback loop also makes changes to the system easier to manage. If performance drops after a new model, data source or guideline is introduced, the carrier can see where quality changed and adjust the authority accordingly.
The system should also preserve a clear record of the information and logic behind each decision so the work can be audited and explained.
<h2>6. Start with one book</h2>
The first ninety days do not need a new system. Pick one book where submissions are frequent. Collect twenty files experienced underwriters have already reviewed and work with a small number of them to turn their judgment into an explicit scorecard.³
Waiting for better technology is not a plan. On Mercor’s open benchmark of expert built professional work, the leading model passes 43.5 % of tasks at one attempt, so even the strongest systems still fail a majority of them.⁵ Carriers already know how to manage high stakes judgment without requiring perfection. They set limits, review the work and expand authority when the evidence supports it.
AI should earn its authority the same way.
<h2>Notes</h2>
<ol><li>Accenture, Underwriting Rewritten, August 2025. Survey of 430 senior underwriting executives across Life, Commercial P&amp;C and Personal P&amp;C in 11 countries, with 25 follow-up interviews. Source of the 17% and 75% AI task figures, the finding that 64% of executives identify talent as an external force expected to have a major impact, and the findings that 81% expect AI to create new roles and 76% expect easier knowledge transfer.</li><li>Capgemini, Unleashing Growth: The Evolving Role of Underwriters, 2024. Reports that commercial and personal lines underwriters spend 41% to 43% of their time on administrative activities.</li><li>Mercor, Agent Eval Systems and Enterprise Evals, 2026. Source for Mercor’s approach to defining tasks, benchmarks, scorecards and automated reviewers for evaluating AI work. Evaluations can be used to grade live traffic, gate delivery or test changes before release. The build sequence, calibration process and use of expert-defined quality criteria described in this article reflect Mercor’s methodology.</li><li>Zurich Insurance Group, Annual Report 2025, and Sixfold’s Zurich North America case study. Zurich reports up to two hours saved per submission on U.S. Middle Market underwriting narratives. Sixfold reports that 80% of submissions were processed in the initial phase, with the underwriter retaining the risk decision. Mercor was not involved in this work.</li><li>Mercor, APEX Agents, 2026. An open benchmark of 480 rubric-graded tasks across 33 expert-built environments in investment banking, management consulting and corporate law. As of August 2026, the leading entry passes 43.5% of tasks at one attempt. Rankings may change as new models are released, so the current result should be checked before publication.</li><li>McKinsey, The Future Underwriting Operating System: From Inbox to AI Nerve Center, June 2026. Source for the human-governed underwriting operating model and the three routes described here as straight through, prepare and refer, and specialist review.</li><li>Accenture, 2025 client example. QBE Insurance Group is scaling AI-powered underwriting solutions co-developed with Accenture. In the product lines where those solutions are in production, Accenture reports that QBE can process 100% of broker submissions. Mercor was not involved in this work.</li></ol>

Vaithiyanathan Sundaram

Member of Technical Staff

Learn how insurers can use AI evals and existing file reviews to measure quality, set authority limits and safely scale AI across underwriting workflows.

AI Underwriting: How AI Should Earn Underwriting Authority

Your AI should earn underwriting authority the way a new underwriter does

With Braden Holstege, Han Pei, Linshu Li, Isaac Chung, Mia Watanabe, Johnny Kim, Brandon Lei
AI Due Diligence Series
This article is Part 2 of a three-part series.
<ul><li>Part 1: <a href="https://www.mercor.com/blog/ai-due-diligence-private-equity/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Why AI Due Diligence Matters</a></li><li>Part 2: The Mercor AI Due Diligence Framework (current article)</li><li>Part 3: <a href="https://www.mercor.com/blog/ai-due-diligence-case-study/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">When AI Strengthens the Investment Thesis</a></li></ul>
<h1>An AI Due Diligence Framework for Private Equity</h1>
In the <a href="https://www.mercor.com/blog/ai-due-diligence-private-equity/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">previous article</a>, we argued that AI diligence is about more than evaluating technology adoption or mitigating AI risk. When executed correctly, AI due diligence is key to unlocking outsized returns, allowing investors to identify exactly where targeted investments can help companies evolve into the next-generation versions of themselves.
At Mercor, we anchor our AI diligence around two closely-related questions:
<ol><li>How will AI change the underlying market? Will AI shift profit pools toward specific players, compress or expand margins, unlock new customer segments, or create an existential threat where frontier models out-compete legacy capabilities?</li><li>Is the target positioned to win in that market? Does the target have the assets, technology, talent, and operating model required to capture this upside? If not, what specific investments are required for them to become a leading player?</li></ol>
<h2>Layer One: Understanding Market Exposure</h2>
The first layer examines whether AI will substitute the market’s underlying value proposition or materially compress the workflows used to deliver it.
For example, a compliance software company built around deep regulatory expertise, is unlikely to disappear simply because frontier models improve. Its economics may still change considerably if AI makes research, document review and policy interpretation faster and less expensive. Companies that adapt can use those advances to improve margins, launch new products, and serve customers more effectively. Legacy players that fail to respond may instead face margin compression, new competitors, and declining market share.
This analysis helps investors understand how the market is likely to evolve. It may reveal that the cost of delivering the core product will fall, that new entrants can compete with smaller teams, or that value will shift toward companies with proprietary data, trusted distribution, or the ability to integrate AI into existing customer workflows.
Market exposure alone does not determine whether an investment is attractive. It establishes the conditions under which the target will need to compete.
<h2>Layer Two: Evaluating the Target</h2>
While underlying market conditions set the baseline trajectory, the second layer evaluates the Target&#39;s specific right to win.
Rather than asking whether the company has adopted AI, this assessment evaluates where AI can create meaningful business value, whether those opportunities are technically feasible, and whether the organization has the capabilities to execute them.
The core objective of this assessment is to pinpoint the specific investments required for the Target to deploy AI effectively and adapt to an AI-evolving market. Market tailwinds mean little if a company lacks the operational foundation to capitalize on them. Identifying these practical value creation opportunities, such as structurally reducing the cost basis, accelerating commercial velocity, or capturing net-new revenue through AI-enabled features, can become the primary lever for driving outsized returns.
To define this investment roadmap, our target assessment centers on three specific analyses: identifying value creation opportunities, assessing organizational awareness, and prioritizing implementation.
<h3>Identifying Value Creation Opportunities</h3>
The aim is to identify the small number of AI initiatives that could materially improve the investment outcome.
Some opportunities reduce the cost of existing work. Others support revenue growth by improving the product, increasing customer adoption, entering a new market, strengthening pricing power, or creating an offering that was not previously possible. Cost savings are often easier to quantify, but the largest sources of value may come from revenue opportunities that change how the company competes.
Finding those opportunities requires a detailed understanding of how the business operates. Customer interviews, workflow analysis, and input from industry practitioners help break the company’s core activities into their component parts and show where expertise creates value, where work is constrained, and where AI can meaningfully change the outcome.
That analysis must then be paired with an accurate view of model capabilities and production requirements. AI researchers can assess whether models are capable of performing the relevant work today and how quickly those capabilities are improving. Forward deployed engineers can determine what it would take to turn a promising use case into a reliable production system.
<h3>Assessing Organizational Readiness</h3>
A feasible initiative may still be a poor investment if the company is not equipped to deliver it.
The readiness assessment examines the company’s existing technology architecture, data infrastructure, engineering capacity, internal skills, and available headcount. It also considers the capabilities required to build and operate agentic systems, including agent orchestration, LLM applications, evaluation and benchmarking systems, enterprise integrations, workflow automation, production machine learning, context engineering, and prompt engineering.
Many companies lack meaningful experience in these areas. They may be able to build an initial prototype but struggle to integrate it into existing systems, evaluate its performance, or operate it reliably at scale. The diligence process should therefore determine which initiatives the company can execute internally, where important capability gaps exist, and whether an external partner could materially reduce execution risk or compress the timeline.
<h3>Prioritizing Implementation</h3>
Implementation planning is not a rote ranking exercise. The most valuable initiatives are not always the easiest to build, and the initiatives with the highest modeled ROI are not always the ones that should be pursued first.
Each opportunity should be assessed based on the business value it creates, the engineering time and direct investment required, its relationship to company strategy, and the urgency created by market conditions. The analysis should also identify dependencies between initiatives, including foundational work that must be completed before higher-value opportunities become feasible.
Estimating implementation effort requires practical engineering judgment. Teams need to understand the likely person-weeks of work, the cash investment required, the complexity of enterprise integrations, and whether the initiative depends on model capabilities that are not yet reliable enough for production. In some cases, the right decision is to move immediately. In others, it may be better to prepare the underlying infrastructure while waiting for model performance to improve.
The result is a sequenced implementation plan grounded in both financial returns and strategic judgment. It identifies which initiatives matter most, when they should be pursued, and whether the company should build them internally or use an external partner to accelerate execution.
<h2>Why AI Diligence Requires a Dedicated Team</h2>
Conducting rigorous AI diligence requires a forward-looking view of model capabilities, practical deployment experience, and detailed knowledge of the target&#39;s workflows.
Mercor brings those perspectives together through AI researchers who understand frontier model capabilities, Forward Deployed Engineers with production deployment experience, and a network of industry experts who understand the target’s workflows and competitive dynamics. That combination enables investors to identify the opportunities that matter, understand what it will take to execute them, and prioritize them based on both technical feasibility and business impact.
Read the full series
<ul><li>Part 1: <a href="https://www.mercor.com/blog/ai-due-diligence-private-equity/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Why AI Due Diligence Matters</a></li><li>Part 2: The Mercor AI Due Diligence Framework (current article)</li><li>Part 3: <a href="https://www.mercor.com/blog/ai-due-diligence-case-study/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">When AI Strengthens the Investment Thesis</a></li></ul>

A practical AI due diligence framework for evaluating market exposure, target readiness, and post-acquisition value creation.

AI Due Diligence Framework for Private Equity | Mercor

Part 2: An AI Due Diligence Framework for Private Equity

With Vaithiyanathan Sundaram, Michael Wang, Biao Zhang, Braden Holstege, Han Pei, Alex Gonzalez, Helen Rhee, Imani Parker-Fong
Frontier models are becoming both more capable and more expensive. As enterprises scale agents into production, token bills can grow quickly, and the most capable model is rarely the right model for every task.
The goal is not to run the best model everywhere. It is to run the least expensive model that still delivers the required level of performance for the work in front of it.
That makes model routing a cost-quality optimization problem, also known as the Pareto optimization problem. Every agent has a frontier of tradeoffs between performance and cost. The job is to operate on that frontier instead of overpaying for capability a task does not need.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/c51aa2a7f5199dfe171ea67371f91eea6b7830c5-2048x1307.png" alt="Figure 1: Each point represents a model-routing configuration tested against the same evaluation suite. The Pareto frontier contains the configurations that are not dominated by another option on both cost and quality." style="width: 100%; border-radius: 10px;" />
Figure 1: Each point represents a model-routing configuration tested against the same evaluation suite. The Pareto frontier contains the configurations that are not dominated by another option on both cost and quality.
How you assign models to work determines where you land, whether the choice is static, query-level, or made dynamically inside the agent. At Mercor, we believe the only reliable way to find that frontier, and keep an agent on it as models change, is rigorous evaluation.
<h2>Evaluations are the system of record</h2>
In <a href="https://www.mercor.com/blog/agent-eval-systems/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Agent Eval Systems</a>, we argued that an evaluation suite is the decision layer for agent quality. It defines what good means for a specific agent, on its actual workload, inside its actual environment.
That same suite should govern cost optimization. Here, good performance and quality corresponds to higher scores on the verifiers rigorously defined in its evaluation suite. Once quality is measurable, every model swap, routing rule, and harness change can be judged by whether performance held, what tradeoffs appeared, and what it cost.
Without that measurement layer, lower token spend is ambiguous. It may mean the agent got more efficient. It may also mean it started skipping work, mishandling edge cases, or failing in places the cost dashboard cannot see.
This reframes model routing. Public leaderboards can tell you how models perform on shared benchmarks, but they cannot tell you which model your specific agent needs for your tasks in your environment. That is why every agent needs a comprehensive evaluation suite. It is the only way to see the real tradeoffs when a model or harness changes. From there, model routing becomes a harness experimentation question of which architecture and model mix gets closest to the Pareto frontier on your evaluation suite.
<h2>Three ways to route models</h2>
There are three broad strategies for assigning models to work inside an agent: 
<h3>1. Static Model Selection</h3>
Sweep every candidate model across the full evaluation suite, then pick the single model that delivers the required performance at the lowest cost.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/67464cc687fca216996574424493618ed196010e-774x540.png" alt="" style="width: 100%; border-radius: 10px;" />
<h3>2. Query Classification Routing</h3>
Classify each incoming query by difficulty, then route it to a model sized for that difficulty. For example, a lightweight router might send hard queries to Opus and routine ones to Haiku.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/6e5eb8f7b6e9993b3f84e97878b27985279d4faa-2080x1360.png" alt="" style="width: 100%; border-radius: 10px;" />
<h3>3. Dynamic Delegation</h3>
Keep a frontier model as the main agent and let it delegate subtasks to subagents running smaller, cheaper models, deciding at runtime, with full context, what it can safely hand off.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ebcceb4446254c2de2f898e7a84e20d18bd5a912-1034x692.png" alt="" style="width: 100%; border-radius: 10px;" />
These three are the most established patterns, not an exhaustive list. Hybrid approaches exist, and new ones will emerge as agent architectures evolve.
Each of these can be the right answer. Which one is right depends on the shape of the agent’s workload, especially how widely task difficulty varies. That shape is exactly what an evaluation suite reveals.
<h2>Workload shape determines routing strategy</h2>
Some agents operate on workloads with relatively concentrated difficulty. The tasks vary in topic, but most sit in the same rough band of complexity. For these agents, static model selection can be enough.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7b41724b3b291fbca161626b52907e83e6a8f9c9-2048x1331.png" alt="Figure 2: Static model selection works best when task difficulty is concentrated. Routing and delegation become more useful when task difficulty varies widely across tasks or subtasks." style="width: 100%; border-radius: 10px;" />
Figure 2: Static model selection works best when task difficulty is concentrated. Routing and delegation become more useful when task difficulty varies widely across tasks or subtasks.
An internal support triage agent is the canonical example. If most tasks involve classifying issues, retrieving known policy, or drafting bounded responses, a full model sweep may reveal a clear knee in the cost-performance curve. Here, the best model is selected against the criterion the deployment actually needs, whether that means the strongest quality-to-cost ratio, the lowest cost above a quality floor, the highest quality under a cost ceiling, or the model that performs best on the quality gates that matter most. In agents with bounded complexity, past a certain capability tier, we have found that additional model quality buys little additional task performance, and the routing problem is mostly solved with static model selection.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/882f104ca595540fd295716f03604c111e1ae9ea-2048x1178.png" alt="Figure 3: A full model sweep for one agent on Mercor Enterprise Platform. Several mid-tier models score within a few points of the most expensive option at a fraction of the cost per task. When the curve looks like this, the knee is easy to find and static selection is enough." style="width: 100%; border-radius: 10px;" />
Figure 3: A full model sweep for one agent on Mercor Enterprise Platform. Several mid-tier models score within a few points of the most expensive option at a fraction of the cost per task. When the curve looks like this, the knee is easy to find and static selection is enough.
Other enterprise agents face workloads with much wider difficulty variance. A due diligence agent might draft a voice of customer summary based on several interviews or be asked to create an end to end diligence deck based on a comprehensive dataroom. A software engineering agent might make a one-line configuration fix in one task and debug a failing distributed deployment in another.
For these agents, any single fixed model is likely to be inefficient on part of the workload. A smaller model may be enough for the easy tasks, while a frontier model is still needed for the hardest ones.
The intuitive fix is query-classification routing, where each incoming request is scored for difficulty before being routed to the right model. That only works when the request contains enough information to predict how hard the task will actually be.
Often, it does not. The complexity of a task is not always visible in the query. &quot;Fix the failing login flow&quot; might be a one-line change or a three-hour investigation across logs, services and deployment state. Nothing in the query text distinguishes the two. A classifier that only sees the query is guessing about the part of the problem that actually determines difficulty. From our internal testing and benchmarking, model routing classifiers tend to over-fit to the tasks they were tuned on rather than capturing generalized judgement of what tasks are truly easy or difficult. When the classifier is wrong, cheap models are handed tasks they cannot finish, or expensive models are handed queries a weaker model could already fulfill. The result is lower accuracy and higher cost - the worst of both.
This is where dynamic delegation becomes useful. Instead of making the routing decision before the agent has context, the system moves the decision inside the agent. The main agent inspects the environment, breaks down the task, and decides what to hand off. It keeps the work that determines correctness, including the plan, the ambiguous calls, and the final review, while a cheaper model handles the token-heavy middle of the trajectory, such as exploration, test runs, and mechanical edits. Because that middle is where most of the tokens go, moving it to a smaller model can lower cost meaningfully while keeping quality largely intact.
Delegation is not free. It pays for reliability by running a frontier model as the orchestrator on every task, including the easy ones. That can be wasteful for very simple workloads, but when difficulty is uneven and hard to predict up front, delegation can win if the cost of misrouting hard tasks outweighs the orchestration overhead. That is an empirical question for the evaluation suite to answer.
<h2>The eval suite is the hard part</h2>
Everything above assumes the existence of an evaluation suite that faithfully represents the agent&#39;s real workload, but building that suite is harder than building the infrastructure around it.
Logging, tracing, and scoring harnesses are commodity capabilities at this point. What is not a commodity is knowing what to measure. A routing decision is only as good as the verifiers behind it, and those verifiers need deep company context. They need to know which tasks actually show up in production, how often they occur, which failures carry real business cost, which ones are cosmetic, what the environment looks like when the agent inspects it, and what a domain expert would consider acceptable for each task type.
None of that context lives in a dashboard. It lives with the people who do the work today. An eval suite built without them measures a workload that does not exist, and every routing decision optimized against it lands somewhere other than the real frontier. Teams that buy observability tooling and stop there end up with excellent visibility into the wrong questions.
Mercor Enterprise builds the evaluation layer behind these decisions. Our FDEs work with the customer’s domain experts to turn production context into a domain-specific eval suite, then use it to test model and harness changes against the same quality bar. The output is not a generic router, but is instead a measured cost-quality strategy that can adapt as the workload and model landscape change.

Mercor on model routing as an eval problem: using evaluation to decide which model runs each enterprise task, and why routing quality depends on measurement.


Model Routing as an Eval Problem | Mercor Enterprise

Model routing is an evaluation problem

With Braden Holstege, Han Pei, Linshu Li, Isaac Chung, Mia Watanabe, Johnny Kim, Brandon Lei
AI Due Diligence Series
This article is Part 1 of a three-part series.
<ul><li>Part 1: Why AI Due Diligence Matters (current article)</li><li>Part 2: <a href="https://www.mercor.com/blog/ai-due-diligence-framework/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">The Mercor AI Due Diligence Framework</a></li><li>Part 3: <a href="https://www.mercor.com/blog/ai-due-diligence-case-study/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">When AI Strengthens the Investment Thesis</a></li></ul>
<h1>The Missing Question in Private Equity Due Diligence</h1>
As AI has become increasingly relevant, many firms now include AI in their traditional diligence process. However, unlike standard commercial or tech diligences, the industry lacks consensus on the critical AI questions to evaluate targets against.
The key opportunity is identifying which targets have untapped AI-driven value creation opportunities and using that insight to make better investment decisions.
Sophisticated AI diligence is not simply about identifying risk. It helps investors determine which companies are best positioned to benefit as industries evolve, where AI can create additional enterprise value after an acquisition, and ultimately which investments are likely to generate stronger returns.
Doing that requires answering two connected questions.
The first is how AI is changing the underlying market. 
In most B2B markets, the deep domain expertise, customer relationships, and institutional knowledge required to operate make wholesale disruption unlikely. Instead, these markets are evolving. The primary risk is the rapid shift of profit pools and which player(s) are best-positioned to capture it. The first layer of diligence assesses this underlying market exposure: how the overall market dynamics will be impacted by access to frontier model capabilities.
Frontier models are reshaping dynamics across industries by reducing the cost of some workflows, lowering barriers to entry, and changing where value accrues. In some markets, proprietary data, specialized expertise, or customer relationships become even more valuable. In others, previously durable competitive advantages begin to erode. Understanding those changes helps investors evaluate how the market itself is evolving, but it does not determine which company is most attractive.
The second question is how well the target company is positioned to capture that change.
Two companies operating in the same market can have very different futures. One may have proprietary data that becomes more valuable as AI improves, a product strategy that expands into new customer segments, or the technical capability to bring AI-powered offerings to market ahead of competitors. Another may have a strong business today but lack the engineering capabilities, organizational agility, or operating model required to adapt as the market evolves. Sophisticated AI diligence helps distinguish between those outcomes before an investment is made.
That distinction leads to better investment decisions. Rather than narrowly viewing AI as a source of disruption risk or productivity gain, investors can identify overlooked assets that are well positioned to benefit from shifting market dynamics and generate outsized returns. For example, AI may unlock entirely new customer segments that were previously uneconomical to serve. Moving quickly to capture those customers can create a durable first-mover advantage. AI can also enable new pricing models that better capture increased consumption and emerging usage patterns, such as agentic workflows, creating meaningful upside over the life of the investment. The objective is not simply to understand how AI affects a business today. It is to identify which businesses are positioned to generate outsized returns over the life of the investment.
Answering those questions requires capabilities that extend beyond the traditional scope of commercial and technical diligence. This on-the-ground experience is essential for understanding real-world feasibility, practical ROI, and the specific organizational adaptations required to implement AI effectively. Ultimately, this provides a clear lens on exactly what offerings and processes are genuinely being disrupted, empowering investors to make accurate &quot;build vs. buy vs. wait&quot; decisions. That requires AI researchers, engineers who have deployed production AI systems, and domain experts who understand how work actually happens inside an industry. Bringing those perspectives together makes it possible to evaluate not only what AI can do in theory, but how it is likely to change the economics of a specific business.
The firms that consistently outperform over the next decade are unlikely to be those that simply identify AI risk. They will be the firms that recognize where AI creates new sources of enterprise value, identify the companies best positioned to capture that value, and develop a strategy to realize it after the investment is made.
The obvious next question is how to evaluate those opportunities systematically. In the next article, we’ll introduce the Mercor AI Investment Framework, a two-layer framework that evaluates both market exposure and target-specific capabilities to help investors make better AI investment decisions.

Read the full series
<ul><li>Part 1: Why AI Due Diligence Matters (current article)</li><li>Part 2: <a href="https://www.mercor.com/blog/ai-due-diligence-framework/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">The Mercor AI Due Diligence Framework</a></li><li>Part 3: <a href="https://www.mercor.com/blog/ai-due-diligence-case-study/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">When AI Strengthens the Investment Thesis</a></li></ul>

Learn how AI due diligence helps private equity firms identify overlooked value creation opportunities, evaluate targets, and make better investment decisions.

AI Due Diligence for Private Equity | Mercor

Part 1: The Missing Question in Private Equity Due Diligence

<img src="https://cdn.sanity.io/images/h6s14f4z/production/9eb151685945537437f8c2621aeb0311748149c5-1520x710.png" alt="" style="width: 100%; border-radius: 10px;" />
Building an AI agent is no longer the hard part. Foundation models can already perform many of the individual tasks required in enterprise workflows. The challenge is deploying systems that can be trusted in production by understanding how work is performed, defining what success looks like for that organization, integrating with existing enterprise environments, and continuously measuring performance over time.
That was the challenge 1Password and Mercor set out to solve when building an SRE agent for 1Password’s engineering organization. Mercor used Codex as the execution layer for an SRE agent that investigates alerts across monitoring systems, deployment history, source code, feature flags, documentation, and past incidents, then surfaces findings for engineers to review. Using Codex, Mercor built the customer-specific context, integrations, and evaluation framework needed to deploy the agent through Mercor Enterprise AI Suite.
<blockquote>An SRE agent operates across boundaries when it traverses monitors, repositories, deployment history, and incident records. The leadership task is not simply to ask whether the model is capable. It is to define who owns the workflow, which systems the agent can reach, what evidence it must return, and which decisions remain human. With Mercor and Codex, we are accelerating incident investigation while keeping consequential production actions human-owned and evidence-backed.</blockquote>
Nancy Wang, CTO, 1Password
<h2>Understanding 1Password’s Environment</h2>
SRE spans monitoring systems, deployment history, source code, documentation, historical incidents, and cross-team engineering conversations. Much of the required context exists across multiple systems, making it time-consuming to find the relevant information, connect it across sources, and quickly assemble a reliable picture of an incident. Mercor’s Forward Deployed Engineers worked with the 1Password team to map investigation patterns and capture relationships between systems, documentation, and services in a context graph reflecting how engineers actually investigated incidents.
1Password and Mercor collaborated to define how the agent would integrate into 1Password’s existing engineering environment. Mercor Enterprise AI Suite supports SaaS, customer-managed, in-VPC, on-premises, BYOC, and BYOK deployments, allowing the deployment architecture to match 1Password’s infrastructure without changing the underlying evaluation methodology.
<h2>Developing the Evaluation Suite</h2>
Many AI projects treat evaluation as the final step. An agent is built first, tested against a handful of examples, and refined only after problems begin to appear. That approach can work for demonstrations, but it breaks down in production, where success depends on consistent performance across a broad range of real-world scenarios.
Before optimizing the agent, Mercor built a Mercor Enterprise AI Evaluation Suite based on real engineering workflows. The teams drew on more than 300 historical incidents to develop evaluation material, connecting the original alerts with their root causes and remediation. Every scenario defined the task, expected behavior, and success criteria, creating a benchmark against which every subsequent change was measured.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/774a88acb36492af28044c2b5bb08ceecb8b8c0f-2048x1434.png" alt="" style="width: 100%; border-radius: 10px;" />
Exhibit 1: Mercor Representative AI Eval Suite
Every change to the agent was evaluated against the current regression suite. Missed context, incomplete recommendations, and retrieval failures became new evaluation cases, allowing the system to improve through measurable iteration rather than intuition.
With the evaluation framework in place, development became a continuous optimization loop. The teams evaluated each iteration against the current regression suite and used the results to decide which configurations to promote to production.
<blockquote>This SRE agent is not just an agent that can investigate incidents. It has to investigate incidents the way 1Password does, across its monitors, deployment history, code, and incident systems. We evaluated the task, the trajectory, and the output: did it look at the right evidence? Reason across systems? Produce a recommendation an engineer could act on? Codex gave us the execution layer, and the eval suite gave us a quantitative way to improve it before each change reached production.</blockquote>
Adarsh Hiremath, Co-founder and Co-CEO, Mercor
<h2>Building the SRE Agent</h2>
Most of an agent’s incident investigation is the same work an engineer performs at a terminal: pulling up the monitor that fired, reviewing any recent PRs, checking which feature flags changed, and often revisiting the postmortem from the last time something similar broke. Codex is built for that work: long-horizon, tool-heavy, code and terminal environments. Mercor used Codex as the execution layer for the SRE agent.
When a monitoring alert fired or an engineer declared an incident, Mercor’s event integration started a Codex investigation. The agent used read-only access to systems including Datadog for metrics, logs, events, monitors, and service ownership; GitHub for diffs across roughly 840 active code repositories, FireHydrant for past incidents and timelines, Unleash for feature flags, PagerDuty for alert context, and sometimes Sentry for errors. It also consulted engineering documentation and used local shell tools to correlate evidence across these systems and investigate recent changes. The assessment identified the likely affected service or failure mode, the responsible team, the most likely cause, and the first action an engineer should consider. Engineers retained responsibility for every production decision and action.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/9451acb5397ff5b1b0dc5cb2d164afea83b9de11-1170x2048.png" alt="" style="width: 100%; border-radius: 10px;" />
Exhibit 2: How the SRE agent turns incident signals into evidence-backed findings for human review, and how it continuously improves.
Codex was a strong fit for the SRE agent because it supports long-running, tool-intensive investigations across code and terminal environments. Mercor kept the orchestration, 1Password-specific context, integrations, and skills in a versioned layer around the Codex executor. Mercor and 1Password evaluated each configuration against the same evaluation suite before advancing it to production. This design also gave both companies’ engineering teams a repeatable way to evaluate new Codex releases and model versions before adopting them. Both companies found that Codex provided an easy-to-use foundation to evolve the agent quickly and safely.
<h2>Deploying Into 1Password’s Environment</h2>
Enterprise deployment means the system must integrate with existing infrastructure, governance, and security controls.
For 1Password, credentials remained securely managed within the organization’s existing credential management system and were retrieved dynamically at runtime rather than embedded in the agent. The agent produced a structured assessment of the likely affected service or failure mode, the responsible team, the most likely cause, and the first action an engineer should consider. Engineers could review every recommendation before taking action, while the evaluation framework continued validating improvements after deployment.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/b6cb5ac25ff484c4de869e90b2904930ec64255d-2048x1170.png" alt="" style="width: 100%; border-radius: 10px;" />
Exhibit 3: Mercor Enterprise AI Suite in Production
The evaluation suite started with a subset of real historical incident scenarios. Every scenario defined the task, expected behavior, and success criteria. Failures and newly encountered scenarios became new evals in the suite, and both teams treated development as a continuous optimization loop.
<h2>Evaluation results and early production observations</h2>
The goal was not to prove that the agent could solve one incident. It was to measure whether the same configuration performed consistently across different failure patterns. The shipped configuration was evaluated in three independent trials against each of 26 historical-incident tasks, for 78 runs total, earning 95% of the grading criteria. The rubric measured evidence gathering, root-cause reasoning, affected-service identification, distractor avoidance, and output quality. This was an evaluation score, not a measure of production accuracy.
Trust depended on more than a plausible answer. Every cited metric, pull request, and log query in the evaluated reports could be traced to a tool call the agent had actually made. No fabricated citations were observed in the defined evaluation set.
In a retrospective before-and-after comparison using historical production incident data, estimated median time to mitigate was 57% lower in the post-deployment cohort. The comparison included 199 earlier incidents and 32 post-deployment incidents and provides a directional comparison, not evidence of causation.
The clearest early production observation was speed to its findings: a median of under 4 minutes across 30 live agent-generated incident reports. These results are preliminary and not a controlled measure of production impact.
<h2>1Password moving the agent to production</h2>
Over a median time of under 4 minutes, for an average cost of under $2 the agent runs roughly 50 queries across multiple systems covering metrics, logs, monitors, past incidents, on-call, feature flags and code changes across some 840 repositories, then posts a written report in the incident channel.
During an incident, the agent greets the responder with a first pass across metrics, logs, monitors, code changes, flags and past incidents, or provides it within minutes. Most of the responder’s resolution time is spent finding the change, and that is the part the agent front-loads.
<h3>Setting authority before granting access</h3>
The 1Password SRE team defined the agent’s authority before launch. It could read the systems responders use to investigate an incident, but engineers retained control over declaring incidents, paging people, rolling back releases, running fixes, and merging code. We also run the agent in a secure container that prevents the agent from changing its own environment.
To illustrate one access decision, our team declined write access to the incident-management tool because it would have also allowed the agent to page people. The tool offered no way to grant write access without that paging capability, so we kept the agent’s access read-only.
Every investigation is also bound by a $3 budget.
<h3>Taking ownership of the service</h3>
When the agent entered production at 1Password, it began investigating declared incidents and monitor alerts routed to it, posting findings directly into the incident’s Slack channel. Six days later, we opened access to the entire engineering organization.
When Mercor’s engagement ended in late August, 1Password’s Observability team took full ownership. Between July 30 and September 24, our merged 85 changes spanning accuracy, output, evaluation, and operations. The handover made the ongoing commitment clear: the agent needed an accountable team to maintain it, inspect its work, and improve it as the environment changed.
<h3>Turning wrong answers into tests</h3>
Reviewing production investigations exposed a recurring weakness. We compared the agent’s conclusions with the findings in subsequent incident reviews. The agent too readily blamed a recent code change, including cases where nothing had changed.
A plausible but wrong explanation can delay responders. Upon noticing the pattern, we required every report to state explicitly whether a change caused the incident and graded that answer against the incident review. Our team turned a recurring reasoning failure into a specific requirement it could test. That gave engineers a way to track the behavior across future changes.
<h3>Continuous improvement after launch</h3>
We built three layers of review into ongoing operations to answer three different questions:
<ol><li>A benchmark drawn from past incidents<ol><li>Could the agent repeat its performance?</li></ol></li><li>Retrospective grading against completed incident reviews<ol><li>Did its conclusion match what actually happened?</li></ol></li><li>Engineers reading the agent’s output:<ol><li>Could a responder use the report?</li></ol></li></ol>
Since September 9, we traced every production investigation end to end and scored it automatically. Those traces let engineers inspect how the agent reached an answer, including which evidence it gathered. Incident reviews provide a later check against the eventual findings. Together, these mechanisms give us a way to identify misses and turn them into the next evaluation case.
<h3>Writing for all levels of responders</h3>
Our SRE team also rewrote the output (delivered in Slack) in plain English so responders could act on it without needing deep familiarity with the affected system. Reports include the agent’s level of confidence and identify any evidence that would change its conclusion. The rewrite makes the agent’s reports more accessible to all engineers, including those who might be less familiar with internal names.
When the responder starts reading, a first pass across metrics, logs, monitors, code changes, flags and past incidents is already there or arrives within minutes. Finding the change is where most of the resolution time goes, and that is the part the agent front-loads.
<h2>What’s Next</h2>
As of September 24, the agent remained in beta while handling real investigations on routed incidents. The team reported a cost of approximately $1.90 per investigation. Operating it also required the continuing engineering work behind those 85 changes. The team’s next priorities were to widen live grading and sustain the accuracy gains.
More broadly, 1Password will continue using agents in production. 1Password’s standard for agentic incident response is clear: respond quickly, show the evidence, stay within the permissions, keep review and action in human hands, and improve when engineers find a miss. That standard gives the team a basis for deciding where the agent should take on more work.
With Codex as the execution layer and evaluations guiding each production update, 1Password now has a repeatable way to improve the agent, validate new configurations, and adopt future model upgrades without rebuilding the surrounding system.

Biao Zhang

Michael Wang

How Mercor uses Codex to build a production SRE agent with 1Password

With Michael Wang, Richter Brzeski, Isaac Chung, Brandon Lei, Daniel He, Braden Holstege, Han Pei, Imani Parker-Fong
Enterprise discussions about agent deployment usually focus on two questions: where the agent runs and whether customer data ever leaves the environment. Both matter, but they describe the deployment boundary rather than how the system changes after it reaches production.
An agent that cannot learn from production behaves much like any other piece of infrastructure. The same failures recur, the same fixes are applied, and the cost of those mistakes remains unchanged over time. That assumption works well for systems like databases, where stability is the goal, but agents are different. Their value comes from observing the outcomes of their work and using those outcomes to improve future behavior. Without that feedback loop, an agent remains fixed at whatever quality it had on the day it was deployed.
We separate the system into three planes because it is solving three different problems. One determines what the agent is allowed to do. One executes the work safely and reliably. One improves future behavior from production signals. Those responsibilities evolve independently, so they should not share the same control plane.
The first is the routing and policy plane, which authorizes every action the agent takes by selecting models, brokering credentials, enforcing tool permissions, applying egress policy, and recording requests. The runtime plane is responsible for execution, providing isolated sandboxes, durable workflows, retries, verification, and production telemetry. The intelligence plane closes the loop by learning from behavioral signals such as verifier failures, retries, latency, cost, and human overrides, improving routing, evaluations, and control logic without requiring customer data to leave the deployment boundary.
The easiest way to see why those responsibilities are separated is to follow a single overnight reconciliation run.
<h2>The ticket arrives</h2>
A reconciliation job fails inside a financial institution’s VPC. The report still needs to be ready before markets open, even though no operator is online.
The agent inspects scheduler logs, queries the warehouse, compares ledger records, generates diagnostic code, identifies the source of the failure, and prepares a corrected reconciliation for review. Along the way it reaches the wrong conclusion once, receives feedback from a verifier, revises its approach, and produces a correct result for an analyst to approve before the reporting deadline.
From an operational perspective, this is a routine run. The interesting part is not what happens during execution, but the decisions embedded in the system long before the job ever started.
<h2>Routing and Policy: The agent requests access</h2>
The first thing the agent needs to do is read data from three internal systems.
A straightforward implementation gives the model credentials directly. Service account tokens live in environment variables, mounted secrets, or prompt context and are available throughout execution. That approach works until the model encounters untrusted input. At that point, a prompt injection attack becomes a credential theft attack because the model has access to credentials that can be exfiltrated.
We instead route every request through the routing and policy plane. When the agent needs access to a resource, it requests a capability rather than presenting a credential. The broker evaluates policy, mints a short-lived token scoped to that operation, and returns only the permissions required to complete the task. The underlying credential never becomes visible to the model.
That same request is also the point where model selection happens. At production scale, routing cannot be reduced to a fixed choice. Sending every request to the largest model increases cost and latency without improving many tasks, while always choosing the smallest model gradually erodes quality. The routing decision becomes another policy evaluation that considers the task, latency requirements, customer preferences, and any other constraints defined for the deployment.
By the time execution begins, authorization, credential brokerage, model selection, and egress policy have already been applied. From the agent’s perspective, interacting with an internal system is a single API call. The routing layer absorbs the complexity while enforcing the security boundary.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/82146c385504dd883b0f8c77ef69eb03969e19df-1278x1066.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Execution: The agent writes code</h2>
The agent generates a diagnostic script and executes it against production data.
From the runtime’s perspective, it makes no difference whether that code was written by a model or a developer. Every execution is treated as untrusted.
Each run starts in a fresh sandbox with only the resources explicitly granted to that task. There is no ambient network access, no mounted credentials, and no persistent state carried from previous executions. If the code attempts to reach an external service, read a secret it was never granted, or leave artifacts behind for a future run, those operations fail because the environment simply does not expose those capabilities.
Isolation solves only part of the problem. Production systems also have to contend with queue delays, transient failures, process crashes, and network interruptions. The runtime preserves execution state, retries failed steps, records intermediate artifacts, and resumes work from the point of failure rather than starting the entire task over.
Verification is part of execution rather than something layered on afterward. Before an output reaches a downstream system, it can be accepted, rejected, revised, or escalated for human review based on the result of a verifier.
Enterprise deployments also introduce another constraint. Most platforms become multi-tenant somewhere in the stack, which means customer isolation cannot depend on operational processes or developer discipline. The runtime has to enforce those boundaries by construction.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/5ba52065ddc5de6f64c28e0b9befa1977ef52282-1267x1097.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>The agent gets the wrong answer</h2>
The diagnostic completes and proposes a reconciliation, but one upstream transaction has been counted twice.
No operator is available to catch the mistake. Before the result reaches a downstream system, a verifier compares the proposed reconciliation against an independently computed control total. The invariant fails, the output is rejected, and the agent receives structured feedback describing the discrepancy. Using that feedback, it revises the reconciliation and produces a corrected result.
This is why verification has to be part of execution rather than something reserved for offline evaluation. Offline benchmarks tell you how a system performed on a test suite. Runtime verification determines whether an incorrect result ever reaches a customer or a production workflow.
The runtime records the entire execution alongside standard production telemetry, including tool invocations, retries, verifier failures, warehouse timeouts, and execution traces. Those signals are exported through OpenTelemetry so they appear in the customer’s existing observability platform alongside the rest of their infrastructure.
Much of this architecture will look familiar to anyone who has worked with container orchestration or CI systems. The execution model is similar, but the workload is fundamentally different because the software making decisions is no longer deterministic.
<h2>Learning: The run finishes</h2>
The report enters a human approval queue. The sandbox is destroyed, temporary credentials expire, and customer data remains inside the deployment boundary.
What leaves the environment is not the execution itself, but a description of how the system behaved. The routing layer learns which model recovered from the task most effectively. The runtime records a warehouse timeout that suggests a retry policy should change. The verifier identifies a reconciliation failure that belongs in the evaluation suite, and human approval confirms that the revised result was correct.
None of those observations require access to ledger contents, prompts, or generated code. They capture the behavior of the system rather than the customer data it processed.
Those behavioral signals become inputs to the intelligence plane. Over time they expand the evaluation suite, improve routing decisions, strengthen verifiers, and refine execution policies. Without them, every run is treated as though it were the first. Routing never improves, retry policies remain static, evaluation suites drift away from production, and verifier quality is frozen at whatever level it reached before deployment.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/dd320034fc4bcf320dc90866c562ea3e617c47c7-1241x1098.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>What crosses the boundary?</h2>
Behavioral telemetry is still telemetry. Retry patterns, verifier outcomes, and workflow characteristics can reveal information about how a system operates, so the boundary between customer data and behavioral signals should be explicit.
Customer documents, prompts, model context, credentials, generated code, scratchpads, and application payloads remain inside the customer environment. The only information that leaves is behavioral metadata describing how the execution unfolded, such as the task category, verifier outcome, model identifier, latency, retry count, execution cost, or whether a human intervened.
That channel should be governed like any other production interface. Customers should be able to inspect it, filter it, rate-limit it, or disable it entirely using their existing infrastructure.
Some organizations choose to disable the channel completely. In that configuration, the intelligence plane runs inside the deployment boundary, including fully air-gapped environments. The architecture remains the same, but behavioral improvements arrive through software updates rather than continuously incorporating production observations.
Customer data, credentials, and execution remain inside the deployment boundary in either model. The only difference is how new behavioral knowledge reaches the system.
<h2>Questions worth asking</h2>
Labels such as SaaS, VPC, BYOC, and air-gapped describe where software runs. They say much less about how the system is designed.
When evaluating an agent platform, we have found a different set of questions to be more informative.
<ul><li>What information can leave the deployment boundary?</li><li>Can that boundary be verified independently?</li><li>Where do model requests actually transit?</li><li>What production signals influence the system after deployment?</li><li>What changes if those signals are no longer available?</li></ul>
The first three establish how customer data is protected. The last two determine whether the system continues improving after it has been deployed.
<h2>The next morning</h2>
By the time the operations team arrives, the reconciliation is waiting for review. An analyst approves the result, and the report is delivered on schedule.
Nothing about the deployment boundary has changed. Customer data never left the environment, credentials were never exposed, and execution remained isolated. What changed was the system’s understanding of the work it performed. Routing decisions become slightly better informed, evaluation suites expand to cover another production failure, retry policies become more representative of real workloads, and verifiers improve over time.
Looking back at the reconciliation run, each plane solved a different problem. The routing and policy plane determined what the agent was allowed to do. The runtime ensured the work was completed safely and reliably. The intelligence plane captured what the system learned so the next run could improve. Those responsibilities change independently over the lifetime of a deployment, which is why we keep them architecturally separate.

Enterprise agent architecture explained: how the routing, runtime, and intelligence planes keep customer data in-boundary while the system keeps improving.

Enterprise Agent Architecture: The Three Planes | Mercor

The Three Planes of Enterprise Agent Systems

Bertie Vidgen

Researcher

Edward Hu

Head of AI Modeling

How frontier models miss risk in routine enterprise work

Today we&#39;re releasing <a href="https://openworldengineer.mercor.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">OpenWorldEngineer (OWE-Bench)</a>, a frontier coding benchmark of 26 long-horizon engineering and research tasks across software engineering, physical engineering, machine learning, and scientific computing.
We built OWE-Bench to test whether AI agents can make real progress on long-horizon engineering projects, ranging from GPU kernels to aircraft design. Every task is graded on unseen cases and scored on a continuous scale. This means that an agent that speeds up a program halfway to the ceiling earns 50% rather than a pass-or-fail zero.
Claude Fable 5.2 tops the current leaderboard with a score of 54.6%, using an average of 211 million tokens per task attempt. GPT-6 Astra follows with a score of 44.2% using 89 million tokens, and Claude Opus 5.5 at 41.6% using 253 million tokens. No model is close to the benchmark ceiling. Fable 5.2 leads all four domains, with Astra within a few points in physical engineering and scientific computing. The benchmark also successfully differentiates between small and large models, with Inkling and Beam only scoring 1.1% and 3.3% respectively.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f32beca963f70533c67d2c095436733c4a45abc4-3200x1800.png" alt="OWE-Bench leaderboard bar chart of mean normalized scores across 26 tasks, 8 trials per model. Claude Fable 5.2 leads at 54.6%, followed by GPT-6 Astra at 44.2% and Claude Opus 5.5 at 41.6%, with 95% confidence intervals. Latest update November 11, 2026." style="width: 100%; border-radius: 10px;" />
OWE-Bench can be downloaded <a href="https://huggingface.co/datasets/mercor/OpenWorldEngineer" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a> and will grow with the community. The next version will be open to outside contributors.
<h2>Engineers work in the open world</h2>
Engineering requires more than recall. Engineers read papers, check specifications, and adapt open-source implementations. Much of the knowledge an engineer needs to complete their work cannot be found in the project brief alone.
Tasks for OWE-Bench come in two shapes. In scaffolded tasks, the agent starts from a working codebase and pushes a measurable target such as kernel speed, latency, or error. In end-to-end tasks, it builds the whole artifact from a bare environment: a trained model, a data pipeline, a mechanical design. Depending on the task, agents get 4 to 18 hours per attempt (median 8), for 257 agent-hours across one pass over all tasks, on hardware matched to the task, such as a CPU sandbox, NVIDIA H100 or B300. Seven tasks use a GPU.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/b79c8f9f363adc77d722c9ed0a3cfc7c00786bf6-860x990.gif" alt="Animated comparison of CAD reconstructions against hold-out target geometry for GPT-6 Astra (volute), Beam 1.0 (guide casing), and DeepSeek V4.1 Flash (pump housing), showing the median and best trials with a signed-deviation color scale in millimeters." style="width: 100%; border-radius: 10px;" />
The 26 tasks span four domains:
<ul><li>Software engineering (8 tasks): databases, data pipelines, interpreters, and layout, such as exactly-once streaming payments ETL into Iceberg tables with alerts under fault injection</li><li>Physical engineering (5 tasks): CAD reconstruction, structural design, and aircraft, such as a parametric Design/Build/Fly electric aircraft scored over 34 hidden flights spanning three missions</li><li>Machine learning (7 tasks): GPU kernels, training, and inference, such as post-training Qwen3-8B on chess tactics through a metered training service, scored on 1,000 private puzzles</li><li>Scientific computing (6 tasks): chemistry and earth-science modeling, such as relocating an earthquake sequence from supplied seismic observations</li></ul>
<h2>Tasks built by practitioners</h2>
Each task starts with a domain expert suggesting a project derived from their real-world work. Domain experts recruited through Mercor write a proposal that describes the relevant public work, the deliverable, and how it will be verified. After maintainers review the proposal, the expert builds the task environment and its automated verifier.
Each task goes through layered review, and can only be accepted if approved by three human reviewers. Automated static checks and an LM-assisted rubric grade for structure, reproducibility, instructions, score design and budgets. We red teamed the quality of the task grading adversarially to find problems that need fixing.
<h2>Graded on what the agent ships</h2>
Evaluation begins with agents submitting an artifact, like a model checkpoint, a data pipeline, a kernel package, a CAD program, or an aircraft design. They are then scored offline by an independent verifier operating without network access, testing the deliverable on CPUs and GPUs using unexposed, held-out inputs. Performance is measured strictly through execution, simulation, or geometric analysis rather than model judgments.
Tasks report scores in their own units (e.g., speed in milliseconds, accuracy). We put them on one scale so the untouched baseline scores 0% and the task&#39;s ceiling scores 100%. A model&#39;s task score is the mean of its eight attempts, and its leaderboard score is the mean over all 26 tasks. We audited every trial for reward hacking. Exploit attempts appear in 39 of 2,912 trials (1.3%). In our grading we include a check for reward hacking, and exploits that increase the score are set to zero.
<h2>What the results show</h2>
Fable 5.2 leads in every domain. It scores 62.2% in software engineering against 49.7% for Claude Opus 5.5 (+12.2pp), and 54.6% in machine learning against 47.8% for Claude Sonnet 5.5 (+6.8pp). The gap with the next best model is far smaller in physical engineering (59.6% vs. 57.3%) and scientific computing (40.2% vs. 39.4%), where GPT-6 Astra is within the margin of error.
We read the trajectories for the 8 runs on the 26 tasks from Claude Opus 5.5, Gemini 4 Argon and GPT-6.1 Sol (i.e., 624 trajectories total) to understand where they fail. Most failed runs shared one problem: the model checked its code on simpler tests than the actual ones in the tasks. Agents validated on small inputs, familiar data or their own simulators, then shipped work that broke at the scale, temperature or input shapes the tasks actually required. GPT-6.1 Sol and Gemini 4 Argon also lose points by overfitting to the practice data, for example by adding shortcuts that only work on public samples.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/c80422328d186c3cf7dd8ac14275c5151678ebbb-2048x1152.png" alt="Two scatter plots titled Score vs. tokens and cost, comparing OWE-Bench mean scores against tokens per attempt and model cost per attempt on log scales, with models on the efficiency frontier in bold. Latest update November 11, 2026." style="width: 100%; border-radius: 10px;" />
Cost varies sharply. GPT-6 Astra scores 44.2% while spending $234.63 on tokens per attempt; Claude Fable 5.2 averages 54.6% while spending $105.08; GPT-6.1 Sol averages 41.2% while spending $52.12, and Claude Sonnet 5.5 averages 39.0%, spending $38.65.
<h2>Get involved</h2>
All 26 tasks in OWE-Bench are fully public: instructions, starting code, development cases, scoring anchors, graders, hidden test sets, and the generators and seeds used to produce them. The graders are the same ones used for the leaderboard.
We’re planning to open OWE-2.0 to the community and are accepting submissions for additional tasks <a href="https://openworldengineer.mercor.com/contribute" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a>. We&#39;ll share more details soon and all contributors will be fully acknowledged in future releases. OWE-Bench was built by Mercor’s Code Data and Research teams. We’re <a href="https://www.mercor.com/careers/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">hiring</a>, so come join us.
Acknowledgments: Thanks to Chirag Mahapatra for backing the project from the start, to Victor Barres for research review and guidance, to Brad Johnson for helping with marketing, to Harish Shankar for helping with analysis, to the Mercor experts who proposed, built, reviewed, and audited the tasks, and to the broader Code Data and Research teams.

Cindy Yang

Aravind Selvan

Senior Staff Engineer

Daniel Babalola

Strategic Projects Lead

Introducing OWE-Bench for Long-Horizon Engineering Projects

How the everyday record of your company’s work is becoming a new source of revenue in the AI economy.
Every company produces a detailed record of how work gets done. It shows the decisions and escalations, the handoffs between teams, and what broke and how your team fixed them. That record lives in your email, your Slack, your documents, your tickets, your code, and the systems each team runs on. Most companies treat it as a storage cost.
Mercor has paid over $50 million for it.
<h2>Why enterprise data Is becoming valuable now</h2>
For years, AI development has relied heavily on publicly available information, including websites, books, code, and other digital content. As AI moves beyond generating content toward completing real tasks, developers increasingly need examples of how work happens inside organizations.
Consider an ordinary work day. Consider a sales operations workflow. An SDR might need to identify an account in Salesforce, review recent conversations in Gmail, retrieve relevant context from Notion, update a forecast, and notify the appropriate team. An engineer might move between GitHub, Postgres, and Slack to resolve an incident. A recruiter might coordinate across Ashby, Slack, and Rippling to bring a new employee onboard. Each of those workflows are full of decisions and interactions that are difficult to reconstruct from public information alone.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/41865126af3fd6c0f8c006e0b9e262b200c61499-4800x2100.png" alt="" style="width: 100%; border-radius: 10px;" />
This shift is creating demand for proprietary enterprise data across the AI ecosystem, from developers building foundation models to companies developing specialized agents and enterprise applications. The challenge is that much of this information is fragmented across systems, difficult to extract, and contains sensitive business or personal information. Even companies that see the value rarely have the infrastructure to process it or commercial relationships to price and bring it to market.
Mercor has both.
AI developers already bring their data needs to Mercor, and Mercor runs the data extraction and anonymization in-house. A company gains access to this market through a single partnership.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ae78a7df5375dac8d33cb3465731d5b195717f8e-4800x1590.png" alt="The Emerging Enterprise Data Economy" style="width: 100%; border-radius: 10px;" />
<h2>What makes a dataset valuable</h2>
The opportunity extends across industries and business functions. Engineering organizations accumulate years of code changes, technical discussions, and issue resolutions. Sales teams generate customer interactions and deal histories. Professional services firms produce research, analysis, and project documentation. Each represents a potentially valuable record of specialized work.
The commercial value of this information depends on several factors, including its uniqueness, historical depth, volume, and the relationships between records across systems. A history that links, for example, a Slack thread to a ticket to the code change shows why a certain decision was made.
Through <a href="https://data.mercor.com/?utm_source=linkedin&amp;utm_medium=blog_post&amp;utm_campaign=2026-11-12" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Mercor Data Partnerships</a>, organizations can license qualifying datasets to AI developers while retaining ownership of their underlying business information. Mercor runs the process required to extract, process, and prepare the data for commercialization.
<h3>How a data partnership works</h3>
Mercor manages the process from initial evaluation to payment.
1. Tell Mercor about your data: Companies begin by sharing which systems they use, how much historical information they hold, and the approximate volumes available. Mercor evaluates the data against current demand from AI developers to identify potential licensing opportunities.
2. Receive an offer and agree on terms: Mercor provides an indicative commercial offer based on the characteristics of the available data. We work together to establish the licensing scope, permitted uses, security requirements, and commercial terms before proceeding.
3. Connect your systems: Mercor connects to the agreed systems and extracts the relevant records. The data is then processed through anonymization and quality-control workflows to remove identifying information while preserving useful relationships between records.
4. Complete delivery and receive payment: Once the agreed dataset meets the applicable requirements, Mercor coordinates delivery and compensation according to the partnership agreement.
The process is designed to minimize the technical work required from your business. Rather than building custom extraction pipelines, preparing datasets, or managing relationships with individual AI developers, partners work through Mercor’s existing infrastructure.
<iframe src="https://www.youtube.com/embed/tr_6T7kCd34?si=LT6dqL2nSomabWXn" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="The Mercor Data Partnership Process" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
What you provide: the tools in scope, sign-off on the license, read-only access, and your payout details.
What Mercor manages: evaluating and pricing your data, drafting the agreement, secure extraction, anonymization and quality checks, and getting you paid.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/556b8fc980e179b95fb17ca74b3327d8b709afe4-5940x1680.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Protecting your company’s data</h2>
Trust and privacy are fundamental to the decision and at the forefront of how we built our system. Enterprise records can contain employee information, customer details, confidential communications, and commercially sensitive material. A responsible partnership must address how that information is accessed, processed, protected, and used.
Mercor’s anonymization infrastructure is designed to remove personally identifying information while preserving the relationships that make enterprise records useful. For example, when the same customer appears across an email, support ticket, and CRM record, consistent replacement can preserve the connection between those interactions without retaining the customer’s original identity.
Identifying information can appear in free-form text, document contents, metadata, and contextual details that indirectly identify individuals or organizations. Mercor runs detection, transformation, and validation procedures appropriate to the data being processed. Anonymization reduces disclosure risk but does not automatically eliminate every re-identification risk.
Data licensing also requires clear contractual boundaries. Before participating, companies should understand which systems and categories of information are included, where their data is processed, who can access it, what uses are permitted, and how long information is retained. Mercor works with partners to establish the applicable terms and processing requirements before extraction begins.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/5cd8502545c623633339f9ea41f9dc552a101efb-3870x1888.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Beyond historical data</h2>
Historical enterprise records are the starting point, but the opportunity may extend further as companies adopt AI agents in their own operations.
Agents generate records of how they complete tasks, interact with software, and respond to real-world conditions. With the right permissions and safeguards, these interactions can become another category of licensable data. Organizations may eventually be able to generate recurring revenue from new operational data rather than relying solely on a one-time historical dataset.
Mercor is developing the infrastructure to support this emerging market, connecting companies that generate valuable data with AI developers who need access to real-world information.
For businesses, the opportunity starts with understanding the value of what they already have.
<h3>Find out what your data is worth</h3>
Your company may already be generating information that AI developers want. Explore Mercor Data Partnerships to understand its potential value and how a secure licensing partnership works.
Explore Mercor Data Partnerships: <a href="https://data.mercor.com/?utm_source=linkedin&amp;utm_medium=blog_post&amp;utm_campaign=2026-11-12" target="_blank" rel="noopener noreferrer" class="text-indigo-600">https://www.data.mercor.com</a>

Turn your company’s operational data into revenue. Learn how Mercor helps businesses license enterprise data to AI developers.

Enterprise Data Monetization: Turn Data Into Revenue

Your Company Is Sitting on a New Asset: Its Data

Last month, we introduced <a href="https://www.mercor.com/blog/introducing-tasknet-a-new-taxonomy-of-work/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">TaskNet</a> with researchers from Mercor, Stanford, O*NET Resource Center, and the National Bureau of Economic Research. TaskNet builds on the structure of <a href="https://www.onetcenter.org/overview.html" target="_blank" rel="noopener noreferrer" class="text-indigo-600">O*NET</a>, the US Department of Labor&#39;s widely used map of occupational tasks.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/0864bb080784bcec65f0fcc503847ddac9cd25e6-2048x1058.png" alt="AI has already changed some jobs far more than others" style="width: 100%; border-radius: 10px;" />
From TaskNet and our conversations with workers, we learned that AI is changing some jobs more than others. The missing piece was asking these workers how they felt about AI progress and how they would shape it going forward. Following work by <a href="https://arxiv.org/abs/2506.06576" target="_blank" rel="noopener noreferrer" class="text-indigo-600">WORKBank (Shao et al., 2025)</a>, <a href="https://www.nber.org/papers/w35677" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Bick, Blandin, Deming and Schumacher</a> and others, we surveyed participants from our original TaskNet occupations about how AI has changed their jobs, which tasks they see as core to their occupation and how they view its future.
<h2>Running the survey</h2>
Each worker received a survey listing their occupation&#39;s TaskNet tasks. Drawing on the WORKBank survey (<a href="https://arxiv.org/abs/2506.06576" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Shao et al., 2025</a>), respondents rated every task they do on several scales, including how much they would want an AI system to do the task for them if it could do it completely (not at all to entirely), how much they enjoy the task, setting salary aside; and how much human-AI collaboration the task would need if AI assisted (none needed to essential).
Workers could then pick the tasks they would most like AI to take over and the ones they most want to keep doing themselves, with a sentence on why. Finally, they answered three questions about the job as a whole: how much AI has changed how they do their job over the past year, what they would most likely do with the time freed up if AI took over the tasks they would like automated, and how they feel about AI’s effect on their occupation over the next five years.
<h2>What workers want to hand off and keep</h2>
<img src="https://cdn.sanity.io/images/h6s14f4z/production/9c604593c7e7ffd03e2bbc5c8f968c9ed2671ede-2048x1902.jpg" alt="" style="width: 100%; border-radius: 10px;" />
Across occupations, workers most wanted to keep tasks that are based in human interaction and tasks that are high-stakes. For software developers, that means mentoring junior developers or reviewing technical designs; for physicians, leading resuscitations; and for lawyers, interviewing clients and advising them on risks and options.
The tasks workers most wanted to hand to AI were often clerical in nature, like keeping medical records up to date. Elementary school teachers showed a particularly clear contrast: most want to hand data analysis and creating teaching materials to AI but keep teaching daily lessons and helping kids with emotional skills.

<img src="https://cdn.sanity.io/images/h6s14f4z/production/48d0796b07100d491d753ea37021193f1118c4ec-2048x950.jpg" alt="" style="width: 100%; border-radius: 10px;" />
We also asked workers for each task, assuming AI was going to help, how much human-AI collaboration would be needed. Of roughly 9,000 task ratings, 14% were &quot;none needed&quot; and 27% &quot;essential,&quot; with most somewhere in between.
&quot;I would like AI to listen to my patient encounter and show me a list of tests to approve based on the discussion. I do not want it to autonomously order the tests without my approval.&quot; - ER physician
Lawyers gave &quot;essential&quot; ratings most often, for 38% of their aggregate ratings, compared with only 19% for software developers.
<h2>What workers would do with the time saved by AI</h2>
Asked what they would do with time saved by AI, 38% of respondents said they&#39;d do more of the tasks they enjoy; 22%, the same work at higher volume; 18%, move into new kinds of work; and 15%, work fewer hours.

<img src="https://cdn.sanity.io/images/h6s14f4z/production/2a0b920d6cbffb2a8cf8a192c31b28ef4ef32fe9-2048x1057.jpg" alt="" style="width: 100%; border-radius: 10px;" />

Elementary school teachers were the most drawn to do more of the work they already enjoy (51%); as one of them put it, AI &quot;would allow me to be more present in my classroom and do a better job directly teaching my students.&quot; ER physicians were the most inclined to do the same work at higher volume (36%), and dishwashers to move into new kinds of work (43%).
<h2>Exposure scores track reported change</h2>
<img src="https://cdn.sanity.io/images/h6s14f4z/production/185de3ade15528527a0003d0942a5becaf8a3876-2047x1008.jpg" alt="" style="width: 100%; border-radius: 10px;" />
Exposure scores help us estimate how much of an occupation&#39;s required skills and tasks overlap with what AI could eventually do. In TaskNet give each task a score from 0 to 100 for how feasible it would be to build a training environment where AI can practice the task and learn from feedback, following the methodology set out in <a href="http://arxiv.org/abs/2605.02598" target="_blank" rel="noopener noreferrer" class="text-indigo-600">What Jobs Can AI Learn? Measuring Exposure by Reinforcement Learning</a>. The idea here is that understanding how easy it is to train AI to do a certain job should give us insight as to how much AI will affect that job in the short term.
We found that people in more exposed occupations were indeed much more likely to say AI has changed their job a lot in the past year, and more likely to worry about AI&#39;s effect on their occupation&#39;s future. ER physicians were a notable exception: they fell below the trend for both reported change and worry. This may be partly because the occupation includes many high-stakes tasks involving human interaction, the kinds of tasks workers most wanted to keep doing themselves. These trends hold when we weight each task&#39;s exposure score by its importance to the job or by the time workers spend on it and are robust to different exposure measures, such as that found in<a href="https://arxiv.org/abs/2303.10130" target="_blank" rel="noopener noreferrer" class="text-indigo-600"> Eloundou et al.</a>

<img src="https://cdn.sanity.io/images/h6s14f4z/production/2a50ca65785be45eb5fed1bab9f0d312189feee6-2048x1041.jpg" alt="" style="width: 100%; border-radius: 10px;" />

Looking across occupations, people who rated their jobs as having changed more were much more likely to be worried than people who rated their job as having not changed much at all. The share who were optimistic was relatively similar across groups, while the share who were neutral was smaller among those reporting more change. 
<h2>Extending the survey to Mercor applicants</h2>
In mid-October, we began offering everyone who completes a Mercor application an optional module on AI and the future of work. Responses came from over 2,000 workers across the world, spanning many more occupations.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ca4d9a84cf470b44df6fb4b87b07355f1b668eb9-2080x1365.png" alt="" style="width: 100%; border-radius: 10px;" />
The link between exposure scores and how much people say AI has affected their jobs holds across this broader set of occupations, with medicine again noticeably below where the trend line would suggest. We note that people on our platform are significantly more likely to be interested in and engaging with AI, so this sample is unlikely to be representative of workers in general.
<h1>Conclusion</h1>
The people who are helping to train frontier models have valuable perspectives on how those frontier models should be developed and used. Mercor has long focused on organizing intelligence, but there is more to intelligence than the ability to solve a certain predefined task. Our expert talent network allows us to ask people at scale about their perspective on AI, with the hopes of helping to answer some of these big questions. Through TaskNet and follow-up work like this one, we hope to better understand what tasks are and are not essential to the identity of an occupation, as well as what workers do and don’t want for the future of this technology.
<h2>Appendix: Exposure correlation under Eloundou-β</h2>
<img src="https://cdn.sanity.io/images/h6s14f4z/production/4d366079918eebec2dddd85faede90cd38ca4d5b-2048x1008.jpg" alt="" style="width: 100%; border-radius: 10px;" />

Aden Barton

Aurash Vatan

What workers want to hand to AI, and what they want to keep

Nicolai J. Foss

Henrik M. Nielsen

A new taxonomy of organizational difficulty, and the blind spot it reveals in how AI benchmarks measure real work.

Playing well with others: Putting AI in the org chart

<h2>Introduction</h2>
AI is rapidly changing our relationship with work, and people are concerned about what those changes mean to them. What will work of the future look like and what skills will be needed? How do we train people for jobs that do not yet exist? Answering these questions requires detailed and up-to-date knowledge of what people actually do at work, yet no taxonomy fully captures the complexity and diversity of tasks and jobs in today’s workforce.
Today, researchers from Stanford, Mercor, O*NET Resource Center, and the National Bureau of Economic Research released a working <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7670419" target="_blank" rel="noopener noreferrer" class="text-indigo-600">paper</a> describing TaskNet, a new open-source catalogue of tasks across six different occupations in the US economy, updated to reflect the impact of AI on professional work. TaskNet builds off the structure of <a href="https://www.onetcenter.org/overview.html" target="_blank" rel="noopener noreferrer" class="text-indigo-600">O*NET</a>, the US Department of Labor&#39;s widely used map of occupational tasks. O*NET provides standardized descriptions of the tasks, skills, and knowledge associated with different occupations, helping both researchers and workers better understand occupations. TaskNet extends O*NET with a fast, scalable interviewer for identifying new tasks that will allow it to keep up with the rapid pace of change. TaskNet focuses on speaking directly to workers in order to better understand a profession, while also eliciting worker judgment on task features relevant to automation like how hard is it to judge the quality of their output or how easy it is to define success a priori.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/cd40258bb201e8485dffc619e1fd1c83a34de070-4000x2916.png" alt="" style="width: 100%; border-radius: 10px;" />
The result is a more comprehensive and accurate catalogue of modern work. In our initial sample of 6 occupations, TaskNet identifies 79 additional tasks that were validated by O*NET analysts but that had no O*NET counterpart. This includes 5 tasks that were newly created or reshaped as a direct result of AI adoption.
<h3>Using an AI Interviewer at Scale</h3>
TaskNet v1 uses a scalable method of capturing expert insight: AI interviews. Traditional occupational taxonomies like O*NET relies primarily on surveys to elicit task information. While surveys allow for a high degree of standardisation, it can be difficult to identify emerging tasks in a static questionnaire. Interviews, on the other hand, are well suited for eliciting and understanding new tasks. They can be more exploratory in nature and adaptive to make sure that new tasks or information are fully articulated. However, interviews conducted by humans are expensive and hard to scale. 
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ea175705138c628adb9a1ab66f0bab1bf814d502-4000x1406.png" alt="" style="width: 100%; border-radius: 10px;" />
TaskNet solves these problems by using a language model as an interviewer. Mercor has long been a pioneer in deploying adaptive AI interviews at scale, having completed over 3 million interviews and counting since 2023. The Stanford research team, on the other hand, has helped pioneer adaptive interviewing techniques with projects like SparkMe. TaskNet v1, an academic research collaboration, brings researchers together to create a mixed-initiative system that interviews workers about their tasks, asks follow-up questions to clarify ambiguous descriptions, and works with them to confirm their task bundles.
To validate the resulting task lists, O*NET analysts reviewed all 229 candidate statements and found that they all fit the O*NET center’s standards for a valid task. Analysts did flag that some tasks departed from O*NET ’s style guidelines or described work primarily performed by another occupation, leading to a rewrite of 21% of tasks.
We also took a holdout set of 1,184 task reports from 57 workers whose interviews were not used to build the TaskNet dataset. Our lists covered 89.3% of reported tasks on average across the six occupations, compared with 78.6% for O*NET itself. For a full technical explanation of TaskNet and its validation procedure, see the preprint.
<h1>Results</h1>
TaskNet increased task coverage for O*NET occupations and identified new occupational tasks not covered by O*NET. When tested on a held-out set of validators, our method produced a task list that workers rated as more representative than O*NET. The 79 new work tasks represent billions of hours of work a year, or an estimated $140B a year in time-adjusted wages.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/409cd3d8cec23bb9c391a87bd984e0f474b1dc88-4000x2628.png" alt="" style="width: 100%; border-radius: 10px;" />
Many of these tasks missing from O*NET fall into the bucket of work that is done, but not often accounted for - elementary school teachers administering basic first aid, for example, or ER physicians calling patients with test results.
“If you understand a doorman simply as a way to open the door, of course you can yield cost-savings by replacing him with an automated system. But if you understand a doorman as a way to elevate the status of the hotel &amp; create a better guest experience (hail taxis, greet guests, carry bags...), then you realize the full role of the doorman cannot be replaced with an automated system.”
— Rory Sutherland
Behavioral economist Rory Sutherland calls this the doorman fallacy - when we assume a job is made up only of its most visible tasks. By interviewing workers directly and at scale, TaskNet helps bring these essential tasks into an official taxonomy.
We also discovered five new tasks that did not exist before the proliferation of modern LLMs. Most of these examples come from software engineering like designing prompts and context pipelines, or writing evals for different AI systems. However, we also observed new tasks in a non-technical field; lawyers reported that they now spend time reviewing and revising AI-generated legal content for accuracy, compliance, and supporting authority before filing or circulation. Expanding the TaskNet interviewer to more occupations and repeating interviews over time will allow us to discover even more emerging tasks, as well as observe which tasks have a shrinking timeshare.
<h3>Exposure Scores</h3>
We found that utilizing this updated taxonomy increased AI exposure scores for software developers, lawyers, and elementary school teachers. Much of this increase was due to the addition of the aforementioned tasks that don&#39;t immediately come to mind for these professions but still constitute millions of hours of work each year. For lawyers, software developers, teachers and physicians these tasks were mainly clerical work around documenting, communicating, and keeping records.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/5c97d9c756de063c63803e41901345ba51aee1ec-4000x2904.png" alt="" style="width: 100%; border-radius: 10px;" />
For personal care aides, we saw a decrease in exposure, mostly due to the addition and refinement of more tasks that were physical or relationship based like helping clients dress, or caring for a client’s pet.
We also ranked tasks by relative expertise, how much better the best person is at this task than the average person. For software developers, the most AI exposed tasks were the ones with the lowest returns to additional expertise. Among the tasks that are easiest for AI to learn, only 34% of developers say top performers clearly outperform the average developer. For the more difficult to learn tasks, 62% say top performers clearly outperform the average developer. Lawyers show a similar pattern: the tasks where expertise matters most, like negotiating settlements, arguing at trial, and advising clients, score lowest on RL feasibility.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/95941851aa3cc9fdedc2ba34326cc997b4b5924d-4000x2584.png" alt="" style="width: 100%; border-radius: 10px;" />
These results show how much our estimates of AI exposure and our understanding of what tasks are durable depend on the work that is included in our taxonomies. Using TaskNet we were able to identify both longstanding responsibilities missing from O*NET and newly emerging tasks associated with AI adoption. We hope to extend TaskNet to more occupations and repeat the interviews over time. Broadly, LLMs offer a way to collect detailed accounts of work and opinions at an unprecedented scale. We believe that a combination of this new technology, along with careful thought from industry, academia, and policymakers, will help us build the foundations to better understand work today, how it&#39;s changing, and what it will look like in the future.
Thank you to Andreas Haupt, Erik Brynjolfsson, Diyi Yang, Jenn Wang, and Phil Lewis for collaborating on this work.

TaskNet, a new open-source catalogue of tasks across six U.S. occupations that captures how AI is impacting professional work.

Introducing TaskNet: A New Taxonomy of Work

We post-trained NVIDIA’s new <a href="https://blogs.nvidia.com/blog/nemotron-3-5-super-vl/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Nemotron 3.5 Super VL</a> ahead of its release. We wanted to see how far a small, targeted dataset moves an open model on real professional work.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/828e38b35fc1ec2cc291b55337427eabd0950749-2048x1103.png" alt="" style="width: 100%; border-radius: 10px;" />
We picked three domains where our customers need agents that get the details right, and brought in a partner for each: Harvey for legal, Cognition for software engineering and Ramp for accounting. Each domain got its own training set of a few hundred tasks from Mercor&#39;s APEX environments, and each post-trained model was tested on benchmark tasks it had never seen. In all three, it beat the base model:
<ul><li>Legal: On the APEX-Agents benchmark legal domain, the post-trained Nemotron 3.5 Super’s mean rubric score rose from 45.1% to 51.8% (+6.7 points), while its full-credit rate (Pass@1, the share of attempts that met every rubric criterion) rose from 19.8% to 25.0% (+5.3 points).</li><li>Software engineering: On the APEX-SWE benchmark, the post-trained Nemotron 3.5 Super&#39;s full-credit rate (Pass@1, every test passing) rose from 26.6% to 34.7% (+8.2 points).</li><li>Accounting: On the APEX-Accounting benchmark, the post-trained Nemotron 3.5 Super’s mean rubric score rose from 34.6% to 41.9% (+7.3 points), while the share of tasks it scored zero on fell from 19% to 13% (−6 points).</li></ul>
With only a few hundred training tasks per domain, that’s a meaningful gain on work the model had never seen, and it makes Nemotron 3.5 Super a practical starting point for building domain-specific agents.
<h1>Where we trained and tested</h1>
We trained and tested the models in Mercor&#39;s APEX environments (<a href="https://www.mercor.com/blog/introducing-apex-agents/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a>, <a href="https://www.mercor.com/blog/introducing-apex-swe/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-SWE</a> and <a href="https://www.mercor.com/blog/introducing-the-ai-productivity-index-for-accounting/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Accounting</a>), which recreate the work professionals do every day, from a law firm&#39;s case files to a company&#39;s books at month-end close to multi-service production software. Experienced practitioners built every task, and each one is graded against expert-defined pass/fail criteria or automated tests.
We tested each model on held-out benchmark tasks and compared it with the base model in the same evaluation run.
<h1>Legal, with Harvey</h1>
Harvey builds AI for legal professionals, and its team partners with Mercor to evaluate legal agents. Legal work requires knowing specific facts and context about a case, in addition to general legal knowledge. We trained the model with a targeted dataset to see if it improved the model’s ability to find and use legal information before providing an answer.
We post-trained the model with supervised fine-tuning on model-generated, grader-selected trajectories from Mercor&#39;s proprietary APEX legal tasks, then tested it on the legal slice of the APEX-Agents benchmark, with no case files shared between the two.
On the legal slice of the APEX-Agents benchmark, the base model met 45.1% of rubric criteria on average and met every criterion on 19.8% of attempts. After post-training, it met 51.8% (+6.7 pp) and met every criterion on 25.0% of attempts (+5.3 pp).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/925eeae20ca22279725fc213307390e73c6e018b-1980x1188.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 1. Mean share of rubric criteria met and full-credit rate (Pass@1), base and post-trained Nemotron 3.5 Super. Source: Mercor evaluation of Nemotron 3.5 Super on the APEX-Agents benchmark, legal tasks.
The improvement is spread across the benchmark. The share of tasks whose average score across attempts was at least 80% of the rubric rose from 15.0% to 21.2% (+6.2 pp), and the share averaging below 20% fell from 25.0% to 18.1% (−6.9 pp).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/a9cb623f236a0790be7eae502db8a6f3be03a5b6-1980x1188.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 2. Share of legal tasks whose average rubric score across attempts is at least a given level, base and post-trained models. Source: Mercor evaluation of Nemotron 3.5 Super on the APEX-Agents benchmark, legal tasks.
Scores improved across every type of legal work in the benchmark. The gain was largest on motion drafting (+23.2 pp) and litigation strategy (+10.3 pp), and legal research, the most common task type, improved by 7.0 pp.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f0a26c08b4fbc7cb59d9007f99c370fde1246ff9-1980x1364.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 3. Gain in mean rubric score by task type, base and post-trained Nemotron 3.5 Super. Source: Mercor evaluation of Nemotron 3.5 Super on the APEX-Agents benchmark, legal tasks.
The post-trained model reads the case file in more detail before it answers.
<table class="table-cell-border" style="width: 100%; border-collapse: collapse; margin: 1em 0;"><thead style="text-align: left;"><tr><th>Behavior</th><th>Base</th><th>Post-trained</th><th>Change</th></tr></thead><tbody style="text-align: left;"><tr><td>Answers without using any tool</td><td>10.6%</td><td>2.6%</td><td>−8.0 pp</td></tr><tr><td>Document reads per attempt</td><td>5.6</td><td>11.3</td><td>2x</td></tr><tr><td>Checks email, chat or calendar</td><td>5.3%</td><td>16.4%</td><td>+11.1 pp</td></tr></tbody></table>
Example: In one task, counsel needs the minimum settlement payment after an oil spill, which depends on a cap and a payment floor defined in a settlement agreement. The base model almost never opened the agreement and subtracted the numbers in the prompt. The post-trained model almost always found it and applied its terms.
Base model - one attempt, scored 0%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Tool calls:&#10;1. code_exec: arithmetic on the prompt&#39;s numbers</code></pre>
Post-trained model - one attempt, scored 100%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Tool calls:&#10;1. code_exec: total the owner&#39;s fees&#10;2. search_files(&quot;*Settlement*&quot;)&#10;3. word_read_document_content: Settlement Agreement and Mutual Release&#10;4. word_get_document_overview: Executive Settlement Memo</code></pre>
<h1>Software engineering, with Cognition</h1>
Mercor built APEX-SWE in partnership with Cognition, the team behind Devin. Cognition reviewed a sample of the benchmark tasks to ensure they accurately represent how real production systems fail and get fixed. Autonomous coding agents like Devin must understand a system, identify problems, make fixes, and confirm everything works as intended. We wanted to see if using a targeted dataset could improve model performance across this entire workflow.
We post-trained the model with supervised fine-tuning on model-generated, grader-selected trajectories from site reliability, security, database and CRM tasks, then tested it on benchmark tasks set in different environments, built on services the model never saw in training.
On APEX-SWE, the base model solved 26.6% of tasks on a given attempt; the post-trained model solved 34.7% (+8.2 pp). It also became more consistent: the share of tasks it solved on every attempt doubled, from 9.9% to 19.9% (+10.0 pp).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/56411be5bc727f5df3bea5c0ccc20678a2a704ee-1980x1188.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 4. Full-credit rate (Pass@1) and share of tasks solved on all four attempts, base and post-trained Nemotron 3.5 Super. Source: Mercor evaluation of Nemotron 3.5 Super on the APEX-SWE benchmark.
Most of the gain came from integration tasks, where the model has to build working systems across several live services.
<table class="table-cell-border" style="width: 100%; border-collapse: collapse; margin: 1em 0;"><thead style="text-align: left;"><tr><th>Behavior</th><th>Base</th><th>Post-trained</th><th>Change</th></tr></thead><tbody style="text-align: left;"><tr><td>Reads the task&#39;s API docs before working</td><td>46.0%</td><td>64.1%</td><td>+18.1 pp</td></tr><tr><td>Checks the live system after its last edit (calls per task)</td><td>0.98</td><td>2.01</td><td>2x</td></tr></tbody></table>
Example: In one task, the agent had to build a subscription-renewal job across a CRM, email, S3, chat and an issue tracker. The base model wrote the script in one pass, put the issue text in a field the tracker&#39;s API ignores, and declared success. The post-trained model read each service&#39;s API docs first, used the right field and checked the issues it created. The base model never solved it; the post-trained model solved it in most attempts.
Base model - one attempt, scored 0%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Key steps:&#10;1. cat &gt; /app/lifecycle.py: writes the whole script in one pass&#10;2. curl …/issues/: lists the issues it created&#10;3. Concludes the format is correct</code></pre>
Post-trained model - one attempt, scored 100%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Key steps:&#10;1. cat plane_api_doc.md: reads the issue tracker&#39;s API docs&#10;2. Writes the script using description_html, the field the API stores&#10;3. sed -i &#39;s|/work-items/|/issues/|g&#39;: fixes the endpoint after an error&#10;4. requests.get(...): fetches the created issues and checks their content</code></pre>
<h1>Accounting, with Ramp</h1>
Mercor and Ramp built APEX-Accounting to test whether AI can do accounting work – not just answer accounting questions. For an accounting agent, the hard part is the detail of one company&#39;s books: reconciling records that disagree, applying the company&#39;s own policies and carrying each figure through to the final entry.
We post-trained the model with reinforcement learning on Mercor’s APEX-Accounting tasks, so it learned from its own best-scoring attempts. We then tested it on the APEX-Accounting benchmark, whose simulated companies share no data with the training set.
On the APEX-Accounting benchmark, the base model’s mean rubric score was 34.6%. After post-training, it rose to 41.9% (+7.3 pp), and the share of tasks where it scored zero fell from 19% to 13% (−6 pp).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/becb1554bc4c626a890b504d46b3815611bb358f-1980x1188.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 5. Mean rubric score and share of tasks scoring zero, base and post-trained Nemotron 3.5 Super. Source: Mercor evaluation of Nemotron 3.5 Super on the APEX-Accounting benchmark.
After post-training, the model took stock of available files more often: 98% of cases, up from 60%. It opened a median of 12 files per task, up from 8.5, and used code to analyze data more often.
<table class="table-cell-border" style="width: 100%; border-collapse: collapse; margin: 1em 0;"><thead style="text-align: left;"><tr><th>Behavior</th><th>Base</th><th>Post-trained</th><th>Change</th></tr></thead><tbody style="text-align: left;"><tr><td>Starts by taking stock of the available files</td><td>60%</td><td>98%</td><td>+38 pp</td></tr><tr><td>Files opened per task (median)</td><td>8.5</td><td>12</td><td>1.4x</td></tr><tr><td>Runs code to analyze the data</td><td>65%</td><td>87%</td><td>+22 pp</td></tr></tbody></table>
Example: In one task, the agent had to review a company&#39;s December journal entries for posting errors. The base model loaded the full year’s journal and exceeded its context limit. The post-trained model narrowed the retrieval to December and recovered from an initial filtering error. It scored 45% on the task rubric, compared with 0% for the base model – progress, but not a fully correct result.
Base model - one attempt, scored 0%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Tool calls:&#10;1. excel_v2_read_tab: reads the full journal tab (450,780 characters), which overflows the context</code></pre>
Post-trained model - one attempt, scored 45%
<pre class="my-6 overflow-x-auto rounded-lg border border-gray-200 bg-gray-100 p-4 text-gray-900"><code class="font-mono text-sm leading-6">Tool calls:&#10;1. excel_v2_filter_tab: filters for December on column &quot;Date&quot;; error, column not found&#10;2. excel_v2_filter_tab: retries on column C, starting below the title row; 300 of 2,639 rows match</code></pre>
<h1>What this shows</h1>
Across legal work, software engineering and accounting, a few hundred targeted tasks were enough to improve Nemotron 3.5 Super on held-out professional work. We saw gains with both supervised fine-tuning and reinforcement learning.
The post-trained models also changed how they worked. In legal, the post-trained model read the case files more often. In software engineering, it read API documentation and checked the live system after making changes. In accounting, it looked through all the files before starting and opened more of the source records. For teams building on open models, it’s evident that with realistic environments and a small, well-built dataset, Nemotron 3.5 Super can be shaped for the work their users actually do.

Harvey

Cognition

Ramp

Small, targeted data, measurable gains: specializing NVIDIA Nemotron 3.5 Super for legal, software and accounting work

Every job in the economy is being reshaped by advances in AI. We’ve seen this with our own APEX benchmarks, where models progressed rapidly in the past year on real-world work.
As they get better, the questions get harder.
How do you measure a model against a job that AI itself is changing? What data actually helps when a model fails at real work? How can human expertise best push the model capability frontier forward?
This is the Mercor Research agenda.
Mercor works with domain experts across thousands of different kinds of work. We use their expertise to build evaluations, understand where models fail, and create data to improve them. Our research team works across that process, from evals through post-training.
<h2>Measuring AI on real work</h2>
The global economy contains thousands of occupations, each with its own workflows and standards. Very little of that work has a rigorous way to measure what AI can and cannot do.
Our suite of APEX benchmarks cover management consulting, investment banking, corporate law, accounting, and software engineering. We are continuing to expand APEX to understand how AI is changing all types of work throughout the economy.
But the work we are measuring is changing too. A task that matters today may matter less in a year. People may spend less time producing work and more time reviewing it. Different failure modes matter as models move from answering questions to taking actions.
That makes task realism and safety increasingly important. We will measure models in the environments where the work actually happens, including whether they can complete the work reliably without introducing new risks such as exploiting flawed incentives, or concealing mistakes.
<h2>Producing better training data</h2>
Much of the knowledge models need is spread across where people actually do their work - documents, software tools, email, and the judgment people use while working across it all.
Organizing that knowledge at a useful scale requires more than having experts produce more data. Experts’ time is expensive and limited. One problem we are working on is how to get more useful training signals from each hour they work with us.
An expert might create the task, define the standard for a good deliverable, or act as judge in the handful of cases where models and automated graders disagree.
We also want to make data quality measurable. That means testing agreement between evaluators, validating against held-out examples, and measuring whether a dataset improves model performance.
<h2>Closing the loop with post-training</h2>
Once we can identify the capabilities it is missing, we can use that signal to train it.
We recently <a href="https://www.mercor.com/blog/training-frontier-knowledge-work-agents-a-397b-rl-training-guide-with-skyrl/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">published</a> a post-training guide showing how we increased Qwen3.5-397B’s Pass@1 on <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a> from 16% to 27%, including the training script, model weights, and evaluation traces.
APEX shows us where a model struggles. Experts help us understand why and produce training data around those weaknesses. We post-train the model and measure it again. The information tells us what worked and what remains unsolved.
We want to understand when specialized models outperform general ones, which capabilities respond to targeted post-training, and how the amount and type of expert data affect the result.
<h2>Mercor Research team</h2>
These problems span AI evaluation, model training, economics, and labor. We are building a team to work on each part.
Our researchers include:
<ul><li>Edward J. Hu: first author of <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener noreferrer" class="text-indigo-600">LoRA</a>, formerly at OpenAI</li><li>Minyang Tian: first author of <a href="https://arxiv.org/abs/2407.13168" target="_blank" rel="noopener noreferrer" class="text-indigo-600">SciCode</a> and <a href="https://arxiv.org/abs/2509.26574" target="_blank" rel="noopener noreferrer" class="text-indigo-600">CritPt</a></li><li>Victor Barres: first author of <a href="https://arxiv.org/abs/2506.07982" target="_blank" rel="noopener noreferrer" class="text-indigo-600">tau^2-bench</a> and <a href="https://arxiv.org/abs/2603.13686" target="_blank" rel="noopener noreferrer" class="text-indigo-600">tau-voice</a>, formerly at Sierra</li><li>Hardik Bhatnagar: co-lead of <a href="https://arxiv.org/abs/2603.08640" target="_blank" rel="noopener noreferrer" class="text-indigo-600">PostTrainBench</a></li><li>Bertie Vidgen: first author of <a href="https://arxiv.org/abs/2601.14242" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a></li><li>Along with researchers from Anthropic, Ai2, Bridgewater, DE Shaw</li></ul>
There is still a lot we do not know about how to measure that gap, what data closes it, and how quickly models can learn from the expertise embedded in the economy. Mercor Research is being built to explore these questions at the frontier.

Meet the Mercor Research team. We measure AI on real work, build expert data around where models fail, and post-train to close the gap.

Why Mercor is building a research team

Benchmark scores have soared over the last few years, but AI adoption seems to be contributing only slightly, if at all, to higher productivity growth in U.S. economic statistics. It&#39;s a trillion-dollar paradox.
That tension is hard to resolve without understanding how benchmarks relate to real work, and how humans themselves would do that work. To help bridge that gap, we hired 12 junior accountants to complete simplified versions of tasks from our recent <a href="https://www.mercor.com/blog/introducing-the-ai-productivity-index-for-accounting/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Accounting</a> benchmark.
We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/4154d9753c98533da19506b9954b9a8dc650440b-2200x1062.png" alt="Dot plot comparing accountants without AI and Claude Opus 5 on four accounting tasks. Accountant accuracy ranged from 0% to about 90%, and most attempts took 30 to 180 minutes. Claude scored 100% on all 20 attempts and finished each one in under 10 minutes." style="width: 100%; border-radius: 10px;" />
More striking is how fast AI took the lead. Just eighteen months ago, the best AI models fell short of the average accountant’s ~37% score. Today, models ace those same tasks.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/0ec454bcb2fba47e59b51bd9bd2ebcd2799dee58-2320x1660.png" alt="Scatter plot of AI model accuracy on the study tasks by release date, from mid-2024 to late 2026. A horizontal line shows the average accountant score of 37%. GPT-4o scored near 0%. OpenAI o3 passed the accountant average in spring 2025, and GPT-5 reached about 69%. Opus 5 and other recent frontier models now score at or near 100%. Some budget and open-weight models, such as Qwen3.5-122B, still score below the accountant line." style="width: 100%; border-radius: 10px;" />
These results do not mean accountants are replaceable, since our tasks ended up testing what AI is best at – e.g., detail-oriented instruction following – rather than the whole job of an accountant, which involves less structured tasks like client communication.
But these data do suggest substantial changes, and productivity gains, in accounting in the coming years. That would be true even if model progress froze today, and even more so if the trend depicted above continued.
These results are provocative. So much so that we considered not publishing them for fear of misinterpretation. But we think transparency about the findings matters as people and institutions prepare for rapidly advancing AI.
Below is a brief study summary and the full paper can be found <a href="https://cdn.sanity.io/files/h6s14f4z/production/4048fb3eece18a56006026d212d25312f7ffc29a.pdf" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a>.
<h2>What we did</h2>
Participants averaged about five and a half years of accounting experience, and all were licensed CPAs. The accountants completed four realistic month-end close scenarios, each requiring them to dig through a company&#39;s working files to find the right numbers, do the math, and deliver a table of results.
We designed the study to measure AI augmentation, but because models alone scored perfectly, there was no room to measure uplift. We focus instead on unassisted humans vs. AI.
<h2>What we found</h2>
As shown above, participants performed substantially worse than frontier models. Models are also less expensive: comparing cost per task criterion, frontier models are more than an order of magnitude cheaper than humans.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/46639e0832dc06d155e05c557a74b3d5f63a3fda-2320x988.png" alt="Bar chart on a log scale. Claude Opus 5 cost $0.21 per rubric criterion met. Accountants without AI cost $10.35, which is 49 times more, based on the US median accountant wage." style="width: 100%; border-radius: 10px;" />
<h2>What do these results mean</h2>
These results don’t imply that accountants are fully replaceable, but they are informative about the future of work.
For starters, our tasks tested only a set of accounting skills that happen to be the same skills models are best at: being detail-oriented, searching through files, and following instructions very closely.
Because the tasks were originally designed to stump models, they evolved to include hard-to-spot (but realistic) accounting requirements that compounded on one another, so a single overlooked number could produce a very low score.
Being able to navigate such a complicated file set is an important skill in accounting (and knowledge work more generally), but it is not the whole job. Given this difficulty, our task authors expected average scores to be quite low: 30% for a junior and 55% for a mid-level accountant.
Further, the setting stripped out much of the job: no coworkers to ask for help, no accumulated job context. That made the tasks harder. It also meant we did not measure the skills likely to matter more as models take on the structured, detail-oriented parts of accounting: client communication, asking the right questions, building tacit context.
As one task author put it, &quot;the traps [hidden in the task] are realistic, and the conditions and the scoring are what drove the results.&quot;
<h2>What this teaches us about benchmarking</h2>
Benchmarks aim to mirror real work, but tasks get made harder to induce model failures, like exam questions built to challenge top students. As models improve, some tasks now take many experts working together for tens of hours to build, so one person couldn&#39;t complete them without similar time and help. The accounting tasks used here are a case in point.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/bd29f5bf06606a00c6f4a6a0df0b364243e2c161-3800x1200.png" alt="Two Venn diagrams of &quot;tasks humans can perform&quot; and &quot;tasks AI can perform.&quot; In the traditional benchmark, all of the tasks sit where the two circles overlap. Over time, the benchmark tasks move toward the AI side, so many of them now fall outside what humans can do." style="width: 100%; border-radius: 10px;" />
This evolution towards work that only AI can fully do makes sense, since that is likely some of the most economically valuable output. We don’t ask humans to run 60 miles an hour, yet transporting people at that speed is exactly how cars boost productivity.
That shifts the goal of frontier benchmarking from measuring how well AI does existing human work to measuring tasks that were formerly impossible or prohibitively costly. Identifying those new tasks, of course, is its own challenge.
<a href="https://cdn.sanity.io/files/h6s14f4z/production/4048fb3eece18a56006026d212d25312f7ffc29a.pdf" target="_blank" rel="noopener noreferrer" data-download="button">Download full paper</a>
Interested in working on similar problems or have feedback for us? Get in touch with us <a href="https://www.mercor.com/partner/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a>. You can also follow the author on <a href="https://x.com/aden_barton" target="_blank" rel="noopener noreferrer" class="text-indigo-600">X</a>.
We thank Whitney Zhang for her help in designing the study and for giving feedback on the write-up drafts.
Additionally, we thank the accounting experts who took part in the study as well as those who helped run and create the tasks and study infrastructure.

We tested 12 licensed CPAs against frontier AI on APEX-Accounting tasks. AI was faster, more accurate, and 10x cheaper. Here's what we found.

AI vs. Accountants: Frontier Models Outscore CPAs | Mercor

Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants

This post is stored in Sanity and rendered as Portable Text in the web app. Edit it in the Studio and the change flows through to the frontend.

Welcome to Mercor Blogs

APEX-Agents tests a model’s ability to conduct real-world work across Investment Banking, Management Consulting, and Corporate Law. Models need to reason over task requirements scattered across files and communications, mirroring the environments faced by working professionals.
As AI models have advanced, so have their methods for solving APEX-Agents tasks. We’ve updated our benchmark with task specifications, tooling, and environments to ensure leaderboard accuracy. Most notably, APEX-Agents 1.1 no longer rewards noncommittal answers. This ensures that our tasks and grading align with how real-world professionals operate.
The Mercor Research team is confident that this revised benchmark better measures how models can conduct economically valuable work. Claude Fable 5.1 remains on top of the leaderboard with 68.6% on Pass@1, though rankings have shifted throughout the rest of the board.
If you’d like to run APEX-Agents 1.1, you can find our dataset on <a href="https://huggingface.co/datasets/mercor/apex-agents-v1.1" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> and <a href="https://hub.harborframework.com/datasets/mercor/apex-agents-1-1/latest" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Harbor Hub</a>. We’ve also released our evaluation agent on <a href="https://github.com/Mercor-Intelligence/apex_loop_truncated_tools_agent" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GitHub</a>.
<h2>Scattergunning</h2>
In early July, we discovered that models were increasing benchmark scores by aggressively hedging their answers. We call it scattergunning. During our updates, we removed scores from the APEX-Agents leaderboard, or did not publish them, for models that engaged in this behavior.
We define scattergunning as “a behavior where models provide multiple answers despite having a single answer that is well-specified by the environment.”
<img src="https://cdn.sanity.io/images/h6s14f4z/production/32ca8804caa44aeb5d20b554b1066b9fa724414f-6864x4722.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 1. A sample scattergunning response on APEX-Agents task “World221_TR_10.”
Since APEX-Agents grades using binary rubric items that only check for inclusion of correct information, models that scattergun generally gain an advantage. Only one answer needs to be right, so hedging responses increases the chance of passing a given criterion.
In some cases, noncommittal behavior is arguably correct. When information is not specified, models scattergun answers to acknowledge gaps. In professional work, this is often suboptimal. For example, a banker doesn’t usually submit an LBO model with separate cases for calculations with the median or mean: they pick one option for their numbers.
On APEX-Agents 1.1, there is an unambiguous answer that models can and should give. That information is often hard to find, but it is present in conventions or communications. 
Overall, we decreased the influence of scattergunning on leaderboard scores through three changes:
<ol><li>Three task audits with Mercor domain experts to ensure that our problems are complex and realistic, but well-specified enough to be solvable.</li><li>A new judge model that scores scattergunned rubric items as a zero.</li><li>Explicit instructions in the system prompt not to engage in scattergunning.</li></ol>
Collectively, these changes make the benchmark better at measuring a model’s ability to conduct economically valuable work.
<h2>Task updates</h2>
We’ve selected 80 tasks per domain as our refined benchmark. These tasks were chosen and refined through three audits with Mercor domain experts. Our first two audits focused on ensuring a single correct and well-specified answer exists. The last audit verified the fairness of information placed in the world files to clarify decisions.
Since the original APEX-Agents release, we’ve noticed areas where tasks could have been more specific. We’ve updated the world files to contain all information needed from the failure modes that we’ve discovered. One answer is always correct, but the clarifying information is not entirely in the system prompt. Working professionals typically don’t specify parameters ad nauseam, so, to preserve realism, neither do our tasks.
Instead, preferences are listed in communications, spreadsheets, and documentation. We wanted models to discover this information, much like a professional would search past documents to follow the analytical conventions used by their organization. Models were told in the system prompt that contextual information could be found in world files, including communications.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/c524a8f0724005979d3e5a6c300bca3704c72570-2400x1566.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 2. Sample task hardening used for APEX-Agents 1.1 via the Mail app.
<h2>Judge model changes</h2>
We’ve also changed our judge model to grade correctly in the presence of scattergunning. DeepSeek-v4-Flash-0731 (temperature = 0.1) provided the strongest baseline accuracy. We then conducted Genetic-Pareto (GEPA) optimization to refine the system prompt. The result was a judge that could accurately score scattergunned rubric items, but not full answers, as a zero.
To conduct the optimization, the Mercor Research team hand-labeled a set of 1,407 rubric items from 337 trajectories across APEX-Agents tasks. On a held-out set of 354 of the rubric items, we observed significant increases in accuracy and detection. This increased the false negative rate from 5.3% to 8.0% compared to the naïve grader, but we believe the massively improved detection rate for scattergunning and better accuracy than the human average for a panel of three independent graders justifies this change.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7994735e3b46e74e0377ceeabf921858df56f522-5148x3762.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 3. Scattergun detection rate for the naïve judge against the scattergun-aware judge as well as the overall average accuracy of independent human evaluators vs. scattergun-aware judge.
Grader variance comes from scattergunning being difficult to detect. The difference between a parenthetical note and an answer hedge is often quite subtle. To ensure model interpretations of scattergunning did not affect evaluation, we explicitly defined the term in the system prompt and informed models that scattergunned answers will receive a zero.
<h2>Reliability fixes</h2>
We also made performance and reliability improvements across our MCP tools. This includes near-complete dependency support for all observed packages invoked by the code execution tool.
We scanned 26,302 trajectories to determine an exhaustive list of all dependencies commonly used by the models. Any dependency that appeared at least 25 total times across the trajectories was included in our environment.
<h2>Performance</h2>
Our move towards better task specifications has resulted in higher scores in aggregate, but it has also created a much more discriminating and fair benchmark. In particular, we observe significantly more headroom between Pass@4 and Pass^4 scores than in APEX-Agents. This gap shows the percentage of tasks that a model solved at least once but could not solve consistently across four runs. In real-world professional work, the latter score is vital.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/423f37887fa584bfc9e87643256f10d12f0bc561-3432x2102.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 4. Pass@4 to Pass^4 gap comparison, APEX-Agents vs. APEX-Agents 1.1.
In some tasks, models would search for information and successfully detangle messy communications in one run, but not even look for parameters in another. Among top models, GPT-6 Astra was the most consistent. It had the highest Pass^4 score of any model at 56.3%, but had a lower Pass@4 score than other models with comparable Pass@1 scores.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/29631f5e355ff9a6766362afe941223524cbe16b-5988x5862.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 5. Updated APEX-Agents 1.1 leaderboard ranked by Pass@1. See the full <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">leaderboard</a>.
Run score variance was affected by three main factors: propensity to seek clarifying information, ability to infer parameters from files, and scattergunning penalties. Gemini models searched communications most often, with Gemini 3.8 Flash searching on 93.1% of runs. Claude models searched slightly less, but were generally more effective at converting the information to pass scores with a 24.3% task-level uplift for Claude Fable 5.1 on Pass@1.
Interestingly, Kimi K3 and GPT-5.6 Sol searched communications on only about 15% of runs; but when they did search, they saw significant score uplift at the task-level. In fact, Kimi K3 received the largest task-level uplift of 29.7% on Pass@1. We believe this failure to search under ambiguity, despite prompting, lowered these models’ scores.
Models also saw score reductions for scattergunning. Compared to the naïve grader, some older models saw penalties over 20% on Pass@1 despite being explicitly warned by the system prompt. Newer frontier models generally received no scattergunning penalty with the new tasks, but some received minor deductions. This occasionally caused newer models to underperform their non-scattergun older counterparts.
<h2>Evaluations for the frontier</h2>
APEX-Agents 1.1’s rank-ordering lines up with other benchmarks; however, there are a few deviations from consensus metrics. We view this as strictly positive. If every benchmark told the same story, the strengths and weaknesses of models would not be surfaced.
It’s exciting when new models top benchmark leaderboards. We see that high score and want to instinctively declare a winner. This ignores the purpose of a benchmark: to show a snapshot of model capabilities. We’re open sourcing our evaluation suite to show the context behind the numbers, so users can see what tasks models failed on and why.
If we want our benchmarks to accurately measure model capabilities, then they must evolve as we push the frontier of AI. As the author of Terminal-Bench <a href="https://x.com/ryan_marten/status/2080321791361527843" target="_blank" rel="noopener noreferrer" class="text-indigo-600">writes</a>, “Benchmarks are not static artifacts. They are software and we should maintain them like software.” We couldn’t agree more and we are committed to upholding this standard.
We will continue to update and audit APEX benchmarks to ensure their reliability. We look forward to sharing learnings with the broader research community and to evaluating future models as we push the frontier of artificial intelligence.
See the full <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents 1.1 leaderboard</a> for updated scores.
<h2>Acknowledgements</h2>
Mercor would like to thank all of the contributors to this project, including our Mercor domain experts for their guidance during task audits and independent evaluation of our grader model.

Arnav Garg

Technical Staff

Akul Datta

APEX-Agents 1.1 removes the benefits of scattergunning. Claude Fable 5.1 leads at 68.6% Pass@1, but rankings across the leaderboard have shifted.

Introducing APEX-Agents 1.1

SkyRL

A step-by-step account of how we post-trained Qwen3.5-397B: six steps, three of them de-risking, five knobs we ablated, and the full training script.

Training frontier knowledge work agents: A 397B RL training guide with SkyRL

Most existing coding benchmarks do not test the full scope of work required to ship a product; they give models an existing codebase and ask for a localized change, rather than requiring them to infer and implement the system’s behavior from scratch. We wanted to extend this scope: what happens when a model has to build an entire software product itself?
SWE-Marathon-Ext builds on SWE-Marathon, a benchmark with 20 long-horizon software engineering tasks spanning library reproduction, full-stack application cloning, and algorithmic optimization. SWE-Marathon includes five product-clone tasks, but the paper explicitly defers analysis of those product clones to follow-up work.
SWE-Marathon-Ext adds to this unfinished part of the benchmark with a dedicated suite of 12 SaaS product clones. We chose widely used enterprise applications spanning scheduling, CRM, project management, HR, commerce, analytics, automation, support, design, and collaboration, capturing both common day-to-day business workflows and a diverse set of product-specific engineering challenges. Each model starts from an empty workspace and constructs a complete application from a product specification. The suite covers Calendly, Figma, HubSpot, Linear, Notion, Rippling (two workflows), Shopify, Tableau, Zapier, Zendesk, and Google Docs.
We evaluate eight frontier models and find several critical failure modes. Models often build the basic surface of a product, including authentication, CRUD, and workspaces, but struggle to implement the product-specific logic that makes those features work correctly together, including concurrency, time logic, validation and rejection paths, API behavior, and interface wiring. The gap shows up across test categories for all model rollouts: authentication features pass 79% of tests, validation 63%, CRUD 53%, concurrency 50%, time logic 43%, and UX tests at 32%.
Our findings suggest that strong performance on repository-based coding benchmarks may overstate how reliably today’s models can build complete products end-to-end.
While models are capable of delivering polished user interfaces, their performance at building out core logic varies by model family.
<h2>Setup</h2>
We focus on product clones because they test one capability that repository-based SWE benchmarks largely avoid: building the product entirely from scratch. A mature repository already contains much of the product substrate, including timezone handling, uniqueness constraints, and error behavior, so an agent editing existing code can inherit those decisions. A model starting from an empty workspace has to infer and implement that logic from the specification alone, making this a useful test of whether frontier models can handle long-horizon software work autonomously.
We evaluate eight frontier models from five developers using the Terminus agent on the Harbor harness, the same containerized infrastructure and grading system used by SWE-Marathon. Every model completes three independent rollouts per task, producing 288 rollouts in total, all of which pass a four-stage audit covering anti-cheat checks, timeout versus infrastructure-failure classification, token-floor sanity, and grader validity.
<h3>What a task looks like</h3>
As in SWE-Marathon, the agent works autonomously in a Linux container with shell access, starting from an empty workspace, and its final build is the graded artifact. Each task consists of a product specification, a golden solution, and two independent test suites, all authored by domain experts. The specification names a well-known product, fixes the operational contract, including the port, health check, persistence requirements, and start script, specifies the API surface in detail, and leaves architectural decisions to the model.
The test suite contains both deterministic and computer-use agent (CUA) tests.
<ul><li>The deterministic test suite executes a pytest suite (39 to 187 tests per task) written against a golden solution: each test passes or fails. Deterministic tests are grouped into themed suites (test_auth, test_convergence) as per the SWE-Marathon convention, which localizes where within the product that performance is lost.</li><li>The CUA test suite evaluates each model rollout through the user interface. For each rollout, Harbor’s Computer1 agent opens the completed application in a live browser and works through the user journeys defined in the task’s rubric.json, using screenshots to perceive the interface and click, type, and navigation actions to interact with it. Each weighted criterion receives a pass or fail based on whether the required functionality can actually be completed through the interface, with screenshots and the agent’s interaction trace recorded as evidence.</li></ul>
Before a task is accepted, experts run the golden solution through the same Harbor harness used for model evaluation, and it must score 100% on both suites across three independent grading runs on the same model rollout.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/3ceb5d8449c1a7d5bb08779ad98116b613c6cba3-2320x1368.png" alt="Diagram of the calendly-clone task folder. Only instruction.md is shown to the model; task.toml, the environment Dockerfile, and the tests/ folder — five themed pytest suites, an anti-cheat script, and a computer-use agent stack with its own rubric — stay hidden and grade the finished build. Scoring is split evenly between a deterministic stack (39–187 pytest checks per task, graded against a golden solution) and a computer-use agent that drives the shipped UI in Chromium with screenshot evidence. Every spec embeds one deliberately hard capability; Calendly&#39;s is timezone- and DST-correct open slots with no double-booking." style="width: 100%; border-radius: 10px;" />
Anatomy of one task: the model sees only the spec (instruction.md). The hidden pytest suites, computer-use agent scripts, and rubric score the completed rollout.
Each task product also embeds one hard axis: the deliberately difficult capability that defines the product’s core value. The remainder is standard SaaS scaffolding, including authentication and sessions, CRUD on core entities, listing and pagination, and workspace setup. The table below lists the hard axis for each of the twelve tasks
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f60f1a33101294c255629c2c840d12d7c8a148f1-1675x1999.png" alt="Table listing the single hard capability each of the twelve tasks targets: Calendly, timezone-correct scheduling; Google Docs, concurrent-edit convergence; HubSpot, derived-state coherence; Figma, export fidelity and error contracts; Shopify, atomic checkout; Notion, structured value validation; Linear, LexoRank ordering and state machine; Zapier, idempotent durable automation; Tableau, query-to-visualization parity; Zendesk, SLA business-hours math; Rippling (leave), PTO accrual by tenure; Rippling (PTO), leave draw-down and balance invariants." style="width: 100%; border-radius: 10px;" />
The hard axis embedded in each of the twelve tasks.
<h2>Results</h2>
Figure 1: Leaderboard
<img src="https://cdn.sanity.io/images/h6s14f4z/production/169c7c1477536cc2009b0e4ca2b1b5747b9c7154-1985x1095.png" alt="SWE-Marathon-Ext leaderboard. Horizontal bar chart of mean pass rate by model, with error bars. GPT-5.6 Sol 74%, GPT-5.5 73%, Grok 4.5 67%, Claude Opus 4.8 65%, Gemini 3.5 Flash 42%, Gemini 3.1 Pro 34%, GPT-5.6 Luna 28%, GLM-5.2 27%. The top four cluster in the mid-60s to mid-70s; the rest fall below 45%." style="width: 100%; border-radius: 10px;" />
Mean overall pass rate per model, averaged over three rollouts per task (mean@3). Higher is better.
By mean@3 score, GPT-5.6 Sol leads at 74%, with GPT-5.5 close behind at 73%, Grok 4.5 at 67%, and Claude Opus 4.8 at 65%. Performance then drops sharply to Gemini 3.5 Flash at 42%, Gemini 3.1 Pro at 34%, GPT-5.6 Luna at 28%, and GLM-5.2 at 27%. 
Even the strongest models leave important product features incomplete or incorrect. GPT-5.6 Sol’s 74% mean@3 means that, on average, about one quarter of the evaluated behavior still fails. Our later analysis shows that these misses correspond to product failures that can make an application unreliable or unusable for its intended business workflow, such as double-booked time slots, oversold inventory, or invalid requests being accepted.
The scores also span a wide range, from 74% to 27%. That separation is important because frontier models often score much closer to one another on established benchmarks such as SWE-bench Verified, making it harder to distinguish models that may behave differently in practice. Hence, SWE-Marathon-Ext 47 point spread makes the benchmark useful for categorizing model capabilities on long-horizon, end-to-end software construction.
Figure 2: Deterministic vs CUA Pass Rates per model
<img src="https://cdn.sanity.io/images/h6s14f4z/production/36ba153e35f33655c3787cfe48d4ed087e79d44f-2621x1174.png" alt="Grouped bar chart comparing each model&#39;s deterministic backend-API pass rate against its computer-use-agent browser-interface pass rate, with error bars. GPT-5.5 78% vs 67%, GPT-5.6 Sol 71% vs 78%, Claude Opus 4.8 71% vs 59%, Grok 4.5 70% vs 65%, Gemini 3.5 Flash 43% vs 41%, Gemini 3.1 Pro 40% vs 29%, GPT-5.6 Luna 39% vs 17%, GLM-5.2 33% vs 20%. Browser scores trail backend scores for every model except GPT-5.6 Sol, and the gap widens toward the bottom of the field." style="width: 100%; border-radius: 10px;" />
For each model, the mean deterministic score (backend API correctness) against the mean CUA score (whether a computer-use agent can use the user interface), both as percentages.
For seven of eight models, mean CUA scores are lower than mean deterministic scores. Across all eight models, the difference is statistically significant (paired t-test, p = 0.026), with CUA lower by 7.6 percentage points on average. This suggests that the two evaluations may capture different properties of the same build, a pattern that also appears at the level of individual models and tasks. On HubSpot, for example, GPT-5.5 passes roughly 85% of deterministic checks, correctly implementing much of the underlying CRM logic for contacts, deals, pipeline stages, and lead scores, but only about 14% of CUA criteria because those capabilities were not reachable through the interface. GPT-5.6 Sol shows the reverse pattern: workflows appear to succeed in the UI while deterministic checks indicate that the underlying state change is incorrect, incomplete, or sometimes does not occur at all. Taken together, these cases suggest that using both evaluations can provide a fuller picture of whether functionality is both correctly implemented and usable through the interface.
Figure 3: Basic functionality vs hard axis, pass rate per task
<img src="https://cdn.sanity.io/images/h6s14f4z/production/8f40d57b80705a988c4343ce7a891754ad07cb70-2621x1228.png" alt="Grouped bar chart contrasting each task&#39;s basic surface pass rate (best test group) with its hard-axis pass rate (worst test group), with error bars. Google Docs 84% vs 60%, Tableau 80% vs 66%, Zendesk 91% vs 49%, Figma 91% vs 0%, Rippling (leave) 84% vs 23%, Rippling (PTO) 75% vs 29%, Zapier 77% vs 0%, HubSpot 68% vs 0%, Calendly 84% vs 33%, Linear 80% vs 31%, Shopify 72% vs 8%, Notion 75% vs 2%. Basic surfaces pass in the 68–91% range everywhere, while hard axes collapse — four tasks score under 10%, three of them at zero." style="width: 100%; border-radius: 10px;" />
For each task, we compare the test group models handled best (basic surface, grey) with the task-specific capability they handled worst (hard axis, indigo), across all graded rollouts.
Each task has two associated bars, aggregated across all model rollouts. The grey bar shows performance on the basic surface: login, workspaces, and CRUD. The indigo bar shows the hard axis, the task-specific capability central to the product’s core business function. The gap between the two is substantial across tasks. Calendly builds pass 84% of basic-surface tests but only 33% of scheduling tests; Shopify drops from 72% to 8% on concurrent checkout; Notion drops from 75% to 2% on validation. Zapier and HubSpot fall to 0% on their hard-axis tests. Overall, models are much more reliable on standard SaaS functionality than on the task-specific capability the product is designed around.
<h2>Where builds break: the difficulty gradient</h2>
<img src="https://cdn.sanity.io/images/h6s14f4z/production/6c4af39fbe03d6fd363707d542a1409809346b52-2048x1411.png" alt="Heatmap of pass rate by task (twelve rows) and capability category (eight columns: login, validation, CRUD, data ops, concurrency, time/scheduling, domain math, UX/browser); darker cells mean higher pass rates, dashes mean no tests in that cell. Login is the strongest column, at 61–91% for every task. Scores degrade moving right: concurrency ranges from 8% (Shopify) to 91% (Figma), time/scheduling from 23% (Rippling leave) to 86% (Figma), and UX/browser bottoms out at 0% for Zapier and HubSpot and 2% for Notion. Notion is the weakest row overall — 75% on login but 6% validation, 7% CRUD, 11% data ops." style="width: 100%; border-radius: 10px;" />
Pass rate (%) by app and verifier axis, averaged across eight models. Apps ranked easiest to hardest (top to bottom); axes easiest to hardest (left to right). The darker-to-lighter drift shows that models clear Login and Validation nearly everywhere and fail Concurrency, Time logic, and UX, where most SaaS complexity lives. Notion and Shopify are the floor of all eight apps; login is the one column strong top to bottom.
<h2>Three failure modes drive the loss</h2>
When we dug into why builds fail the hard axis so consistently, three patterns emerged. Each cell in the table below depicts the number of failing deterministic checks over the total number of deterministic tests for that mode, aggregated across a model’s rollouts (3 rollouts per each of 12 tasks):
<img src="https://cdn.sanity.io/images/h6s14f4z/production/abddd7fab31591ca74b3717e1d196f047c314164-2320x902.png" alt="Table of failed checks per model across three failure modes. Totals: GPT-5.5 138/720, GPT-5.6 Sol 215/720, Claude Opus 4.8 191/720, Grok 4.5 175/720, Gemini 3.5 405/706, Gemini 3.1 503/720, GPT-5.6 Luna 422/636, GLM-5.2 528/720 — 2,577 of 5,662 checks failed overall. Broken out: concurrency and idempotency failures (two racing requests corrupting shared state) 630/1,265; time, timezone and scheduling failures (clock treated as fixed, every zone as local) 951/1,672; unhappy-path contract failures (bad input not rejected with correct status codes, caps, and permission checks) 996/2,725." style="width: 100%; border-radius: 10px;" />
<h3>Concurrency and idempotency</h3>
Concurrency and idempotency failures happen when two requests modify the same state at once, or when the same request is processed twice. Across model rollouts, we observed builds failing to protect against these cases with transactions, locks, or idempotency keys.
In Shopify, for example, two concurrent checkout requests race for the last unit and both orders are confirmed, thus overselling the unit, as observed in 22/24 Shopify rollouts (8 models, 3 rollouts each). Calendly shows the same pattern, with two invitees booking the same time slot in 16/24 rollouts.
We saw this failure mode frequently in tasks like Shopify and Zapier, where inventory and durable job execution are core functionality, and much less often in Figma, at 4/24 rollouts, where most operations do not contend over the same state.
<h3>Time, timezone, and scheduling</h3>
Time-based logic is the most prevalent failure mode across all model rollouts because correctness depends on product-specific rules: timezone conversion, daylight-saving changes, business hours, tenure thresholds, etc,
In Calendly, model rollouts compute open slots through local clock times without checking a timezone database, thus offering a slot during an hour that does not exist on the spring-forward day (in 18/24 rollouts). In Rippling, small errors in the accrual rate at a tenure threshold affected later tests that depend on the employee’s balance, appearing in 17/24 rollouts.
These failures were most common in products with substantial time-based logic, such as Calendly, Rippling, and Zendesk, with 15/24 Zendesk rollouts also showing time-related failures, and were rare in products like Figma, at 3/24 rollouts, where time plays little role in core functionality.
<h3>Unhappy-path contracts</h3>
The third failure mode is incorrect handling of invalid or disallowed requests. Most builds can handle valid input. The harder question is whether they return the correct error, enforce limits and permissions, and reject malformed data.
For web APIs, client errors should usually return a 4xx status code: 404 for a missing record, 409 for a conflict, or 422 for a validly formed request that violates a constraint. Weak builds instead return 200 as though the request succeeded, or 500 as though the server failed.
In Notion, for example, we saw model rollouts accepting a malformed date or invalid multi-select value instead of rejecting it in 23/24 rollouts. Figma exposed a failure where accounts in model rollouts could access private workspaces belonging to other accounts (in 18/24 rollouts). Unlike the first two failure modes, this pattern appeared across nearly every product because every API has an error contract.
<h2>Why this matters</h2>
Across twelve zero-to-one product tasks, frontier models reliably build standard SaaS functionality such as authentication, workspaces, and CRUD flows, but struggle with the task-specific behaviors that make each product correct and usable.
These results suggest a next step for evaluating long-horizon SWE tasks: stress-testing completed products under sustained, multi-user interaction, where concurrent actions and conflicting state changes can expose failures that fixed test cases may not. Future evaluations could use agentic swarms or simulated users with varied goals and behaviors to generate this traffic, for example by having two shoppers compete for the same inventory, multiple users update the same record, or overlapping bookings contend for the same time slot. Thus, as coding agents take on increasingly complete software projects, the decisive question for evaluation shifts from whether they can build a product to whether that product holds up under real conditions of load and use, the point at which it becomes usable and capable of delivering real business value.
Acknowledgments: Shoutout to our friends at Abundant for coming up with a novel benchmark. Thank you to Bertie Vidgen and the broader Research team for their thoughtful feedback and guidance throughout the review process. Thank you also to Sameer Bhadouria and Aravind Selvan for their editing and feedback, and to the broader Code Data Team for helping ideate and shape the task design behind SWE-Marathon-Ext. We are grateful for the many iterations and discussions that helped refine both the benchmark and this analysis.

Keya Varia

Product Manager

Chirag Mahapatra

Member of Technical Staff, Mercor

Abhi Kottamasu

Claire Gauthier

Senior Strategic Product Lead

Eight frontier models built twelve SaaS products from an empty workspace. They ship login and CRUD reliably, then break on concurrency and time logic.

SWE-Marathon-Ext: Can AI Build a Whole SaaS Product?

SWE-Marathon-Ext: Can AI build a whole SaaS product?

Introducing <a href="https://www.mercor.com/apex/apex-accounting-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Accounting</a>, a new benchmark built in collaboration with Ramp.
Accounting benchmarks often test whether a model can produce the right answer once. Closing the books demands more. An agent must reconcile conflicting files, apply company-specific context, carry conclusions across multiple steps, and produce the correct result consistently. A model that drops a correct intermediate answer can still create a bad journal entry. APEX-Accounting builds on the evaluation system Ramp developed while building Stack, their AI operating platform for accountants. Ramp and Mercor built APEX-Accounting to test whether AI agents can complete real accounting work to professional standards. The benchmark includes 160 tasks across 10 simulated companies. Experts with prior experience at firms like Deloitte, PwC, EY, and KPMG hired through Mercor&#39;s platform created the tasks and grading rubrics.
<iframe src="https://player.vimeo.com/video/1214103504?h=3e64e11412" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="APEX-Accounting | Mercor x Ramp" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
The headline result is not which model leads, but how rarely any model succeeds consistently. We ran every model on every task eight times because accounting work must be repeatedly correct, not correct once. Even the most consistent model solved just 2.6% of tasks correctly in all eight runs.
This is a very different result from performance on accounting exams. Models can produce the right answer in a controlled test and still fail to carry accounting judgment through a real workflow.
<h2>Leaderboard results</h2>
Claude Fable 5 tops the leaderboard at 56.4%, followed by Meta&#39;s Muse Spark 1.1 at 52.6%, and GPT-5.6 Sol at 51.5%. The best models manage to complete just over half of the work that a professional would.
For Pass@8, which measures whether a model is successful in at least one of eight attempts, Muse Spark 1.1 is highest at 21.5%, just ahead of Fable 5 (20.1%).
Models frequently earned partial credit. At least one model earned some credit on more than 95% of tasks. Full solutions were more difficult. 58% of tasks were never fully solved by any model on any run.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/55f5493d06f186cc9c8f60edb2948d624ebc836b-3530x2913.png" alt="Horizontal bar chart titled &quot;APEX-Accounting leaderboard&quot; ranking nine AI models by mean percentage of criteria passed, with error bars. Claude Fable 5 leads at 56.4%, followed by Muse Spark 1.1 at 52.6% and GPT-5.6 Sol at 51.5%; Qwen 3.5 is lowest at 24.4%." style="width: 100%; border-radius: 10px;" />
<h2>Performance vs. cost</h2>
We tested models at spending budgets of $1, $5, $10, and $50 per task. A larger budget allows models to use more tokens, which generally improves results. But the effect varies enormously. Fable 5 was extremely budget sensitive, scoring 11.8% with a $1 budget and improving to 55.2% with a $50 budget. Muse Spark 1.1 is the opposite. It&#39;s already strong on a tight budget and barely improves with more money. This is because above the $1 cap, models typically do not fully utilize the token limits. There is only a moderate increase in token usage when going from $10 to $50 and, on average, models use just 64.7% of the maximum budget available at $50.
Models differ in how they utilize the capped budget. At the $50 maximum spending budget, Fable 5 actually spends ~$32 per run while Muse Spark 1.1 spends ~$5, yet their scores are within 4 percentage points.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/2b80bc85d99be2ccddc06f84f73a22f19ff366be-5700x2422.png" alt="Two-panel line chart, &quot;APEX-Accounting | Budget scaling,&quot; tracking five AI models&#39; scores as spend increases. Most models plateau by a $5 budget cap, while Claude Fable 5 keeps improving, rising from about 13% at $1 to 56% at $50 and overtaking Muse Spark 1.1 between $10 and $50." style="width: 100%; border-radius: 10px;" />
<h2>What the leaderboard measures</h2>
The canonical leaderboard runs every model in the Loop Harness, Mercor&#39;s standard model-and-tools architecture. The paper separately compares it with a purpose-built Ramp Harness, created with Ramp to mirror parts of a specialized accounting-agent architecture. The Ramp Harness improves Mean Criteria@3 by 1.2 percentage points on average.
This is a harness ablation, not an evaluation of the complete Ramp Stack product. It does not measure Stack&#39;s production integrations, accounting skills, memory, controls, auditability, or recent spreadsheet tooling.
<h2>Why do models fail?</h2>
Working with accounting and bookkeeping experts, we developed a taxonomy of the mistakes that prevented models from completing each task. These included failures in reasoning, information gathering, instruction following, and planning and reflection.
The top three models all fail in much the same way. Roughly seven in ten failures came from flawed reasoning, not an inability to find the right information. A model might correctly identify a discrepancy early in the workflow, then omit or contradict that finding in its final journal entry. Better retrieval alone will not solve this. Models need stronger accounting judgment and more discipline in carrying conclusions through an entire workflow.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/8165b25b11a0e8180beaa98cad14ca03b470fda2-4038x2422.png" alt="Grouped bar chart, &quot;APEX-Accounting | Failure taxonomy,&quot; showing how three AI models&#39; graded failures break down across four categories. Reasoning dominates for all three: 19 of 24 failures for Claude Fable 5, 21 of 28 for GPT-5.6 Sol, and 13 of 22 for Muse Spark 1.1, with information gathering, instruction following, and planning and reflection accounting for the rest." style="width: 100%; border-radius: 10px;" />
<h2>Methodology</h2>
More than 40 accounting professionals authored and solved the benchmark tasks. They had a median of 11 years of experience, and more than half had worked at a Big Four accounting firm. Each task takes place in a self-contained world: a fictional company frozen at month-end close, with its own accounts, records, business history, accounting software, spreadsheets, PDFs, and other files.
The companies themselves are fictional but every transaction, discrepancy, and edge case inside was written by accountants who do this work for a living. The experts authored a rubric for each task with, on average, 13.7 criteria. An open-source AI judge performs the grading and achieves 97% agreement with expert graders.
APEX-Accounting focuses on month-end close and bookkeeping workflows. It does not evaluate tax, audit, consolidation, multi-entity or multi-currency accounting, external reporting, or how agents respond when they need clarification. Its findings should be understood within that scope.
The APEX-Accounting leaderboard comprises a heldout test set of n=160 tasks (associated with 10 worlds), kept hidden to resist contamination. We have released an open-source sample set with 1 world and 10 tasks on <a href="https://huggingface.co/datasets/mercor/apex-accounting-harbor" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a>, and Archipelago, our internal framework for running agent evaluations, is available on <a href="https://github.com/Mercor-Intelligence/archipelago" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GitHub</a>.
Read the APEX-Accounting <a href="https://arxiv.org/abs/2607.27189" target="_blank" rel="noopener noreferrer" class="text-indigo-600">technical report</a> to learn more about the methodology and results.
<h2>APEX benchmarks</h2>
APEX-Accounting joins Mercor&#39;s family of APEX benchmarks, which evaluate AI models&#39; ability to do economically valuable work. Other APEX benchmarks include APEX-SWE for software engineering tasks across integration and observability, and APEX-Agents for professional services domains including corporate law, management consulting, and investment banking.
We thank <a href="https://ramp.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Ramp</a> and all the accountants on the Mercor marketplace who contributed their time to creating APEX-Accounting.
As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request. Reach out <a href="https://www.mercor.com/partner" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a> to find out more.

Austin Bennett

Ryan Stevens

Director of Applied Sciences

Julien Benchek

Senior Strategic Project Lead

APEX-Accounting, built with Ramp, measures whether AI agents can reconcile accounts, analyze variances, and close the books. See the full results.

APEX-Accounting: AI Productivity Benchmark for Accounting

Introducing the AI Productivity Index for Accounting

Today&#39;s frontier AI models can pass nearly any exam you give them, but put an agent inside a real workflow and it stalls on the first task an entry-level employee could handle. This is because the model lacks experience.
Experience comes from doing real work: navigating messy documents, legacy enterprise software, and sprawling company context while making hundreds of small decisions along the way. It means knowing what to do when something breaks or when the next step isn&#39;t obvious. A model learns to perform well only by practicing repeatedly in the environment where the work takes place. Building environments rigorous enough for that kind of training is incredibly hard. That&#39;s where <a href="http://deeptune.ai/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Deeptune</a> excels.
<h2>Today we&#39;re announcing that Mercor will acquire Deeptune, a leading company building environments for reinforcement learning.</h2>
Reinforcement learning has reached the point where a model can learn almost any task that can be clearly defined and scored. The constraint has shifted to the environments themselves: the places where models practice the work and get measured on whether they did it well.
Building environments like these is exceptionally difficult. Each one has to faithfully recreate the software people use, translate expert judgment into realistic tasks, and measure performance clearly enough for a model to improve.
Mercor&#39;s experts are already doing much of this work by writing evals. Deeptune provides the missing piece: the software platform where that expertise becomes realistic training environments at scale.
<h2>Building the place agents learn to work</h2>
Every environment has three parts: the software where the work takes place, the tasks that define what the agent must accomplish, and the verifiers that determine whether it succeeded.
Mercor has built the expert layer behind that system. Our network of more than five million domain experts creates the tasks and verifiers that translate human judgment into learning signals for frontier models. Through APEX, we measure model performance on economically valuable tasks across thousands of real-world workflows.
Deeptune builds the software layer. Over the past two years, the team has recreated hundreds of enterprise applications — from spreadsheets to Salesforce — and built some of the most sophisticated training environments used by frontier AI labs. We&#39;ve seen the quality of their work firsthand because Mercor is already a Deeptune customer.
Together, Mercor&#39;s expert network and Deeptune&#39;s software platform make it possible to build realistic training environments across far more industries, roles, and workflows than either company could alone.
<h2>The team</h2>
None of that happens without the people building it. I first met Tim, founder and CEO of Deeptune, two years ago, long before we started talking about working together. I was impressed by the technical ambition of Deeptune and Tim&#39;s conviction that environments would become the defining bottleneck for AI, long before most of the industry saw it. He and his team executed on that vision with remarkable speed. That&#39;s why I became an early investor, and ultimately why it felt obvious to bring our companies together.
Deeptune is a small, focused team of engineers and operators with deep ties to frontier research. Earlier this year, Andreessen Horowitz led Deeptune&#39;s $43 million Series A. Tim and the entire Deeptune team are joining Mercor as we expand our team in NYC.
<h2>Organizing human intelligence together</h2>
For the labs we work with, this means environments that ship faster and train better. For enterprises deploying agents into their own workflows, it means one partner that can build the environment, staff the experts, and tell you whether the agent is ready.
Every frontier lab is scaling environment production, and every enterprise will need to do the same. Training agents is becoming one of the largest categories of work in the economy, and the demand cuts across every profession.
That is what we are building toward, and it is how we organize human intelligence to power the AI economy.
Welcome to Mercor, Deeptune.

Mercor's five million experts plus Deeptune's platform make it possible to build realistic training environments across more industries and roles.

Mercor acquires Deeptune to build AI training environments

Mercor to acquire Deeptune

<h2>Abstract</h2>
Enterprise agents increasingly operate across tools, documents, databases, and systems of record. Their value depends not only on model capability, but on whether their work can be measured, diagnosed, improved, and safely deployed. We present a framework for evaluating enterprise agents using verifier-backed judgments over agent trajectories and outputs. A verifier is an isolated program or agent that inspects a run, returns a normalized score in [0, 1], and explains the basis for the judgment. Verifiers can be scoped to trajectory behavior, final artifacts, external state changes, or combinations of all three.
The same evaluation object supports four deployment functions: production grading, offline regression tests, model and prompt benchmarking, and runtime quality gates. We define the core objects in this system, including task, environment, trajectory, output, verifier, activation condition, and cumulative reward, and describe how they compose into an optimization loop for agent improvement. We also describe the deployment architecture required for enterprise use, including customer-controlled data planes, configurable artifact sharing, and managed update postures.
The central claim is that enterprise agent reliability is an evaluation-systems problem. Agents improve when their work is converted into calibrated, hard-to-game feedback signals that can drive benchmarking, rollout decisions, and continuous optimization.
<h2>Part I. Evaluations</h2>
<h3>1. Definitions</h3>
An evaluation is a run plus a verifier-backed judgment of that run. The terms below are used consistently throughout.
<h4>1.1 Vocabulary</h4>
<ul><li>Task. The instruction prompt given to the agent to execute. For example, &quot;Create a 10-page due-diligence report on a target company,&quot; or &quot;Screen a candidate in the applicant tracking system.&quot;</li><li>Environment. The external APIs, databases, and integrations the agent can access during execution. It is the production environment for online runs, a mocked or synthetic environment for offline runs, and is sometimes unnecessary when the agent only reads inputs and writes to its own filesystem. For example, a mock document dataroom with a folder for the agent to deposit its output.</li><li>Trajectory. The agent&#39;s full execution history: its reasoning and messages, and the inputs and outputs of every tool call.</li><li>Agent output. The artifacts the agent emits and the external state it changes. For example, a deck created, an email sent, or a record advanced in a system of record.</li><li>Verifier. A separate agent or script that grades the trajectory, the output, or both. It returns a score in [0.0, 1.0] and an explanation, and its definition carries a weight. A verifier runs in its own sandbox and may use a different model from the agent under test. For example, a verifier might compare a deck to a gold-standard reference, or check that a required file exists in the right folder.</li></ul>
<h4>1.2 Evaluation tests</h4>
Evaluation tests are the offline mechanism for measuring, experimenting on, and improving an agent before changes reach production. Each test is governed by three objects. The task is the prompt the agent must complete during the offline run. The offline environment is a mocked or synthetic version of the production instance, including the integrations, fixtures, and external state needed for the task. The verifier is the grader that inspects the resulting trajectory and output, then returns a score and explanation.
<h3>2. How evaluations work</h3>
<h4>2.1 Online and offline runs share one shape</h4>
An online run is invoked by a user in production. It uses the live production deployment, and production verifiers grade the run after completion. An offline run is launched from evaluation tests: the platform spins up the agent inside the offline environment, runs the task, and then applies the listed verifiers.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/0ee21c35be52b9351b67d9cf952c665f1cda99ab-931x105.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 1. The evaluation shape. Online runs happen in production and are graded afterward; offline runs use the tasks, mocked environments, and verifiers defined by evaluation tests.
For online runs, each production verifier has an activation condition so irrelevant checks do not run. A deck-quality verifier activates only when the task or output indicates a deck was requested, and does not activate for a screening task. For offline runs, activation is usually simpler, because the evaluation tests already name the task, environment, and verifiers that should run.
<h4>2.2 One definition, three roles</h4>
A verifier definition is reused in three places. As a runtime quality gate, blocking verifiers can prevent delivery until the output clears a threshold, then return feedback to the agent for revision. In production grading, activated verifiers grade production runs after completion, adding quality scores and explanations to cost, latency, and tool telemetry. In offline experimentation, evaluation tests run when a model, prompt, skill, or harness change is proposed and quality needs to be measured before rollout.
<h4>2.3 The cumulative reward, and what verifiers grade</h4>
Each verifier states what it grades (trajectory, output, or both) and its relative importance. Scores are normalized to [0, 1] so they remain comparable across tasks, dashboards, models, and optimization runs. The cumulative reward is the normalized weighted average. 
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ef9a20db0587c244549f886b699730e87e572e19-856x64.png" alt="" style="width: 100%; border-radius: 10px;" />
Inactive verifiers are excluded from both sums. Explanations are retained because the optimization loop uses both signals: the reward supports candidate selection, and the explanation guides the next revision.
<h4>2.4 Base versus agent-specific verifiers</h4>
Every agent inherits a small set of base verifiers for platform-wide expectations such as instruction following, required artifact delivery, and tool-calling efficiency. Each agent can then add agent-specific verifiers for domain quality. An account intelligence agent might add account context, source grounding, support risk, product usage, recency, and actionability verifiers. A verifier definition includes its model, activation prompt, verifier prompt, scope, production flag, blocking settings, threshold, and weight.
<h4>2.5 Worked example: enterprise account intelligence agent</h4>
Consider an agent asked to produce a strategic account brief for an upcoming executive meeting with a large customer.
The task is the user instruction and any attached requirements, such as the customer name, meeting objective, required sections, source constraints, and output format. The environment includes the CRM, prior meeting notes, support tickets, product usage data, approved internal documents, public company information, and the filesystem where the brief must be written. The trajectory records the agent&#39;s tool calls, source reads, extracted facts, intermediate drafts, and revisions. The output is the final account brief and any state changes made in connected systems. A verifier set for this task could include the following.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/dce918620c0b9f763c6cad045daf1bd972f53ca3-919x538.png" alt="" style="width: 100%; border-radius: 10px;" />
Each verifier returns a score and explanation. The weighted average becomes the cumulative reward for the run. If a blocking verifier fails, the agent can be asked to revise before the brief is delivered. If a non-blocking verifier fails in production, the run can be converted into a new offline evaluation test so future agent versions are checked against the same failure mode before rollout. The same structure generalizes beyond account intelligence. Once tasks, environments, trajectories, outputs, and verifiers are defined, they become a reusable package for benchmarking, optimization, rollout gates, and production monitoring.
<h2>Part II. Benchmarks and the optimization loop</h2>
<h3>3. The reusable package</h3>
Once an agent has evaluation tests, the same package can be reused for benchmarking, A/B tests, automated prompt optimization, rollout decisions, and model selection. The package has three parts. The environment is the agent&#39;s action space and external integrations, wrapped in a container together with mock tool calls. The tasks are prompts that cover the full distribution of work the agent must perform. The verifiers are attached to each task, producing scores and explanations that are combined by weight into the cumulative reward defined in Section 2.3.
The same grammar is portable. Tasks, environments, verifier definitions, weights, and scoring outputs can be adapted to external benchmark task and environment formats, which makes evaluation tests easier to port into outside benchmark systems while preserving customization for customer-specific tools, verifiers, and deployment constraints. The workflows below vary with different inputs, but they all consume the same bounded reward and verifier explanations.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7607d081c66d79c4b7f53a4cc5a363bb65a4779b-926x354.png" alt="" style="width: 100%; border-radius: 10px;" />
<h3>4. The optimization loop</h3>
Optimization starts with the main agent and a task suite. The task suite is a set of representative offline evaluation tests, each with a task, environment, and verifier set. It defines what the agent is expected to do well, and provides the standard against which every candidate version is measured.
The loop optimizes a generator agent. At the beginning of the loop, the generator agent is initialized as a copy of the main agent: same prompts, skills, harness, and model configuration. From there, the generator agent becomes the moving target. Each iteration runs the current generator agent through the task suite, grades its performance, and then updates the generator agent based on the verifier feedback.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/1027e3ec8e0501bc30cfa2bc5f4efc97e261e161-735x626.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 2. The optimization loop.
1. Initialize the generator agent. Start with the current main agent as the baseline. This creates the first generator-agent version.
2. Run the task suite. The generator agent runs across the offline evaluation tests. For each task, it acts in the test environment, produces a trajectory and output, and is graded by the relevant verifiers.
3. Collect reward and explanations. The verifiers return per-task scores, per-verifier explanations, and a cumulative reward. The reward shows how well the generator agent performed; the explanations describe why it failed or succeeded.
4. Call the fixer agent. The fixer agent reads the generator agent&#39;s current files, the task trajectories, the verifier scores, and the verifier explanations. It diagnoses the failures and edits the generator agent directly, updating prompts, skills, harness instructions, or other agent-layer behavior.
5. Advance the generator version. The fixer agent&#39;s edits create the next version of the generator agent. That updated generator agent is then rerun on the task suite.
6. Keep improvements. If the new generator version improves cumulative reward on the validation set, while staying within acceptable cost and latency bounds, it becomes the leading candidate. If it regresses, the system keeps the stronger prior version and continues.
<h3>5. What the loop depends on</h3>
The loop is only as good as three things, and it is worth being direct about where it is hard.
Verifiers can be gamed: if a verifier rewards a shortcut, the agent will learn the shortcut, so verifiers need to be deterministic where possible and calibrated against human judgment.
Coverage is a real constraint: the evaluation tests are only as useful as the slice of real work they cover, and gaps in the task distribution show up as blind spots in production.
And the strength of the improvement loop is bounded by how much signal is available to it: with little evidence from production, there is less ability to diagnose regressions, derive test environments, and validate updates.
That last point is a genuine tradeoff, not only a data-sensitivity question, and it is the subject of Part III.
<h2>Part III. Where this is differentiated</h2>
<h3>6. Where this is differentiated</h3>
Evaluation is the control point for the agent lifecycle. It determines which model to use, which prompt to ship, whether a harness change improved performance, and whether an output is ready for production. The team with the strongest evaluations can build the stronger agent, because it can measure the work that matters.
Mercor&#39;s advantage comes from doing this work in both frontier model development and enterprise deployment. For frontier AI labs, evaluation work requires decomposing complex tasks, defining quality bars, sourcing expert judgment, and building feedback loops that improve model behavior. For enterprise customers, the same evaluation discipline has to be adapted to agents that operate inside business workflows: reading internal documents, calling tools, updating systems of record, producing artifacts, and making recommendations that business teams can trust. Three capabilities decide whether that system works.
First, verifiers have to be hard to game. A weak verifier does not just miss problems. It trains the wrong behavior. Strong verifiers are deterministic where possible, calibrated against expert judgment, and designed to catch the failure modes that actually matter in production.
Second, coverage has to reflect the real distribution of work. Evaluation tests cannot only cover clean prompts and easy cases. They need to include ambiguous instructions, missing context, stale data, tool failures, conflicting evidence, and the edge cases that appear once an agent is used by real teams.
Third, domain expertise matters. A generic evaluator can check format, length, and surface-level completeness. It cannot reliably judge whether an account brief identifies the right risks, whether a support escalation is represented accurately, whether a market map is shallow, or whether a recommendation is commercially useful. Those judgments require practitioners who understand the work.
The differentiation is the combination of expert evaluation, hard-to-game verifiers, and a platform that turns those verifiers into measurement, experimentation, optimization, and rollout decisions. Mercor is not only deploying agents. It is deploying the evaluation system that makes agents measurable, improvable, and safe to update over time.

Mercor Enterprise's framework for agent evals: how to measure, improve, and safely deploy enterprise agents using verifiers over trajectories and outputs.

Agent Eval Systems | Mercor Enterprise

Agent Eval Systems

Electric. That was the feeling I had when I walked into the Mercor office. The vibrant energy was reminiscent of my first day at Tesla, where I spent the majority of my career. The office was buzzing: a group huddled on the couches by the elevator executing their near-term roadmap, a team whiteboarding product features behind the glass panes of the conference room, people moving with urgency between rooms for the next customer call. Mercor possesses the same early-day Tesla energy, and I knew immediately this place was special. I’ve spent my career chasing that feeling: the pull to help build a generational company just as it defines a new industry.
I entered the workforce during the Great Recession: interning at Merrill Lynch on Wall Street the summer before it was fire-sold to Bank of America, and joining Citigroup in investment banking full-time after graduation. The job market was grim, so I felt fortunate to land a job in 2009 — especially in finance. The Great Recession shaped my worldview and defined my career trajectory.
During the recession, the Obama administration passed the American Recovery and Reinvestment Act: a stimulus package that poured investment into the emerging cleantech industry. On the energy team at Citi, I advised dozens of fast-growing companies in the space. Wanting to be closer to the technology, I took my first operating role at First Solar, which is today the largest American solar module manufacturer. It was my first taste of the exhilaration of being hands-on in the wild-west days of a new industry.
A couple of years later, I left for an MBA at Wharton. A new era had emerged: smartphones became dominant, social media was omnipresent, and cloud computing transformed enterprises. The Big Tech companies of today were still relatively young and rapidly evolving. I interned at Amazon during my MBA, but the company was already established, and the energy, pace, and ambition felt off.
I entered the second year with my eyes set on Tesla. Elon had radical ambition, and I was an early believer that Tesla would be a transformative, generational company. I spent an entire year hustling to be hired, and finally received an offer months after graduation to join as the first finance hire to support a new idea — a bet that would become Tesla Energy.
Although a public company at the time, Tesla was hardly mature — and Elon’s unconventional leadership style greatly influenced me. I worked in sales to assist in deliveries, and on the manufacturing floor to facilitate material handling. We were encouraged to be hands-on — and that firsthand knowledge enabled fast strategic execution to scale the business while defining an industry. We built Tesla Energy — and the commercial battery storage industry — from scratch; today it’s a $13B revenue business. With the itch to build again, I joined an early stage AI robotics company, and helped close their Series C and land a strategic partnership with FedEx.
The throughline in my career has been a desire to partner with generational founders who have the ambition to build transformative companies that define new industries. I felt the spark of inspiration immediately upon meeting Brendan and Adarsh.
A few things made Mercor an obvious choice for me. The first is the two of them. I’m inspired by founders seeking to achieve the impossible. I love the challenge of identifying and executing the path from where we are today to where we aspire to be. Brendan and Adarsh see the future clearly the way Elon does, and they propel Mercor forward to shape the world they see. Their conviction and optimism — and how Mercor will drive extraordinary change in society — is infectious: it sparks ambition in everyone around them. I’ll always remember the glee Brendan showed when he showed me Mercor’s product for the first time — proud of what the company had built, grateful for the talent it had drawn, and electric about what lies ahead.
The second is the people. Mercor reminds me of Tesla: a collective culture — each individual operates with exceptional autonomy, yet shares a deep sense of humility and interdependence in pursuit of a common goal. I’m continuously inspired by my colleagues, each driven by insatiable curiosity, optimism for what we can achieve, and agency to effect change. We’ve hired extraordinary people.
The third is the mission. Mercor operates at the unique intersection of humans and AI, and has built the human intelligence infrastructure that powers the most transformative technology in our lifetime — and, I believe, for the better. I’ve spent an inordinate amount of time in my nearly two-decade-long career on manual, tedious, and repetitive work. We are creating the largest new job category, and advancing the frontier to unburden each profession of repetitive tasks and free people to do work that matters.
Mercor is creating a new industry that will reshape the economy, and is growing faster than any company in history. Finance is at the center of building the foundation for scale: data infrastructure, systems implementation, operational and financial visibility — and we are tasked to lay that groundwork on a compressed timeline. What excites me most is that what we’re building will change finance itself — a function that has historically been intensely manual. Legacy companies run armies of people managing general-ledger entries, expense approvals, financial analysis, and business-intelligence dashboards. At Mercor, we are designing the future of finance: a finance function built in an agentic world. Finance here will build the evaluations and train the agents that do that work, freeing us to spend our time drawing business insights and driving execution.
We will be defining what finance will be. That’s the work, and it’s why I’m hiring. If that’s the thing you want to be doing, come build it with us: <a href="http://www.mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">www.mercor.com/careers</a>.

Mercor's CFO spent eight years at Tesla. He joined for the same early-day energy, the founders, and the chance to redefine finance.

Why I Joined Mercor – Kevin Shiau

Why I joined Mercor – Kevin Shiau

In late March 2026, Mercor was affected by a supply chain attack involving LiteLLM, a commonly used open-source tool. The hack impacted a significant number of companies globally. We quickly discovered the activity, secured our systems, and conducted a robust and comprehensive investigation assisted by leading third-party forensics experts. That investigation is now complete. We want to share what happened, what we found, and what we’ve done to protect our customers, experts, and employees.
<h2>What happened</h2>
In March, a malicious actor published compromised versions of LiteLLM designed to exfiltrate credentials from any system that installed them. Our security team determined that Mercor was affected during the relevant timeframe and immediately took action to contain unauthorized activity.
We worked with leading industry experts including Google’s Mandiant, Latacora, industry peers, and law enforcement to investigate and take appropriate action. We worked to get answers as quickly as possible while also prioritizing accurate information.
<h2>What the investigation found</h2>
<ul><li>Experts: Of our nearly five million experts, only a very limited subset had sensitive information affected. There is no evidence that any of this data has been used fraudulently. We are in the process of notifying these individuals directly on June 25 and June 26 from mercor@notifications.cyberscout.com. The notifications will include details about the type of information affected along with an offer of TransUnion identity protection services.</li><li>Customers: Because of how our work is structured, many of our customers operate on their own platforms rather than ours, meaning the impact to customer information was very limited. We were in direct and regular contact with our customers throughout the investigation and have shared findings specific to each of them. We are grateful for their cooperation as we worked through this situation and are pleased to report that all frontier labs have increased their work with us over the last few months.</li><li>Employees: No employee data was affected.</li></ul>
<h2>What we’ve done</h2>
<ul><li>We prioritized timely and direct communication with our customers, experts, and employees as we had information to share.</li><li>We’ve taken steps to further invest in and strengthen our security posture, including:<ul><li>Auditing all third-party dependencies</li><li>Regularly rotating all credentials and access keys across our cloud platforms, GitHub, and SaaS systems</li><li>Deploying restrictive cloud security policies and tightened network controls</li><li>Ongoing open-box penetration testing by independent security researchers</li><li>Implementing 24/7 managed detection and response</li></ul></li></ul>
<h2>Looking forward</h2>
Mercor has taken many steps to further strengthen our systems, including implementing more safeguards, expanding security protections, and enhancing our monitoring processes. We will continue to invest in our team and systems to ensure we are a trusted partner to experts and customers. We appreciate the patience, support, and trust of our community. 
We believe as AI gets more powerful, human expertise gets more valuable. We&#39;re building the company that makes that true and creates real economic value for people. We look forward to continuing that focus with them and building a generational company focused on truly consequential work.

Official update on Mercor’s March 2026 data breach: a LiteLLM supply chain attack, what our investigation found, who was affected, and security changes.

Mercor Data Breach: Investigation Findings and Updates

Update on Mercor security incident

In late July 2025, Mercor engineering took on an impossible mission: rewrite a critical, bottlenecked service called Contracts – in one week, with zero regressions in production. Internally, we called it John Wick.
I&#39;ve shipped hundreds of projects across my career. Not once – has any shipped with zero regressions in production (even after dedicated bug bashes &amp; QA). Rewriting an entire service in one week is close to impossible on its own. Stacking both felt like a dare.
We took it.
The resulting system was more than 10,000x more capable. Reliability improved by over 75x. Today it&#39;s handling workloads that would have collapsed the old system entirely, and is more reliable than our vendors&#39; own SLAs.
<blockquote>If impossible missions inspire you, explore our open roles on our engineering, product and design teams: <a href="https://mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">mercor.com/careers</a></blockquote>
<h2>Here&#39;s how we did it</h2>
I joined Mercor because of unprecedented growth, and speed and the level of challenge on the interview process was a clear indication of my predisposition. Mercor is on the <a href="https://www.forbes.com/lists/ai50/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Forbes AI 50</a> for the second year in a row. It is an annual list of the leading private AI companies, produced with Sequoia and Meritech.
I started on the Payments team, working on critical features and correctness problems like idempotency. In late July, the VP of Engineering pulled me aside: The Contracts service was breaking, and they needed help.
I joined the team. After a day of analyzing the rising volumes and P90 metrics – I emailed our CTO and VP of Engineering. 
Before August, the system handled around 3,000 active contracts a month, a scale it had been tuned for. 20 to 50 concurrent requests, each completing on the order of 100 seconds.
In early August, volume started climbing. Fast. Ops team members would extend or update an offer, wait five minutes, and watch the frontend time out. They had no idea if the operation had succeeded. Some would refresh and retry while the backend was still mid-operation. Partial successes stacked up. Nobody knew what state the system was in. Partial updates manifested into KTLO tickets.
Underneath all of it: most operations were making calls to external systems running at two nines of reliability, with aggressive rate limits and no proper retry or fallback logic in place. Every failure mode that had been hiding at low volume was now fully visible.
<h2>Gift of scale</h2>
Contracts is the system every contractor on Mercor flows through. Every hire, extension, update, transfer, offer acceptance, and onboarding goes through it. So does everything that defines the relationship between a company and a contractor – payment terms, permissions and access, eligibility, etc. When Contracts service works, nobody notices. When it doesn&#39;t, a contractor doesn&#39;t know if they&#39;ve been hired and a company can&#39;t tell if an update went through.
The original system had handled the first 10x — around 3,000 active contracts a month, 20-50 concurrent operations, each on the order of tens to hundreds of seconds. The next wave of growth was a different problem.
But between July through October, contract volume grew roughly another 10x. Total system operations grew more than 25x. The failure modes that had been latent at low volume became continuous at high volume: timed-out frontends, retries on operations that may or may not have already succeeded, partial updates that the database and the UI disagreed about.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/75535b7f6e3c6e47397dd9eb96ee06511fda490b-1010x490.svg" alt="Active contracts over time" style="width: 100%; border-radius: 10px;" />
<h2>The challenge</h2>
Scale reveals everything. Every assumption baked into the original design becomes a liability the moment volume climbs past what the system was built for. Contracts had several.
<h3>1. Complexity you can&#39;t see until it breaks</h3>
A single hire request was synchronously touching: company data, IAM permissions, referral checks, payment readiness, onboarding steps, background check status, and multiple external vendors.
At 20 concurrent operations, this was fine. At 100+, every downstream system was getting hammered simultaneously. Different vendors would hit rate limits, timeouts, and surface low-occurring bugs in their systems. MongoDB connection pool exhaustion. All failing at different moments, making analysis nearly impossible. The system failed in a different way every time making maintenance very difficult.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/fdd154c5caf9497392a2a7694da248571279219b-2560x1382.png" alt="Single hire request" style="width: 100%; border-radius: 10px;" />
<h3>2. Optimistic responses are a hidden time bomb</h3>
A dangerous pattern in the codebase was code that assumed success. A bulk update function would touch 30 records sequentially, assume all succeeded, and return a 200. If 3 of 30 failed, nobody knew. The frontend showed success. The database disagreed. Downstream systems acted on a state that didn&#39;t exist.
At low volume, you catch this in support tickets and fix it by hand. At scale, optimistic responses become a distributed consistency problem. There’s no way to understand what actually happened.
<h3>3. External dependencies need reliability wrappers before you need them</h3>
Every Contracts operation – hire, transfer, dismissal, update – made calls to external services: Time-tracking providers. Communication platforms. Payment processors. These calls had no retry logic, no circuit breakers, no graceful degradation. Rate limits were aggressive. And HTTP status codes were not always correct.
At 50 operations a day, this is tolerable. At 500+, you see every failure mode every vendor has, plus the rare bugs in your own system that low volume had been hiding. Scale exposes the long tail — the more operations you run, the more often you hit each rare failure (failure rate × volume).
<h3>4. Engineering toil becomes the product</h3>
When failures cascade and correctness can&#39;t be guaranteed by the system, engineers become the reliability layer. Manually fixing database state. Correcting vendor API calls. Triaging tickets from Ops who don&#39;t know whether their operation succeeded.
At some point, more than 60% of engineering time was going to manual triage and maintenance, not engineering. Meanwhile, the business couldn&#39;t stop – new features were still being demanded, timelines hadn&#39;t moved. We needed to rebuild a critical system for our new scale while keeping the existing one running.
<h3>5. You can&#39;t fix what you can&#39;t see</h3>
The growth had outpaced the technology, and the technology had outpaced any documentation. One engineer on the team carried deep context from the original build; while Contracts was being rewritten, he kept critical business features shipping in parallel so the rest of the company didn&#39;t slow down.
The rest was reverse engineering. We had to identify the critical paths, address observability, and reconstruct the tribal knowledge behind the code. There was no structured visibility into which operations were in-flight, which had partially succeeded, or which had failed silently — meaning every triage started from scratch. Before we could fix integrity, we had to be able to measure it.
Other constraints (an in-flight Kafka migration, no Temporal, the frontend API contract frozen), ruled out the obvious options and forced a leaner design.
<h2>Paving the way</h2>
<h3>The new system</h3>
We built this in three layers: API → validation → database. Core reads &amp; writes are O(1) on average. Every side effect is async—the hire endpoint returns fast, and downstream work runs in the background through a uniform execution framework with built-in retries, idempotency, and observability.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/ace133b37fa73097e6013f8af9dbc34ce36e2bf5-2560x928.png" alt="System architecture" style="width: 100%; border-radius: 10px;" />
<h3>EventExecution: every operation has one declared intent</h3>
Every async handler in the new system inherits from a single base class with exactly two methods.
should_execute() is where idempotency lives. Before anything runs, the handler checks whether it already happened. A retry after a timeout, a duplicate event from a queue, a webhook redelivered — all of them hit should_execute() first and skip cleanly. No double-fired hire emails. No duplicate payment events.
execute() is the business logic. It returns a typed result — ok, fail, or skip — with optional retry scheduling.
The discipline this enforces is the point. Every operation has a single, declared intent. No side effects hidden in the call stack. You can look at any handler and know exactly what it does, when it runs, what it returns, and how it retries.
<h3>Observability, by default</h3>
The typical pattern with observability: write the business logic, bolt on logging later, add metrics after that, and six months later you still can&#39;t tell why a specific operation failed last Tuesday.
The framework emits observability by default. Because every handler inherits from the same base class, instrumentation is structural. Every operation reports its own outcome, timing, and retry history automatically, and the framework pulls the identifying context (which job, which contractor, which company) into every log line on its own.
Every record tells you which operation failed, on which attempt, with what payload. The handler author writes none of it.
From the first day the new system handled live traffic, we had full visibility. Engineers and Ops could see the state of every contract operation in real time. We built an internal dashboard before the first endpoint went live.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/cd7a42994bba6bd5ff881c9ad599b8136f96588b-1304x680.png" alt="" style="width: 100%; border-radius: 10px;" />
<img src="https://cdn.sanity.io/images/h6s14f4z/production/3a3ed9ccd5d93a6c306bb16381b30dd82cee64f9-1390x546.png" alt="" style="width: 100%; border-radius: 10px;" />
<h3>Boring by design</h3>
The architecture was deliberately boring. We kept the database schema frozen. We kept the frontend API contract frozen. Changing either would have required coordinated migrations we didn&#39;t have time for, and constraining scope to the application layer eliminated entire categories of risk.
Single-responsibility functions. Flat call stacks. Files aimed at 200 lines. Low cyclomatic complexity. Nothing clever. Readability =&gt; Maintainability.
<h3>How we got there</h3>
Analyze first, code second. The first two days we wrote zero product code. We analyzed every primary endpoint to identify common patterns and shared logic, catalogued every critical and non-critical issue, and mapped every external dependency — reasoning through exactly what happened when each one went down. That work defined the architecture: what needed to be rewritten, what could be preserved, and where the real risk lived.
Ship behind a fallback. We didn&#39;t delete the old code. We built a route_to_api decorator on each original endpoint that, when a feature flag was enabled per company, per user, or per project, transparently routed to the new implementation. If the new system threw, it fell back to the old one and logged the failure. One flag flip to revert; no deployment required. The fallback is what gave us the confidence to ship at all.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/1c5b2c5c4303ecbd1338bcd0ad4873ded3f45256-1284x462.png" alt="" style="width: 100%; border-radius: 10px;" />
Parity before confidence. Before cutting traffic, we built parity tooling that replayed requests through both paths and compared outputs at three levels: API responses, database state, and vendor call behavior. Does what the system persists match what the API returns? Does what the API returns match what the vendor received? Parity at all three is the only honest definition of correctness for a system this interconnected. No-op dry run capability is underrated.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/24bcbffe532602243217cf1b9df6b2787dcb4a88-1284x462.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>The result</h2>
Over the rewrite window, we built the foundation and ported the most critical, highest-volume endpoints onto it. Together they carry the bulk of Contracts&#39; traffic – the paths that were causing failures at scale.
EventExecution was as much a template as a framework. Once the pattern was in place, the rest of the system had a clear path forward: every async operation declares its intent, runs idempotently, retries on failure, and emits its own observability – same shape every time.
The boundary was the point. Inside the scope of the rewrite – the highest-volume, highest-risk paths – we held the bar at bug-free, and hit it. Outside that operability boundary, the known issues stayed known issues, on a list we&#39;d come back to.
<h3>Scale: more than 10,000x</h3>
The old system could handle 20 to 50 concurrent operations, each completing on the order of 100 seconds. The new system processes async operations that return in milliseconds — the heavy work happens in the background, reliably, with automatic instrumentation and observability. Measured in requests handled per second, the capability improvement exceeds 10,000x.
Volume grew dramatically. The system absorbed it without incident. Reliability: more than 75x in the first month and more so since then.
In the first month of full event tracking, the system processed millions of async operations with a failure rate under 0.4%, most days recording zero failures. That&#39;s better than the uptime guarantees most of our external vendors provide. Measured as the old error/failure rate divided by the new one yields an improvement of more than 75x.
Ops stopped refreshing and retrying. The partial success problem went away. The unknown-state problem, which caused most of the KTLO burden, was gone.
<h3>Engineering time reclaimed</h3>
At peak KTLO, more than 60% of engineering time was going to manual reliability work for a broken system. That number is now close to zero.
<h3>How the system is doing today</h3>
<table class="table-cell-border" style="width: 100%; border-collapse: collapse; margin: 1em 0;"><thead style="text-align: left;"><tr><th>Metric</th><th>Before</th><th>After</th></tr></thead><tbody style="text-align: left;"><tr><td>System capability</td><td>20–50 concurrent ops ~100s each</td><td>&gt;10,000x improvement (based on extrapolated real load and test-load) </td></tr><tr><td>Contract volume growth</td><td>Baseline</td><td>&gt;9x in under a month (Today: ~20x)</td></tr><tr><td>Operations volume growth</td><td>Baseline</td><td>&gt;25x within 2 months</td></tr><tr><td>Reliability</td><td>Cascading failures, unknown state</td><td>&gt;75x improvement (&lt;0.4% failure rate)</td></tr><tr><td>Engineering KTLO</td><td>&gt;60% of eng time</td><td>~0%</td></tr><tr><td>Response time (new ops)</td><td>Minutes (synchronous)</td><td>Order of 100 ms (async)</td></tr><tr><td>Peak operation load</td><td>Baseline</td><td>&gt;200x peak to peak operational load </td></tr></tbody></table>
<h2>What’s next</h2>
We’re building a world class technology and engineering org that can scale with Mercor&#39;s growth. 
<h3>Reliability and integrity</h3>
Reliability at 75x improvement is a milestone, not a destination. The next phase is pushing toward the kind of system integrity that is provable. Invariant definitions and assertions at service boundaries. Auditability built into the system from the start. Every state transition is traceable, every outcome explainable.
<h3>Product as platform</h3>
The EventExecution model already makes it straightforward to add new consumers for new operations, a new side effect is its own independent class, cleanly separated from everything else. The next step is taking that further: turning Contracts from a service into a platform that a product can build on top of, without pulling in an engineer for every change. Service level boundaries defined clearly enough that new product surfaces don&#39;t create new reliability risks. Extensibility that grows with the business instead of creating new complexity.
<h3>Engineering that enables, not constrains</h3>
The goal is zero KTLO. A system that is stable and well-instrumented that engineers spend their time building, not maintaining. The kind of codebase where adding a feature feels like writing a paragraph. Contracts is upstream of everything — integral to Payments, the source of truth for who to pay, how much, and when. Today, Mercor <a href="https://www.mercor.com/blog/when-you-go-from-2-million-a-month-to-2-million-a-day/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">processes more than $3M a day</a> in contractor payments, and none of it moves correctly without Contracts working. We&#39;re building the foundation to handle whatever scale comes next.
<h2>Come build with us</h2>
Mercor is growing faster than most companies ever will, and the systems that support that growth have to be built by people who care about getting it right. If you want real scope, real problems, and the chance to redefine what&#39;s possible – we&#39;d like to talk: <a href="https://mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">mercor.com/careers</a>

Mercor engineering rebuilt a critical production system under 10x volume growth, in one week, with zero regressions. Join our engineering team.

Rebuilding a Critical Service in One Week | Mercor Engineering

When volume grew 10x in a month and we had one week to fix it

Mercor is on the <a href="https://www.forbes.com/lists/ai50/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Forbes AI 50</a> for the second year in a row. It is an annual list of the leading private AI companies, produced with Sequoia and Meritech. We went from $0 to $500M run rate in 17 months, crossed $1B in annualized revenue run rate earlier this year, and we pay out over $2 million every day to more than 30,000 weekly active contractors across the world.
Growth like this is a privilege and a stress test. Moving from $200K to over $14M a week on the platform in a short timeframe exposed every place where our early-stage processes, data models, and controls hadn&#39;t kept up with our scale. Some of those gaps we closed on our own terms. Some of them we closed after they bit us. We’ll talk about both kinds of lessons.
The job from here is to build through the next 50x. This is more than just an engineering challenge; it is fundamentally a human one: there are not many engineers in the world who have built systems that move billions safely at this kind of growth rate, and we need more of them. What follows is the record of what we&#39;ve built and where we&#39;re heading. 

If that below sounds like your kind of work, we&#39;d like to talk.
View the role at <a href="https://www.mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">mercor.com/careers</a> 
<h1>What Mercor&#39;s Payments System actually does</h1>
Mercor is a two-sided marketplace at the forefront of AI; we bill leading AI labs and enterprise clients for access to the best global talent, and we pay independent contractors everywhere for their expertise and work.
Our payments infrastructure has to do two things, simultaneously: invoice clients across multiple, complex billing structures and contractual terms while paying out thousands of experts across dozens of countries and currencies. We have to get this right, all the time, every time. Each of those transactions touches a chain of systems that all have to agree: contracts, work tracking, billing logic, payout orchestration, ledgering, and reconciliation. When any layer in that chain disagrees, someone doesn&#39;t get paid. At $2M a day, it&#39;s a very real existential and reputational risk to the entire company.
It&#39;s worth pausing on what &quot;someone doesn&#39;t get paid&quot; actually means. These aren&#39;t anonymous transactions. They&#39;re a software engineer in Atlanta who depends on Wednesday&#39;s payout to cover rent. A contractor in São Paulo who has no visibility into why their earnings haven&#39;t cleared. If we fail a payment, we&#39;re deeply affecting someone&#39;s livelihood. That weight is something the payments team at Mercor carries deliberately, and it&#39;s a big part of why we treat reliability, auditability, and transparency as first-class obligations rather than as things we&#39;ll get to later.
The team building this sits at an unusual intersection in the company: the breakneck pace, intensity, and ownership culture of a hyper-growth AI startup, with the transaction volume, &quot;always correct,&quot; and compliance responsibilities of a scaled fintech. Those two cultures don&#39;t naturally coexist, and building a team that holds both is part of the work. Our payments and financial infrastructure group brings experience from Scale AI, Google, Airbnb, Robinhood, Two Sigma, Doordash, and Amazon: people who have built and scaled financial systems through exponential growth before. They are people who joined Mercor precisely because the problems here are hard in interesting ways, and because getting them right matters.
<h1>Five things we learned building payments at speed</h1>
<h2>1. A ledger is a first-class primitive, not a side effect</h2>
When you&#39;re moving fast, it&#39;s tempting to treat your ledger as a byproduct of your service: a table that simply records what happened. That framing creates a ceiling you don&#39;t notice until you&#39;re already past it.
A proper ledger must be a resilient, independently searchable source of truth, and natively integrated across your end-to-end operations. When it isn’t – when billing logic, payout state and revenue recognition are all entangled in a single mutable data structure, the system works fine at low volume and starts to actively resist you at scale. Changes become high-risk. Debugging requires holding the whole system in your head. Finance can&#39;t close cleanly at period end.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f1c0af18eb0ea016ec673505d29ccd17c111feb5-1600x900.png" alt="payments-1" style="width: 100%; border-radius: 10px;" />
The architectural shift we&#39;re making at Mercor is a move from mutable, monolithic payment records to an immutable, append-only ledger: every financial event is written once and never updated with a canonical state machine (accrued → approved → payable → paid → reversed) that makes the current state of any payment, readable and reconcilable at any point in time. This is the foundation for accurate revenue recognition, clean ERP integration, and the kind of auditability that a Series C company with our transaction volume requires.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7d1dd284844deef6792c2731b04cb4efc50b7162-1600x1198.png" alt="payments-2" style="width: 100%; border-radius: 10px;" />
The ledger also connects downstream to every system that needs to read financial state: revenue reporting, data analytics treasury forecasting, expert earnings visibility, and the audit trail. Building it right means all of those systems get reliable data without each team maintaining their own bespoke view of financial truth.
<h2>2. Manual processes are risk surface, not just tech debt</h2>
Early-stage teams make a rational bet when they operate something manually: it&#39;s faster to ship, and automation can come later. That bet has a time limit, and the clock moves way faster than you expect.
What makes it insidious is what happens in the intervening period. Engineers build on top of the manual process, designing around it rather than replacing it. Operations start treating it as normal. Workarounds accumulate. Runbooks get longer. New engineers onboard and inherit the process as if it were intentional.
One of the most instructive examples is payout dispatch. Weekly payout dispatch started as a reasonable script that one engineer could run in a few minutes. Over time it grew into a multi-hour orchestration involving manual SQL validation, cross-referencing forecasting numbers across multiple payment providers, treasury balance confirmation, and a group sign-off ritual before anything moved. At millions of dollars a week, the process was absorbing 10+ engineering hours per week - time that could be spent scaling and automating systems. Correctness depended entirely on the specific person running it and the conditions under which they ran it. Nobody intentionally designed it that way. It just happened.
The solution isn&#39;t just to automate for efficiency&#39;s sake, but to make the system&#39;s behavior deterministic. This is the goal of the fully automated payout dispatch system we&#39;re shipping. Zero human operational oversight required for the weekly cycle, knowing exactly who, when, and how much to pay, with intelligent monitoring and alerting that surfaces anomalies proactively rather than relying on human spot-checks.
Every manual step is a potential failure mode, and the longed you wait, the more it costs. Every month a manual process runs in production, another layer of complexity grows on top of it. The time to replace it is always earlier than it feels. The sooner you treat manual processes as temporary scaffolding rather than permanent infrastructure, the sooner you can build the safety systems that let you move faster, not slower.
<h2>3. Automation requires canonical data</h2>
A critical, yet often underestimated, principle in payments engineering is this: automation doesn&#39;t lower risk if the underlying data is unreliable; it merely amplifies existing risks. A manual process running on unreliable data might generate a few errors weekly, but an automated one will produce thousands.
At Mercor, task-based payments proceed through a complex series of systems: contracts, work tracking, ingestion, pricing, ledgering, and invoicing. This process only functions smoothly if the output from each stage is understood by the next. Our core challenge stemmed from that fact that upstream data sources were never designed to be financially authoritative. Task statuses were vague, transitions were non-deterministic, and some crucial data resided completely off-platform.
This data deficiency can manifest in two ways:
<ul><li>Inaccurate Bonus Payouts: When a contractor completes a task, the project lead records completion in a manual, off-platform spreadsheet and must manually issue the corresponding bonus. This can lead to forgotten bonuses or mistyped amounts. Crucially, the source of truth for a contractor&#39;s earnings becomes a hand-edited cell in a spreadsheet.</li><li>Inefficient Invoicing and Reporting: Before every billing cycle, our accounting team spends a significant time manually verifying payment amounts by cross-referencing disparate upstream tables. This data might be scattered across different tools and data sources. The necessary solution is to establish a single, canonical definition of &quot;task complete&quot; that every tool adheres to and every downstream system trusts.</li></ul>
We treat upstream data architecture as a first-class citizen of our payments infrastructure, rather than agnostic input. This requires establishing and enforcing strict, canonical work completion signals across the platform: deterministic triggers that every tool must adhere to and every downstream system can trust without manual validation. From there, the end-to-end chain of work tracking, ingestion, pricing, ledgering, and invoicing runs seamlessly and automatically.
<h2>4. Observability investment compounds early</h2>
One of the best decisions the team made early was to invest in continuous monitoring, alerting, and payments tracking.
Real-time payout status, unified earnings views, and forward payment forecasting aren&#39;t the most glamorous infrastructure work (and most of the company doesn&#39;t know it&#39;s there unless it breaks), but they have a measurable impact on the thing that matters most in a payments system: trust. Experts depend on Mercor to provide reliable cashflow and want to make sure they get paid for their hard work. We invested early in automated forecasting and reliable payment status data, and surfaced this to our experts via a redesigned Earnings dashboard. Today, Earnings is the third most visited page on the Mercor Work Platform. Since launching these features, payout-related support tickets have dropped significantly.
On the operational side, we built centralized monitoring across our payment providers, tracking payout success rates, provider-specific failure modes, treasury balances, and reconciliation gaps, with structured alerting that catches anomalies before they become incidents. When something goes wrong, the team can trace it in minutes rather than hours.
Building observability into the system early and treating it as core infrastructure rather than a nice-to-have is what gave the team the feedback loops needed to move faster as complexity grew.
<h2>5. Controls aren&#39;t a compliance checkbox. They&#39;re load-bearing infrastructure.</h2>
At Mercor, we&#39;re building toward a world where many of our operations can run autonomously, where agents can operate projects, dispatch payouts, generate invoices, and close the books.
This comes at a tradeoff. Manual processes have one advantage: a human in the loop. The operator entering a discretionary bonus catches a typo before it ships. The engineer running the weekly payment dispatch can notice if the numbers look off. When you automate, these human controls go away, so the controls that replace them have to be well-designed, because there&#39;s no one left to catch what the system misses.
We&#39;ve invested in applying agentic controls across our billing orchestration. We implemented a two-phase commit pattern across payment-triggering actions: every action first creates a pending record with a complete audit event, including the agentic initiator, amount, billing account, reason, timestamp. The audit trail is written before money moves, never as a side effect after.
On top of that, we built a threshold-based approval routing engine: configurable spend limits with automatic escalation requiring approval above defined thresholds. Large payments get flagged proactively and require explicit sign-off before they execute. We also invested in an aggregation service that monitors total spend and bonus activity across the platform in real time, so the system itself knows when something is trending toward an outlier before anyone has to ask.
The most effective controls are structural ones. By creating a system where it is architecturally impossible to issue a payment without a threshold check or an audit record, we enable automation to scale safely. Because proactive safeguards are built directly into the core platform, agentic functions can manage tasks like orchestration, dispatch, and bonus approvals while remaining strictly governed.
<h1>What we&#39;ve shipped</h1>
Since October, the team has shipped:
<ul><li>Automated treasury forecasting: Automated forecasting of our float eliminated overdraft risk and shifted treasury operations away from engineering, establishing a clean separation between financial operations and product development.</li><li>Expert earnings visibility: Real-time payout status, unified earnings views, and forward payment projections, now the third most visited page on the Work Platform, with a measurable reduction in &quot;where is my money&quot; support tickets.</li><li>Billing canonicalization: Billing Accounts introduced as the primary financial primitive, linking projects and contracts to financials and establishing Finance as the owner of revenue recognition.</li><li>Centralized observability: Real-time KPI dashboards, structured alerting across payout providers, and incident runbooks that have significantly reduced production incidents and mean time to resolution.</li><li>100% bonus auditability: Immutable audit records on all on-platform bonus activity. Every payment now carries initiator attribution, billing account, reason code, and a complete approval history. Two-phase commit architecture ensures the audit trail is written before money moves, not after.</li><li>Time agnostic billing: Migrated billable data to UTC, eliminating a class of timezone-related billing errors that had real cost implications at scale.</li></ul>
These represent a shift from a payments system that was outrunning its infrastructure to one with the controls, visibility, and data quality needed to scale further.
<h1>What we&#39;re building</h1>
The foundation is almost in place. The next phase is where it gets interesting.
Correctness as a primitive: Shipping ledger-backed payouts, where all money movement is written and executed from a universal, immutable, append-only ledger with absolute correctness. That means zero double payouts and a payout failure rate below 0.01%.
Canonical payment state machines: Decomposing the current tightly coupled orchestration into a state-based billing model that cleanly separates what is owed from how money moves. This is the unlock for faster development, safer launches, and future capabilities including multi-instrument wallets, accruals, and dynamic pricing.
Completely autonomous payout dispatch: Fully automated weekly payout execution with zero human operational oversight required. The target is 95%+ of payouts executed at or ahead of SLA, with zero orchestration regressions.
Human data tasks as a native billing construct: Native task-based payouts and invoicing based on canonical task completion signals, targeting automation of 90% of task-based payment activity.
Budgets and approvals: Threshold-based approval routing for all non-billable spend, with configurable per-expert and per-account thresholds, automated escalation tiers, pattern-based fraud detection, and real-time spend dashboards for Finance and account owners.
Plug-and-play and self-service: Shifting payment operations away from a reactive model by enabling Finance, Ops teams and AI agents to proactively manage payouts, fixups, and disputes through self-service tooling with end-to-end audit trails and role-based permissions.
The scale is already exceptional. The engineering to match it is what comes next.
<h1>Come build with us</h1>
Mercor is growing faster than most companies ever will, and the payments system is where a lot of that growth has to be absorbed. Some of what we&#39;ve built, we built because we saw it coming. Some of it we built because we had to. All of it is load-bearing, and all of it needs to get better.
We&#39;re building a team of exceptional engineers who know what scale looks like and will build the financial backbone that supports the next phase of growth. This includes automating payout orchestration, building an immutable operational ledger, designing settlement and reconciliation pipelines, improving end-to-end observability across payout to processor to bank, hardening controls and fraud detection across the marketplace, and decoupling processor dependencies from core infrastructure.
This is a place for people who care deeply and build boldly, who want the pace and ownership of a hyper-growth startup with the rigor of a fintech, and who understand that at our volume, those two things are the same job. You&#39;ll work in-person with a team of ambitious, diverse people on problems that matter, with direct exposure to the research, labs, and startups at the AI frontier. The problems are genuinely hard, the stakes are genuinely real, and the people working on them are genuinely exceptional.
If that sounds like your kind of work, we&#39;d like to talk.
<a href="https://www.mercor.com/careers/?ashby_jid=3c02ed8c-3807-4aef-91cc-ccca5ab578bd" target="_blank" rel="noopener noreferrer" class="text-indigo-600">View the role at mercor.com/careers →</a>
Mercor is a profitable Series C company valued at $10 billion, trusted by 6 of 7 Mag 7 companies. We work in-person five days a week in San Francisco.

Derek Shimozawa

When you go from $2 million a month to $2 million a day

In a<a href="https://www.mercor.com/blog/scaling-data-apex-agents/" target="_blank" rel="noopener noreferrer" class="text-indigo-600"> previous post</a> we post-trained GLM-4.7 on an agentic dev set to create AC-Small, a model that rose from 17th to 4th on the <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents leaderboard</a>. The obvious remaining question: did the model just learn to improve on the held-out benchmark or did its capabilities actually improve?
To find out, we tested AC-Small on three held-out industry benchmarks: <a href="https://arxiv.org/abs/2510.25726" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Toolathalon</a>, which evaluates agents on multi-step workflows spanning more than 600 tools; <a href="https://arxiv.org/abs/2509.25721" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX</a>, which evaluates non-agentic professional reasoning across four domains; and <a href="https://arxiv.org/abs/2510.04374" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GDPva</a>l, the closest end-to-end analog to APEX-Agents. 
Performance improved substantially across the board: +5.7 points on APEX, +8.0 points on Toolathalon, and +7.7 percentage points on GDPval.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/d357f447b9b4ca5c72dff24e5f0c713fd91e7838-2525x1444.png" alt="Table 1: General benchmarks stay flat, while held-out tool-use and professional benchmarks improve" style="width: 100%; border-radius: 10px;" />
Table 1: General benchmarks stay flat, while held-out tool-use and professional benchmarks improve. HLE and GPQA are reported as Avg@1 and Avg@3 respectively. APEX intervals are derived from 8 runs per task. GDPVal is based on a single run and placement is based on the official <a href="https://evals.openai.com/gdpval/leaderboard" target="_blank" rel="noopener noreferrer" class="text-indigo-600">leaderboard</a>; Toolathalon result is based on 3 runs.
<h2>Results</h2>
<h3>GDPVal: Improved performance on economically valuable work</h3>
GDPVal measures professional work across 44 occupations and 9 sectors, judged by subject-matter experts. It is the closest held-out analog to APEX-Agents: an end-to-end test of whether the gains from training on APEX-Agents data generalized to out-of-distribution agentic tasks.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/5c718675875242cfd8a430f7671f16f7c8cf2d60-2200x2048.png" alt="Figure 1: On GDPVal, AC-Small rises from 7th to 5th, surpassing Opus 4.5" style="width: 100%; border-radius: 10px;" />
Figure 1: On GDPVal, AC-Small rises from 7th to 5th, surpassing Opus 4.5
We saw AC-Small’s win+tie rate rise from 55.0% to 62.7% (+7.7pp). If placed on OpenAI’s official GDPVal leaderboard alongside frontier models, AC-Small would rank 5th, behind GPT-5.4 (83.0%), GPT-5.4 Pro (82.0%), GPT-5.2 Pro (74.1%), and GPT-5.2 (70.9%), and ahead of Claude Opus 4.5 (59.6%).
<h2>Where the GDPVal gain comes from</h2>
<h3>Better tool use across the board</h3>
Toolathalon evaluates language agents on multi-step workflows across 32 software applications and 604 tools, requiring coordination across platforms over an average of 20 tool-calling turns. As an ablation it isolates tool-use fluency as a capability. AC-Small scores 34.6% versus GLM-4.7’s 26.5% (+8.0pp), lifting the model four spots on the leaderboard and past Claude-4-Sonnet, Kimi-K2.5, and Grok-4. 
<h3>Better professional reasoning, even without tools</h3>
APEX tests whether models can complete economically valuable work in management consulting, investment banking, law, and primary healthcare when they are given the exact files they need and no tool interaction is required. On this benchmark, AC-Small rises from 52.5% to 58.2% (+5.7pp), moving from 24th to 17th on the leaderboard.
The strongest single-domain effect is Medicine. AC-Small places 4th with a score of 64.8%, behind only Opus 4.6 Max (70.4%), Opus 4.6 High (70.0%), and Sonnet 4.6 (66.7%). That is notable because the training set solely covered tasks in law, consulting, and corporate finance. 
<img src="https://cdn.sanity.io/images/h6s14f4z/production/a923c2a14e5b728bdeb4a2e4b25770313ebe4817-2055x916.png" alt="Figure 2: APEX gains concentrate in medicine and law, with smaller movement in consulting and a slight regression in finance that falls within the original confidence interval" style="width: 100%; border-radius: 10px;" />
Figure 2: APEX gains concentrate in medicine and law, with smaller movement in consulting and a slight regression in finance that falls within the original confidence interval.
We dug deeper into how training on law, consulting, and banking could improve performance on medical tasks and found that the gains came from improved processes. For example, for one task in the medical eval set which asked for a diagnostic writeup, both models identified the right categories of tests, but GLM-4.7 dropped clinically significant details in its final answer. 4.7 omitted some veins that should be scanned, recommended colonoscopy without specifying biopsies, and listed only one source of false-negative troponin results. AC-Small preserved those sub-details, finishing the task with a perfect score.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/8a689e1e3a72cd13716d9ad898222d779980b8d9-1782x569.png" alt="" style="width: 100%; border-radius: 10px;" />
Figure 3: An example task from APEX where AC-Small outscored GLM 4.7
<h2>What the transfer pattern suggests</h2>
Taken together, these results suggest AC-Small gained capabilities that generalize beyond the dev set it was trained on. Toolathalon shows improved tool use. APEX shows better professional reasoning even without tools. GDPVal shows those gains translate to higher-quality end-to-end work as judged by a human-preference benchmark.
Beyond specific benchmark gains, AC-Small seemed to pick up some of the habits that lead to quality work output: preserving details, sanity checking intermediate outputs, and revising when those checks fail. These learnings allowed AC-Small to substantially improve on APEX medicine, a non-agentic domain that wasn&#39;t in the dev set it was post-trained on. This suggests that as expert data scales across the economy, models will become more useful not just because they know more, but because they work more like professionals.

How 2,000 expert tasks improved tool use and professional reasoning

Generalization Results from APEX-Agents Dev Set | Mercor Research

Generalization results from training on the APEX-Agents dev set

At Mercor, we’ve spent years on the frontier working with AI labs to accelerate model capabilities through human expertise. We see firsthand how raw intelligence compounds daily. However, enterprises still haven’t seen that same compounding value in their own AI deployments. What’s the bottleneck?
<h2>Stop guessing about your agents</h2>
Most enterprise agent deployments follow the same pattern: a team guesses at what the agent should do, hand-writes prompts, manually configures tool calls, and hopes it performs as intended. When it doesn&#39;t, they guess again. And again. This is why enterprise AI stalls.
Not because the models aren&#39;t smart enough, but because your agent development approach is backwards.
Here&#39;s the guesswork loop most enterprises are stuck in:
They guess where to start. Companies don’t know which agents to deploy because no one has the full picture. The teams closest to the work know which tasks are tedious, but only see their slice of a cross-functional, multi-step web. The people who own the function see the end-to-end process, but are a step removed from where real friction lives. So someone picks a use case that sounds promising, and the team runs with it, without evidence of where agents will actually create value.
They speculate on how agents should behave. The agents they do build rely on improvisation: hand-tuned prompts, ad hoc tool selection, and configurations based on what one person thinks the workflow should look like — not how the highest performers actually do the work.
They assume agents work as intended. Errors compound because there&#39;s no clear way to tell whether an agent&#39;s output is acceptable, good, or quietly degrading. The bar becomes &quot;it seems to work,&quot; but teams don’t know what’s failing or what could be better. They discover failures weeks later through customer complaints or downstream breakdowns, and spend more time repairing damage than the agent saved.
They improve agents on vibes, not verification. There&#39;s no systematic way to improve agents over time. Updating workflows means painstaking manual reviews, rewriting prompts from scratch, and repeating the whole cycle.
If you&#39;re guessing at any of these steps, you&#39;re guessing about your agents. No amount of model intelligence closes that gap on its own.
<h2>Introducing the Mercor Enterprise AI Platform: Groundwork for your agents</h2>
Breaking the guesswork loop requires the opposite approach. You start by understanding how your work actually gets done. Then, you programmatically convert that understanding into agent behavior, and continuously measure output against quality. The gap between the output and the desired quality becomes the feedback signal that improves the agent over time. Together, these are the groundwork that separates agents that compound in value from agents that stall.
The<a href="https://www.mercor.com/enterprise/" target="_blank" rel="noopener noreferrer" class="text-indigo-600"> Mercor Enterprise AI platform </a>delivers these three core capabilities:
<h3>1. Know where and what to build: understand organizational context</h3>
Solid groundwork starts with evidence. We map your workflows by understanding how your people actually work — screen-level workflow capture, extraction from internal wikis and application system logs, and AI-led employee interviews that surface the institutional knowledge living in people&#39;s heads. The judgment calls, the edge-case handling, the &quot;we do it this way because of X&quot; reasoning that never makes it into documentation.
We capture all of it as structured, machine-readable data, so you know exactly where agents will create value and what they need to do it well.
<h3>2. Teach agents how to behave: programmatically translate context into agent behavior</h3>
Understanding context is only valuable if you can systematically convert it into agent behavior. We programmatically translate organizational context into agent behavior specs and quality guardrails. Instead of hand-tuning prompts, we generate a robust set of evaluations and success criteria from how your top performers work, and then automatically optimize agent behavior against them. Agents are deployed in hours rather than days, and at a higher baseline quality than manual specification can achieve.
<h3>3. Learn continuously: measure agent output against defined standards</h3>
Agents without a quality signal fail silently, and fixing them is manual. We catch errors in real time using quality verifiers that compare every output against your organization&#39;s definition of &quot;good.&quot; When an agent&#39;s confidence is low, we flag it for human review before it causes damage. When corrections come in — from automated detection or human feedback — they feed directly back into the agent&#39;s behavior specs and organizational context.
Every failure becomes a permanent improvement.
This closes the remaining gap: you stop guessing whether agents are working and how to fix them — because quality is measured continuously and corrections are systematic.
<h2>Tools built on frontier AI infrastructure</h2>
This platform evolved from the same technologies we&#39;ve pioneered to help the world’s leading frontier AI labs organize and structure human expertise to teach their models. The tools we use to capture expert behavior, build evaluation criteria, and systematically close performance gaps have been tested at scale across the most advanced AI development pipelines.
Now we apply that same technology to your organization.
We deliver the groundwork through a set of discrete, composable tools designed to scaffold your agents on any orchestration platform:
Organizational context graph: Structured map of how your organization actually works, built from data integrations with your organization’s systems and tools (e.g. ticketing platforms, CRMs, communication tools, codebases, knowledge bases, internal docs), application traces, team workflow capture, and AI-moderated interviews that surface institutional knowledge in people’s heads.
Agent Specification Engine: Human-readable, machine-executable agent specs translated from your context graph. Instructions are derived from how your best people actually work.
Quality Guardrails: Task-output scoring functions that compare agent output against expert-defined quality thresholds and automatically gate outputs that fall below your standards for human review.
Continual Learning Harness: Machine- and human-led feedback loops that automatically update agent behavior based on evals — task objectives, agent behavior, and output quality scoring functions — that pinpoint where and why your agents fail. Every correction makes the system better.
Agentic Workflow Data: Proprietary datasets of completed tasks and reasoning traces across economically valuable domains — law, finance, HR, accounting, software engineering, market research, competitive analysis, and more — so your agents don&#39;t start from scratch.
The modularity means we can deploy the right combination of components for your specific challenge — capturing context for a single workflow, deploying quality guardrails on agents you&#39;ve already built, or running the full pipeline end to end.
<h2>Why Mercor</h2>
Three things set us apart.
We&#39;ve proven this with frontier AI labs. We are a leading partner to frontier labs for organizing and deploying human expertise to improve model performance. The infrastructure we use to capture expert behavior, build evaluation criteria, and systematically close performance gaps is battle-tested at billion-dollar scale. We understand what actually matters for strong agent performance at a deeper level than anyone else — because we&#39;ve been defining and measuring it for years.
We&#39;ve deployed this for ourselves. Mercor runs highly sophisticated, self-improving agents at scale — including AI-driven hiring processes that evaluate millions of candidates for our expert network. This isn&#39;t theoretical. We&#39;ve iterated these systems to a level of quality that our competitors cannot demonstrate.
We understand self-learning. We are the best in the world at teaching models to improve using benchmarks. We’ve spent years working with labs and developing benchmarks like <a href="https://www.mercor.com/blog/introducing-apex-agents/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX</a> to measure whether AI models can perform economically valuable work in law, finance, and software engineering. Through that work, we’ve built technology that pinpoints exactly where and why an agent fails. We deeply understand the common patterns behind agent failures: confusing implications with grounded facts, insufficient output verification, incomplete tool access, and more. That same expertise and technology is the foundation of everything we’ve built. Without measurable, specific corrections, no amount of context helps. Your agent will keep making the same mistakes.
Your best people become coaches to the agents they work with every day. The agents remember and operate by your gold standards, continuously learning from real evidence of what great work looks like.
Build on groundwork, not guesswork.

Introducing the Mercor Enterprise AI Platform. Build AI agents designed for your company. 

Introducing Mercor Enterprise AI

Introducing <a href="https://arxiv.org/pdf/2601.08806" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-SWE</a>, a new benchmark created in collaboration with <a href="https://cognition.ai/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Cognition</a>. It measures whether frontier AI models can handle real software engineering work – shipping systems, diagnosing failures, and implementing fixes.
Our results show that even top performing AI models hit a wall when tasked with the complexities of real-world software engineering. At the time of release, the top score on the APEX-SWE leaderboard is GPT-5.3 Codex at 41.5% Pass@1, leaving plenty of room for hillclimbing.
Cognition was critical to making this benchmark reflect the expectations of software engineers. They reviewed a subset of integration and observability tasks, created by SWEs hired through Mercor’s platform, pressure-testing how real production systems actually break and get fixed. The resulting tasks reflect the expectations agents must meet to be useful in real software engineering work.
APEX-SWE builds on Mercor’s family of benchmarks for evaluating AI models at economically valuable work, including <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a>, the <a href="https://www.mercor.com/apex/apex-v1-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">AI Productivity Index (APEX)</a>, and the<a href="https://www.mercor.com/apex/ace-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600"> AI Consumer Index (ACE)</a>.
<iframe src="https://player.vimeo.com/video/1176599561" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="Introducing APEX-SWE | Mercor x Cognition" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
AI coding models and assistants are now a core part of software engineering, with recent industry reports indicating that over 90% of developers use AI coding assistants. Nearly half of all code at major technology companies is AI-generated.
Traditional coding benchmarks have become saturated. GPT-4 has improved from 67% to 90% on HumanEval in just two years, and the most recent Opus models consistently score over 75% on SWE-bench Verified. OpenAI has declared some <a href="https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">SWE benchmarks contaminated</a>, with models able to reproduce original patches verbatim from task IDs alone.
However, even before saturation, these benchmarks presented a misleading picture of AI models&#39; real-world coding ability. According to <a href="https://my.idc.com/getdoc.jsp?containerId=US53204725" target="_blank" rel="noopener noreferrer" class="text-indigo-600">IDC</a>, developers spend only 16% of their time writing code and building new features. The remaining 84% involves CI/CD, infrastructure monitoring, deployment, and debugging.
Professional software engineering extends far beyond writing short functions or patching a single file. Real production environments involve cross-platform integration, infrastructure provisioning, and debugging production failures with incomplete information.
<h2>Leaderboard results</h2>
All of the models we evaluated fail to reliably solve the real-world production software engineering tasks in APEX-SWE.
GPT-5.3 Codex (High) tops the leaderboard at 41.5%, followed by Opus 4.6 (High) at 40.5% on Pass@1 and Opus 4.6 (High) at 38.7%.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/9d25202271de2ab93eb4fa34f7684504589733d7-4470x2825.png" alt="Performance of Models on APEX-SWE Leaderboard" style="width: 100%; border-radius: 10px;" />
APEX-SWE is split evenly between two task types that reflect real-world software engineering work:
<ul><li>Integration tasks, which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services.</li><li>Observability tasks, which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context.</li></ul>
Models perform better on Integration tasks than Observability tasks. For Integration, Claude Opus 4.5 (High) and GPT 5.4 (High) lead at 50.7%. Observability scores are lower overall, with GPT 5.3 Codex leading at 33.3% Pass@1. For both task types, top-performing models demonstrate strong capabilities but still do not meet the production bar.
For both task types, when models succeed they demonstrate epistemic reasoning—treating their generated code as a hypothesis that must be tested against the actual system state before being finalized. This iterative and careful approach results in more successful code executions as problems are overcome through debugging.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/1f232db8cee704cb0b4f6a2598e2a26abd8d76d0-3560x2247.png" alt="Model Performance Comparison - Integration vs Observability" style="width: 100%; border-radius: 10px;" />
<h2>Our process</h2>
Each integration task contains an environment for the agent to operate it. They all include an ephemeral PostgreSQL database and Plane, as well as six other services: LocalStack (56%), which emulates AWS primitives such as S3, Lambda, DynamoDB, and Kinesis, EspoCRM (35%), MailHog (33%), Mattermost (32%), Medusa (31%), and Zammad (26%). Tasks were created by software engineers with 3+ years of experience who ran validation tests and created gold standard outputs.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/41f64b98bc44cbeb18e17a4f2ad2514034d0fb01-4426x1788.png" alt="Integration task process" style="width: 100%; border-radius: 10px;" />
Each observability task deploys a containerized environment orchestrating five services: a client workspace, Loki and Promtail for log aggregation, Grafana for visualization, and Plane/Mattermost for ticket and chat context. Engineers scripted synthetic logs (500 to 1,000 lines of normal operation mixed with 10-20 lines of bug symptoms) and chat history to replicate a production failure, as well as Dockerfiles, task metadata, and patches. Tasks were derived from real-world GitHub Issue-PR pairs, sourced from repositories with at least 350 stars that we filtered for complexity and stability. Observability tasks are distributed across five widely-used languages: Go (30%), Python (25%), TypeScript (25%), Java (10%), and C++ (10%).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/52b314ca7c2bd9a13063f4c83aa654abef28bd82-4606x1788.png" alt="Observability task process" style="width: 100%; border-radius: 10px;" />
<h2>Open source</h2>
We have released an <a href="https://huggingface.co/datasets/mercor/APEX-SWE" target="_blank" rel="noopener noreferrer" class="text-indigo-600">open-source dev set via Hugging Face</a> (n=50) with a CC-BY license, and our eval harness is available as an <a href="https://github.com/Mercor-Intelligence/apex-swe" target="_blank" rel="noopener noreferrer" class="text-indigo-600">open-source repo on GitHub</a>. The full leaderboard comprises n=200 tasks that are heldout and hidden, with a similar distribution to the open-source set. Read <a href="https://arxiv.org/pdf/2601.08806" target="_blank" rel="noopener noreferrer" class="text-indigo-600">the APEX-SWE technical report</a>.
We thank all the software engineers on the Mercor marketplace who contributed their time to creating APEX-SWE.

Ben Pan

Member of Technical Staff, Cognition

Sam Lee

Silas Alberti

Founding Team Member, Cognition

Introducing APEX-SWE, a new benchmark created in collaboration with Cognition, measuring whether frontier AI models can handle real software engineering work.

Introducing APEX-SWE | Mercor x Cognition 

Introducing the AI Productivity Index for Software Engineering

<h4>Amresh spent nearly a decade advising on AI strategy at McKinsey. He joined Mercor as Director of Operations to build the data infrastructure that actually moves model performance.</h4>
I spent nearly a decade at McKinsey, most recently as an Associate Partner in the Tech practice in San Francisco. I worked with some of the best minds in the industry, sharpened my thinking on AI strategy, and built relationships I&#39;ll carry for the rest of my career. But somewhere along the way, I started craving something different. I wanted to build, not advise. To own outcomes, not just recommend them.
When Mercor came across my radar, the vision was immediately compelling: become the defining talent platform for the AI economy. But what really made the decision easy was a conversation with Brendan. His passion was obvious, but what struck me more was his maturity and clarity of thinking. We were aligned on what we were trying to do - build something genuinely legendary. That kind of founder conviction is rare, and it changes what&#39;s possible.
The timing couldn&#39;t have been better. Over the past year, I&#39;ve watched the AI market shift in a profound way. As frontier models push toward higher-order reasoning and agentic capabilities, the demand for specialized human expertise has accelerated dramatically. Better models need better training data, and that means better experts. Mercor sits right at the center of that dynamic.
Our role is becoming increasingly strategic. We&#39;re helping customers think through their data collection approaches, shaping what &quot;expert-in-the-loop&quot; actually looks like in practice, and growing fast. In my first year, I&#39;ve had the chance to grow accounts to nine-digit run rates and build a team of more than 20 people. I&#39;ve also learned more about AI research than I ever expected; specifically, what kinds of data actually move the needle on model performance. It&#39;s been the steepest and most rewarding learning curve of my career.
What grounds all of that, for me, is the impact on our network of experts. I’ve heard many stories about their motivations for joining Mercor: people who want flexibility, who&#39;ve hit a rough patch, or who simply want to put their expertise to work on their own terms. Now, we’re paying out $2 million to experts every day. It’s clear to me that this new category of work is accelerating as quickly as the AI economy itself is.
The thing that stood out to me early on, and still does, is the talent density. The team skews young, we work hard (yes, including Saturdays), but the culture is genuinely collaborative and down-to-earth. Decisions get made fast and in the open. Our weekly all-hands is a good example: substantive and informative, but also adds levity. We&#39;ve had rap performances, a gumbo band, investors, and experts all show up to share their perspectives. We also highlight experts who share their experience with Mercor and provide the team real-time feedback. It’s a moment to be proud of the impact, but also a reminder there’s always more we can do. It feels like being in on something.
I&#39;ve also been surprised by how much room there is to shape things beyond your immediate role. From organizing weekly happy hours to having direct conversations with the founders about career progression and culture, the scope for impact is real. At McKinsey, influence moved slowly and through layers. Here, if you have a good idea and the drive to push it, things actually change.
Looking ahead, the opportunity feels almost hard to overstate. We&#39;re already one of the fastest-growing companies of all time, and the market headroom is enormous. That growth brings real challenges — some exciting, some humbling — but that&#39;s exactly what makes this moment worth being part of.
If you&#39;re considering Mercor and wondering whether to make the leap: do it. It won&#39;t always be smooth, but I genuinely believe many of us will look back on this period and feel proud that we helped steer something important at a critical moment in how humanity works, learns, and builds. Check out our open roles: <a href="http://www.mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">www.mercor.com/careers</a>

Amresh spent nearly a decade advising on AI strategy at McKinsey. He joined Mercor to build the data infrastructure that actually moves model performance.

Why I Joined Mercor – Amresh Subramaniam

<h4>Anish Bathwal spent years as a consultant at McKinsey before joining Mercor to be closer to the answer every company was asking: &quot;how do we use AI?&quot; </h4>
What does a consultant actually do?
The honest answer is we get dropped into rooms where the problem isn’t even defined yet and told to figure it out. A software company wants to embed AI into a product that’s been built the same way for fifteen years. A university is watching enrollment decline and can’t tell if it’s the cost, the experience, or the competition. Nobody agrees on what’s wrong, let alone what to do. The job is to synthesize ambiguity into something actionable—fast.
I spent my time in consulting doing this across software and education. The industries were different, but by the time I left, every client was asking the same question: how do we use AI? The urgency was striking—organizations that had nothing in common were all hitting the same wall. But the bottleneck was never buy-in. It was never implementation. It was performance. The models weren’t good enough on the problems that actually mattered—the domain-specific work where expertise couldn’t be faked.
No amount of consulting could fix that. What could fix it was better data. The kind produced by real professionals with years of work experience or advanced degrees. I started paying attention to how models actually improve: through high-quality, knowledge-intensive training data generated by people who deeply understand their field. The question that stuck with me was deceptively simple. What would it look like to solve the performance problem at the source?
Then I heard from some friends in the Bay Area about a startup called Mercor. They were hiring thousands of lawyers, doctors, financial analysts and even consultants and deploying them on projects for the world’s top AI labs. The thesis resonated immediately: to build next-generation models, you need highly specialized data, and to create that data, you need real experts generating complex tasks that require deep reasoning. You could have domain experts create evals that teach a model to replicate their reasoning at scale—not expertise delivered to one client at a time, but expertise encoded permanently. Two days later I was on a flight from NYC to SF.
Now I work on web browsing—training models to persist through complex, multi-step queries rather than stopping at the first result. Think about what that unlocks: a model that can dig through dozens of sources to answer a genuinely hard question, the way a researcher or analyst would. Today’s models quit early. The data we create is designed to teach them not to. PhD-level annotators write research-grade queries at volume, paired with quality frameworks rigorous enough to ensure every task actually moves the needle against benchmarks. Researchers at the lab fold what we produce directly into their models. You can watch the benchmarks move with each model release.
The best AI models a year from now will be built on data that doesn’t exist yet. Someone has to create it, and that requires systems that can channel real expertise into training signal at scale. That’s what we’re building.
For anyone at a similar crossroads—where you’ve gotten good at navigating ambiguity but want your work to compound beyond a single engagement—I’d encourage you to <a href="http://www.mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">check out our open roles</a>.

Former McKinsey consultant Anish Bathwal on why he left consulting to join Mercor, where expert knowledge is training the next generation of AI models.

Why I Joined Mercor – Anish Bathwal

There’s a paradox in AI right now. We are witnessing undeniable leaps in model performance. AI models are coding and reasoning at superhuman levels. Yet the massive economic promises of AI still haven’t materialized. Most companies are still relying on the same manual workflows they used five years ago.
Why hasn&#39;t the revolution hit the bottom line?
The answer is that models are smart, but they’re not trained to do the job. Today’s frontier models are like new college graduates – talented, but if you drop them into a company without onboarding, they&#39;ll fail. They don’t know how to navigate messy workflows, use the tools and apps systems are built in, or recognize what ‘good’ looks like for your specific customers in your industry.
The future of work requires training agents with human expertise: from teaching models the fundamentals of professional work to encoding your company&#39;s specific context. In order to unlock their economic potential, agents will have to be taught to navigate the huge distribution of real-world workflows.
Meeting this new demand for human expertise at scale requires a new kind of infrastructure – a platform that understands what every person knows, what they’re capable of, and where their expertise can create the most value.
That is what we are building at Mercor: the definitive network for human knowledge and capabilities.
<h2>The future of foundation model training</h2>
While frontier models have improved dramatically in general reasoning, we’re in the first inning of the training required to unlock the full economic potential of knowledge work.
The surface area of knowledge work is vastly larger than most people realize. There are over 800 distinct occupations in the U.S. alone, each with dozens of core workflows. A corporate lawyer doesn&#39;t just &quot;practice law&quot; – they review NDAs, redline vendor contracts, flag regulatory risk in M&amp;A filings, and negotiate indemnification clauses, all while toggling between a document management system, email, and a spreadsheet tracking deal terms. Multiply that complexity across every profession, and you begin to see the surface area that models need to master.
We&#39;ve seen what happens when that surface area gets measured through evals. SWE-bench gave AI labs a structured way to evaluate coding agents against real GitHub issues. Once researchers could measure it, they could hillclimb it. However, software engineering benefits from a pre-existing corpus of open-source repositories and unit tests that make evaluation natural. The vast majority of knowledge work – law, medicine, finance, consulting, operations – has no equivalent.
The hyperscalers will need to pour hundreds of billions into constructing environments that mimic the full distribution of professional contexts, apps, and tasks. Because complex environments are far more labor-intensive to build than legacy training approaches, budgets are shifting dramatically toward human data, with many labs 10x-ing their environments budgets this year alone and already spending over $1 billion each.
While labs will continue to build synthetic data pipelines and training infrastructure in house, none of them want to recreate an enormous, operationally complex talent network from scratch. And they can’t afford to wait. For the companies whose core businesses depend on workspace productivity, this transformation is existential. Hundreds of billions in annual revenue hang in the balance if they fall behind or launch ineffective agents.
<h2>The future of enterprise deployment</h2>
Even a model that has perfected simulated tasks in law or finance still lacks the one thing no foundation model can arrive with: your company&#39;s internal context. The new job of the knowledge worker will be to encode their expertise – the intuition that lives in their head – into agents.
At Mercor, we’re already training agents like employees. Our customer support representatives no longer respond redundantly to hundreds of similar tickets. They update documentation and evals to train an agent instead, giving themselves time to focus on higher-value tasks. And our fraud team doesn&#39;t manually catch people cheating in interviews. They build evals that teach an agent to help them spot the patterns.
This is what convergence looks like. When you can turn repetition into training, you get compounding returns: faster turnaround, consistent quality, and larger scale. The boring parts of the job get absorbed into the system, leaving people to focus on the work that actually requires strategy and judgment.
This creates a flywheel between the labs and enterprise. Benchmarks for real-world workflows make models robust enough for production. Once deployed, real enterprise usage will reveal the edge cases and reasoning gaps that no simulation could predict. That signal feeds back to the labs to refine core capabilities, and back to the enterprise to sharpen the agent&#39;s playbook – each cycle unlocking more complex workflows.
<h2>The most important network effect</h2>
The prevailing narrative is that AI will render human workers obsolete. The reality is more nuanced. There will be tasks that humans can solve where models fail for decades to come because of the long tail of human expertise. Models are improving quickly within the broad surface of knowledge work, but progress in the physical world will be far slower.
The reality is that there are millions of different categories of professional work, and the same job has completely different requirements at different enterprises. As we move from chatbots to agents, and from demos to deployment, the limiting factor – the fuel that powers the entire system – will be human expertise.
So the real question isn’t whether human expertise is needed but who will build the infrastructure to find it, organize it, and deploy it at the scale AI demands.
That’s what Mercor is building. We&#39;ve created a network of over 4 million vetted experts and over 1.9 million referrals – a self-reinforcing system that turns loose professional knowledge into structured, reliable data. Every project teaches us more about what our experts know and what they&#39;re capable of, so the next match is better than the last. We scaled from paying out nothing to over $2M/day within two years by answering society&#39;s most important question: what role will humans play in the AI economy?
Our unique ability to answer this question is framed by the fact that we know who&#39;s an expert in what, all around the world. Uber built a driver network. Airbnb built a host network. Mercor is building a knowledge network – the definitive platform that understands everyone&#39;s expertise and the economic value they can deliver. This is the most important network effect that hasn&#39;t been built yet.

Mercor is building the definitive network for human knowledge and capabilities.


Organizing human intelligence to power the AI economy

Every nine seconds, someone starts an interview with Monty, our AI interviewer. That someone might be a software engineer, a banker, a farmer — and none of them are talking to a person. Around 10,000 of these conversations happen every day, each one fifteen minutes long, across hundreds of job categories. Three engineering problems make this hard: keeping every session alive, making turn-taking feel natural, and making sure every candidate gets an interview that&#39;s right for them.
<h2>Keeping every session alive</h2>
Every interview has to work. A candidate who gets a broken session doesn&#39;t get a second chance at that role. Each session runs in its own container on <a href="https://modal.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Modal</a> — spun up on demand, torn down when the call ends. A crash in one container affects one interview and nothing else. In 2025, we scaled from a few hundreds interviews each week to over ten thousand a day. We made no changes to the hosting. What we did change was how we deal with cold starts.
When a candidate clicks &quot;Start Interview,&quot; the session should begin immediately. A fresh container takes several seconds to boot and spin up a video room — long enough that a candidate would notice they&#39;re waiting. We handle this with a warm pool: Modal keeps about 30 containers pre-booted at the compute level, and a background job running every five minutes keeps about 10 fully initialized — room URL registered in Redis, ready to go. When a session starts, it grabs one in well under 200ms. The harder problem is calibrating pool size: too few and cold starts leak through at peak; too many is waste. We track demand by hour and size the pool ahead of it.
The same logic applies everywhere else in the stack. Audio runs on <a href="https://daily.co/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Daily</a> — Pipecat&#39;s WebRTC layer — handling peer connections, media routing, and cloud recording so recordings land in S3. Speech recognition, the LLM, and text-to-speech each run across a mix of commercial APIs and open-source models, with automatic failover at each stage — any individual outage is invisible to the candidate. Whenever we make a config change, we blue-green it over a week, meaning a bad prompt update gets caught well before it ruins a thousand interviews.
<h3>Container count — typical day (Pacific)</h3>
Figure 1 — Container count by hour (Pacific Time), calibrated to now. Each interview runs in its own isolated container; a warm pool keeps cold-start latency under 200ms. Peak at noon: ~200 containers. Floor overnight: ~80. Drag to explore.
<h2>Getting turn-taking right</h2>
Sounding like a human is hard!
A response that would land naturally at 800ms feels like an interruption at 400ms and dead air at 1500ms. The pipeline runs on <a href="https://pipecat.ai/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Pipecat</a>, an open-source voice AI framework created by Daily, fully streaming end-to-end: speech recognition runs continuously, turn detection fires as soon as silence is detected, and TTS starts synthesizing on the LLM&#39;s first sentence before the rest has finished generating. For turn detection, we use smart-turn-v3 — an ONNX model from the Pipecat team, running on Modal — at P50 ~150ms. Add LLM first-token (~350ms) and TTS first audio (~200ms), and the median time from candidate silence to Monty&#39;s first word is about 700ms. Anything past a second starts to feel broken.
<h3>End-to-end latency — candidate silence to Monty&#39;s first audio</h3>
Figure 2 — Smart-Turn classifies the end of the candidate&#39;s turn (150ms p50) → LLM streams its first token (350ms p50) → TTS produces first audio (200ms p50). STT runs as a continuous stream — no added latency. TTS begins on the LLM&#39;s first sentence while the rest is still generating.
If you cut in too early, you end up interrupting a candidate mid-thought; wait too long and it starts to feel like lag. After tuning against candidate self-reported experience and session completion rates, we settled on 900ms as the production threshold — enough pause that candidates feel heard without letting silence linger. This breaks down into three parameters: a 120ms floor prevents triggering on mid-sentence pauses, a 1.6s ceiling is the hard fallback, and the right setting shifts by interview segment, and some special handling for VAD. These numbers came out of rounds of A/B testing against session completion rates; none were set once and left alone.
<h3>End-of-turn pause distribution — drag to adjust threshold</h3>
Figure 3 — Each circle is a candidate&#39;s end-of-turn pause duration. Red dots are candidates Monty interrupts; green dots feel natural; yellow dots are candidates left waiting in silence. Drag the slider to see the tradeoff.
Threshold tuning only gets you so far. Short acknowledgments — &quot;yes,&quot; &quot;uh-huh,&quot; &quot;got it&quot; — often lack trailing silence: the candidate is done, but voice activity detection (VAD) never fires because the signal is too brief. A 400ms aggregation timeout catches these; without it, Monty waits indefinitely for speech that has already ended.
The other gap is echo: Monty&#39;s TTS output can leak through the candidate&#39;s microphone and get transcribed as candidate speech. We run a simple LLM-based classifier that detects when the candidate is essentially repeating Monty back to itself, and discards those turns. Both are invisible at small scale.
<h2>Giving every candidate the right interview</h2>
Mercor runs interviews across hundreds of job categories — engineers, bankers, lawyers, data scientists. The obvious approach is one assessment per job title; we built that first, and it doesn&#39;t scale. Maintaining hundreds of distinct assessments is expensive, and small variations between nearly-identical roles — &quot;backend engineer&quot; vs. &quot;platform engineer&quot; — don&#39;t justify separate configs. For candidates, retaking the same interview for every role they apply to is pointless.
We cluster job listings by the skills they actually test, weighted by candidate volume and hiring outcomes. The clusters are fewer than you&#39;d expect — most job titles, stripped of their labels, test a fairly small number of underlying things. The dominant one is what we call Domain Expert: a single assessment that covers medicine, economics, history, law, software architecture, and almost any other knowledge domain. It accounts for over 70% of all sessions. Add code and language assessments and you&#39;ve covered 90% of volume with three types.
<h3>Sessions per day — click a category to explore</h3>
Figure 4 — Session volume treemap (areas ∝ cumulative historical session totals by category; labels show daily counts). Domain Expert Interview is the most common assessment type by a wide margin.
Completing Domain Expert once qualifies a candidate for every role in that cluster, with no retakes required. We deploy unified assessments as the default for all listings. Today, more than half the offers on Mercor go out proactively — the candidates didn&#39;t apply to those jobs, they just took an interview some time ago.
The interview is personalized before it starts. We process the candidate&#39;s resume before the session begins, and that context shapes what gets asked, how deeply to probe, and what to skip. For coding interviews, the problem is generated fresh from that profile and the live conversation: a Go engineer and a Python engineer get different starter code; a senior candidate gets a more open-ended problem than a new graduate.
All of this means that interviews are efficient. While we think the AI interviewing experience is fun, we try to make the most of the time our users choose to spend with us.

Lucas Rothman

Richter Brzeski

How Mercor keeps AI interview sessions alive at scale, tunes turn-taking latency, and routes candidates to the right assessments.

Engineering Monty: Scaling an AI Interviewer

Engineering Monty: Scaling an AI interviewer

<h1>Applied Compute&#39;s model, trained on Mercor’s agentic data, is now top of the APEX-Agents leaderboard in corporate law</h1>
In January, we showed that <a href="https://www.mercor.com/blog/expert-data-drives-model-performance/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">fewer than 1,000 expert-labeled tasks</a> could double an open-source model&#39;s performance on APEX-Agents, our frontier benchmark for cross-application, long-horizon tasks in professional services. The training trendline was near-linear, signaling that more data would keep yielding gains so we scaled the dataset for <a href="https://appliedcompute.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Applied Compute</a> to almost 2,000 high-quality cases. They post-trained <a href="https://huggingface.co/zai-org/GLM-4.7" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GLM 4.7</a> and their new model Applied Compute: Small is now top of the APEX-Agents leaderboard in corporate law, with a Pass@1 score of 26.6% and a mean score of 54.8%. 
The new model places 4th on the APEX-Agents leaderboard overall by mean score, a dramatic improvement on GLM-4.7 which is 17th. This is significant as GLM-4.7 is a strong open source model, but at 355B (MoE) and a total context length of 200k, it is much smaller than the commercial-grade models that it is now competing with.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/c3765ea997225a0aa54cc431a56f1cf44fcc49c1-2979x1622.png" alt="Scaling up data lead to significantly higher performance gains on APEX Agents." style="width: 100%; border-radius: 10px;" />
<h2>How has the model’s behavior changed?</h2>
To understand what changed, we compared the trajectories of GLM-4.7 and Applied Compute: Small head-to-head. On average, Applied Compute: Small consumes roughly 2 million tokens per trajectory, 4x the token usage of GLM-4.7, and much closer to frontier models like Gemini 3.1 Pro and Claude Opus 4.6. But GLM-4.7 can be served for as little as $0.50 per million input tokens, while Opus 4.6 costs 20× that for long tasks and GPT 5.2 Pro costs 40×. This represents a substantial difference in the economic utility of each dollar spent on tokens.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/829a6e2bb052f7db9e1d1883d44cfc69f950ad87-2062x1048.png" alt="The probability density function of the number of steps used by Applied Compute: Small and GLM-4.7." style="width: 100%; border-radius: 10px;" />
Applied Compute: Small completes tasks in far fewer steps (43 vs. 72) than its base model. As Figure 1 shows, its step distribution is tightly concentrated, while GLM-4.7 exhibits a long tail extending past 150 steps. This is driven in part by doom-looping behavior, where GLM-4.7 repeatedly attempts broken tool calls or falls into cycles. Applied Compute: Small, by contrast, is more efficient at locating the files it needs—but because those files tend to be large, it incurs substantial token costs when reading them in.
Applied Compute: Small learnt to rely on code execution, using it at least once in 98% of trajectories. It also searches the file system 96% of the time, and typically either reads at least one PDF (84% of the time) or xlsx file (63% of the time). In contrast, GLM-4.7 only uses code execution in 19% of trajectories and often fails to read in the right documents. This is a critical limitation given that APEX-Agents requires navigating the file system to complete each task.
<h2>Has the model lost any capabilities?</h2>
We evaluated GLM-4.7 and Applied Compute: Small on two industry standard benchmarks, HLE and GPQA, to measure whether Applied Compute: Small has regressed, potentially losing other valuable capabilities while improving at APEX-Agents. We see a small improvement on GPQA of 0.4pp and a small decrease on HLA of 1.3pp. Neither of these differences are statistically significant and, acknowledging that we only tested on two benchmarks, we did not find evidence of Applied Compute: Small losing general capabilities.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/5a31a16a5d8ed234927c71ab8989ac7b25d4b070-1466x372.png" alt="Performance of GLM-4.7 and Applied Compute: Small on industry benchmarks. Note: Reference benchmark scores were taken directly from https://z.ai/blog/) [1]" style="width: 100%; border-radius: 10px;" />
We also looked at the length of the final outputs. On APEX-Agents, the average number of tokens in Applied Compute: Small’s final outputs is ~3x that of GLM-4.7 (1,956 vs 631). This is a moderate increase, but indicates that Applied Compute: Small is not just scattergunning information and hoping for the best. The difference is also inflated by the large number of cases where GLM-4.7 responds with a short statement that it cannot find the right files.
<h1>Conclusion</h1>
In January, we showed that fewer than 1,000 expert-labeled tasks could nearly double an open-source model&#39;s performance on APEX-Agents. By scaling to 2,000 cases, Applied Compute: Small now leads the corporate law leaderboard outright, outperforming models that are orders of magnitude more expensive to serve. 
This result reinforces a broader thesis at Mercor -- quality expert data, paired with the right post-training infrastructure, can close gaps on model performance. The behavioral changes we observe (more effective tool use, fewer wasted steps, richer final outputs) suggest that the model is not just memorizing patterns but learning the requirements of professional work.
For Mercor, post-training is a vital part of our stack. It validates the quality of our data, helps us allocate experts to produce measurable and generalizable performance gains, and drives our failure analyses, letting us understand not just how often models fail, but why, and how to fix it. To learn more about <a href="https://arxiv.org/abs/2601.14242" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a>, data quality, and post-training at Mercor, contact apex@mercor.com.
Applied Compute helps enterprises build and own AI agents trained on proprietary data, aligned to specific workflows, and designed to continuously learn. To learn more, reach out to team@appliedcompute.com.
<h1>Footnotes</h1>
[1] HLE and GPQA are reported as avg@1 and avg@3 respectively. 95% confidence intervals are calculated naively using the normal approximation to the binomial.
[2] Base models were evaluated using the ReAct agent harness, as introduced by Yao et al. (2022) (<a href="https://arxiv.org/abs/2210.03629" target="_blank" rel="noopener noreferrer" class="text-indigo-600">arXiv:2210.03629</a>). While this richer scaffolding often improves raw task performance, it is not conducive to training or efficient inference as the system prompt changes throughout the trajectory. As such, post-trained models were trained and evaluated using a simple loop agent architecture. For implementation, see here: https://github.com/Mercor-Intelligence/archipelago

Isaac Robinson

Michael Haines

Applied Compute's model, trained on Mercor’s agentic data, is now top of the APEX-Agents leaderboard in corporate law

Scaling Data leads to SOTA Legal Performance on APEX-Agents

<h3>Luna Aizarani has spent her career within expert marketplaces. She joined Mercor as General Manager of Growth to build the systems that connect human expertise to opportunities at global scale.</h3>
I’ve spent over a decade operating on the frontlines of how organizations find and deploy talent, across labor marketplaces and talent intelligence platforms. As AI reshapes entire industries and accelerates economic change, I consistently encountered the same tension: the type of talent in demand was evolving faster than the systems designed to identify and deploy it. Highly capable people were underutilized, while organizations struggled to translate ambition into execution.
Across tech, private capital, and workforce development organizations, the same pattern repeated itself. Companies raced to deploy AI systems faster than their organizations could absorb them, while skilled experts sat on the sidelines or were misallocated. From where I sat, the issue wasn’t intelligence or motivation, it was infrastructure.
Throughout my career, I’ve worked as a commercial leader in high-growth environments. I spent years scaling revenue engines inside professional services organizations, managing global teams, owning P&amp;Ls, and turning underperforming units into durable businesses. More recently, I worked in workforce development and partnerships, helping large institutions respond to rapid shifts in skills, automation, and AI literacy. I found the work meaningful, but I increasingly felt the impact was incremental relative to the scale of the change underway.
When I started talking to Mercor, it felt like someone had finally named the problem clearly and, more importantly, was already building practical solutions to address it.
The AI economy isn’t constrained by ideas; it’s constrained by execution. Models are only as good as the data that trains them, and that data increasingly depends on deep human expertise: engineers, researchers, and domain specialists who can reason, evaluate, and create at a high level. Mercor is the company that identifies real expertise, deploys it quickly, and operates globally. The shift from ad hoc sourcing to programmable talent is what makes the company so central to the future of the labor market. That framing immediately resonated with me.
In my role, I work directly inside that shift. I develop systems that expand access to expertise, connect the right talent to meaningful work, and make complex, human-heavy workflows repeatable. Over time, that responsibility has grown into owning the broader talent pipeline across the company, focusing on ensuring our talent network keeps pace with demand and that experts have a clear, supportive path to contribute. It’s part growth, part operations, part product thinking.
One of the most meaningful aspects of my job has been engaging with people whose expertise isn’t always captured by traditional credentials. I’ve spent time with athletes and coaches, for example, learning how their real-time decision-making, pattern recognition, and tactical judgment can inform how AI systems learn and improve. I’ve seen similar value from farmers, therapists, and sales managers - people whose knowledge compounds through lived experience in the field. We are creating pathways for expertise to shape the systems increasingly influencing how work gets done.
What surprised me most after joining Mercor wasn’t just the team’s ambition, but the rigor of execution. From my earliest conversations with the founders, it was clear that Mercor operates with unusually high standards, very little red tape and a strong culture of ownership. Within days of joining, I was responsible for addressing supply constraints across multiple workstreams, proposing structural fixes, and moving quickly from assessment to execution.
I’m excited about what’s next in my role at Mercor. This includes deepening strategic partnerships, making the expert experience more seamless, and continuing to adapt our systems alongside a rapidly changing labor market. As AI reshapes how work is created and distributed, the opportunity is to build infrastructure that works well for both partners and experts.
For those considering Mercor and feeling unsure, the work here tends to suit people who enjoy building, take real ownership, and are motivated by driving tangible results.
Explore careers at Mercor: <a href="http://www.mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">www.mercor.com/careers</a>

Luna Aizarani, GM of Growth at Mercor, shares how she’s building systems that connect real human expertise to meaningful work at global scale.

Why I Joined Mercor: Building Infrastructure to Scale Expertise 

Why I Joined Mercor – Luna Aizarani

<h3>Peter Zhang, Mercor Engineering Lead, shares why he transitioned from a career in quantitative finance to be at the forefront of AI.</h3>
Why do we pay quantitative traders <a href="https://work.mercor.com/jobs/list_AAABmpZDfV9yE40R8g1Hw4j3/quantitative-traders" target="_blank" rel="noopener noreferrer" class="text-indigo-600">$300/hr</a>?
We pay them to resolve challenging ambiguity—questions about the world that are difficult to scope and harder still to answer. Markets cannot afford to be wrong about the future of financial capital. Trillions of dollars swing on how the world will look in one, five, and twenty-five years, making markets early bells for world-altering shifts.
The bells have been ringing. AI is the question mark of the decade. Markets are scrambling to assess the scale of the buildout, the potential productivity gains, the ultimate winners. Machine intelligence is rippling through industry and everyday human life, raising new questions about our future. Last year, from the trading floor of Jane Street, I felt the gravity of these questions and frustration at the limits of a trading terminal. I wanted to get closer to the ambiguity.
I wanted to study what AI would mean for human capital. Quantitative finance understands talent better than any industry. Firms live and die by identifying, measuring, and utilizing talent. Well before Jane Street, I wondered whether we could encode our heuristics about how to do work into rules and models. These questions have always been considered too complex, too human, to systematize—so we&#39;ve built entire fields of organizational psychology and management theory on intuition rather than data, surrendering to subjective judgment what should be measured and understood.
In early 2025, I heard from a former colleague about a startup called Mercor. They were taking the best experts from around the world and pushing the frontier of model capabilities. They were hiring thousands of experts every month across hundreds of industries and verticals. They were operating at a scale where they couldn’t afford to be wrong about human capital. It was the perfect frontier.
Mercor moves fast. I left New York in April. By May, I was describing my work to a friend. &quot;How can you agree to work on something that might even not be possible?&quot; he asked. &quot;There’s no roadmap!&quot; I was working on assessments: how to move past resumes and directly measure skill. These problems felt more like research papers than product specs: What makes an interview “good?” How can AI <a href="https://talent.docs.mercor.com/support/ai-interview" target="_blank" rel="noopener noreferrer" class="text-indigo-600">run and score interviews</a> at scale? How do you quantify performance when you&#39;re hiring across industries as different as software engineering and law? &quot;The ambiguity,&quot; I told him, &quot;is exactly why I’m here.&quot;
Since then, Mercor has run over a million AI interviews and placed thousands of people into roles they wouldn&#39;t have found otherwise. Traditional screening would have filtered out many of our top performers—the self-taught engineer in Lagos, the Portuguese lawyer. Our models found them and they&#39;re excelling.
These days, I spend my time engineering solutions to new questions. How do we structure teams that multiply rather than average individual brilliance? How do we measure who&#39;s truly excelling when simple metrics fail? And how can we automate these processes without losing the nuance that makes them work?
AI is the new center of gravity for the brightest minds. If you’re excited to face challenging ambiguity at the intersection of AI and how we do work, I’d love to <a href="https://www.mercor.com/careers/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">work with you</a>.

An Engineering Lead at Mercor shares why he left quantitative finance to build AI systems for measuring human talent at scale.

Why I Joined Mercor: From Quant Finance to Engineering

Why I Joined Mercor – Peter Zhang

States that Summit's shareholder distributions are disproportionate

States that Summit's disproportionate distributions violate US tax code

States that a nonresident alien receiving shares violates tax code

<h2>How Mercor’s data and Applied Compute’s long-horizon RL unlock real capability gains for AI models</h2>
Mercor partnered with <a href="https://appliedcompute.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Applied Compute</a> to post-train an open-source model using one of our expert-labeled dev sets, resulting in substantial performance gains on the <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a> benchmark.
With fewer than 1,000 high-quality data points from Mercor, the post-trained model&#39;s Pass@1 and mean score nearly doubled. On the corporate law evals, the Pass@1 score tripled. The training trendline is near-linear, indicating that additional data would likely keep yielding performance gains.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/c15fd60509a1f5ad2dbd7209a4d47d68a0ea85da-1860x510.png" alt="APEX Score across training showing consistent improvement" style="width: 100%; border-radius: 10px;" />
APEX-Agents assesses whether agents can execute the real day-to-day work of investment banking analysts, management consultants, and corporate lawyers. There are 480 tasks, created by a team of Mercor experts: Vice Presidents, Managing Directors, and Managers with 10+ years of experience at top-tier firms. They simulated the demands of the profession to challenge agents to navigate instructions, use a range of applications, manage complicated file systems, and plan over long horizons. See the <a href="https://arxiv.org/abs/2601.14242" target="_blank" rel="noopener noreferrer" class="text-indigo-600">technical report</a> for more details. You can also download the dataset and all of the files from Mercor’s <a href="https://huggingface.co/datasets/mercor/apex-agents" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> and our infra service, Archipelago, from our <a href="https://github.com/Mercor-Intelligence/archipelago" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GitHub</a>. For this experiment we used the full benchmark (n=480), measuring Pass@1, Pass@3, and mean criteria passed.

<h1>Training set up</h1>
Mercor provided a dev set of 874 tasks for post-training, split across 50 unique “worlds” of data that represent a scenario. Each world has data and applications that can be found in a real enterprise environment, such as Google Sheets, Docs, and Code Execution. None of the tasks or worlds appear in the APEX-Agents benchmark.
Applied Compute deployed its proprietary long-horizon RL stack to stress-test data quality under realistic training dynamics and measure whether gains transferred to the hardest APEX-Agents tasks. Training was performed single-epoch with no SFT warmup, no filtering, and no task or rubric modifications. Two held-out worlds per domain were reserved for validation to detect overfitting.
Applied Compute evaluated frontier and open-weight models on APEX-Agents to establish baseline capability and identify where learning was possible. GLM 4.6, a mid-scale 355b parameter MoE model with 32b active parameters, emerged as an appropriate model to start from, offering the right tradeoff between iteration speed and baseline competence. It scores 3.8% with Pass@1 and 12.1% based on mean score, which is typical for an open-source model on APEX-Agents.
<h1>Results and continued training gains</h1>
The post-trained model outperforms the baseline across all metrics, with the largest gains in corporate law (Table 1).These improvements come from just 874 expert-labeled tasks. In a low-data regime, the dominant risk is not under-training the model, but misallocating scarce expert effort.
RL training and evaluation reduced this risk by turning each run into a high-signal measurement. By running end-to-end RL with detailed behavioral observability, Applied Compute enabled Mercor to see how much benefit was being delivered by the data, demonstrating that hundreds, not tens of thousands, of examples are sufficient to drive real gains.
Full trajectory-level observability showed how models attempted tasks, allowing Applied Compute to distinguish learning from undesirable behaviors like refusal or premature termination that can be masked in aggregate metrics. Targeted ablations over reasoning budget, tool access, and training configuration were evaluated against optimized prompt baselines to isolate training gains. When issues were identified, fixes were validated through retraining without adding new data. Training eval trendlines showed consistent improvement. Gains across law, banking, and consulting were close to linear, which is atypical in data-limited settings and signals strong alignment between the data and target capabilities. Rapid feedback from training runs allowed expert effort to be redirected toward concrete gaps instead of expanded uniformly. We anticipate that similar amounts of high-quality data will continue to deliver benefits.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/0b5b233542dc1f4af8354bc7dabd5d472f4efb24-3840x2160.png" alt="Training trendline based on APEX Score" style="width: 100%; border-radius: 10px;" />
<h1>Example trajectory</h1>
One of our worlds (see below) shows a corporate lawyer on a due diligence project, tasked with analyzing whether the company’s shareholder payouts comply with US tax law. The world contains a full suite of corporate records: shareholder agreements, the distribution schedule, the original S-Corp election filing form, and a reference copy of the tax code that governs S-Corporations.
The baseline model produces a professional looking but factually flawed memo that incorrectly states that the corporation is in substantial compliance with US tax code. It used 18 tool calls and worked linearly. It listed files, read the distribution schedule and tax returns before submitting its final answer. The baseline model took the documents at face value and justified compliance despite evidence that proved otherwise. It correctly found the shareholder and payment records, but assumed the distribution matches the ownership stakes rather than checking.
In contrast, the post-trained model correctly states that the company is non-compliant. It also finds out that one shareholder was overpaid in 2022 and 2024 by correctly citing 26 USC §1366(a)(1) regarding the failure of pro-rata allocations and stating that the transfer to one individual violates §1361(b)(1)(C). The post-trained model improved in its reasoning and planning. It spent 9 steps just on reading specific US tax code sections, then executed several Python scripts to check the math on the distributions. It even performed a self-correction when it realized it hadn&#39;t completed its checklist.
<h1>Conclusion</h1>
This experiment demonstrates that quality datasets can push open-source models to unlock new capabilities, adopting the rigorous approach required for high-stakes professional work. With under 1,000 tasks and one epoch of training, we saw dramatic improvements and have high confidence in further gains from adding more data. The benefits of expert-created specialized data will only increase as training stacks adapt and models become more capable. Run evals yourself on <a href="https://huggingface.co/datasets/mercor/apex-agents" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a> now.
<h1>Example prompt with responses from the baseline and post-trained models</h1>

How Mercor's data and Applied Compute's long-horizon RL unlock real capability gains for AI models

Expert data drives model performance

Today, we’re introducing <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a>, our new benchmark designed to test how well AI agents complete real, long-horizon tasks in investment banking, consulting, and corporate law.
It’s the latest addition to the AI Productivity Index (APEX), Mercor’s family of benchmarks that measure economically valuable capabilities, joining <a href="https://www.mercor.com/apex/apex-v1-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX</a> for tool-free evals and <a href="https://www.mercor.com/apex/ace-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">ACE</a> for consumer applications.
With APEX-Agents, we set out to find if today’s AI agents do economically valuable tasks. Are they ready to work with teams, in real software tools, and deliver client-ready work?
Workplace context is messy, incomplete, and spread across documents and chat threads. Tasks take hours, not seconds.
Most existing benchmarks don’t reflect that. They evaluate models on isolated prompts or narrow skills. They don’t measure whether an agent can navigate multiple workflows and produce something a manager or client would accept.
That is why we approached APEX-Agents differently. Every task is designed to mirror the complex work that professionals do and the tasks they wish an AI agent could help with. This let us identify cases where current models do not perform reliably.
<iframe src="https://player.vimeo.com/video/1156906982" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="Mercor APEX-Agents" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
<h2>Our approach</h2>
APEX-Agents simulates the demands of the profession to challenge agents to navigate instructions, manage complicated file systems, and produce outputs that justify a professional fee.
Our four-step approach:
Surveys: We started out by surveying hundreds of experts from professional services including Goldman Sachs, McKinsey, and Cravath to understand how they spend their time.
Scenarios: A team of Mercor experts—Vice Presidents, Managing Directors, and Managers with five-to-ten years’ experience at top-tier firms—worked in Google Workspace, simulating how coworkers would collaborate on a project.
This might look like a week-long consulting project for a fictitious European oil &amp; gas company focused on cost-cutting measures.
We worked with <a href="https://www.box.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Box</a> to define what a real-world file system looks like. We mapped rigorous, domain-specific challenges to the complex file structures found in a mix of industries, resulting in datarooms that truly simulate the daily workflow of a professional.
The result is a high-context workspace that mimics what professionals navigate every day.
Task creation: Using these custom scenarios, the experts defined specific tasks and the exact grading criteria that define what “client-ready” means. Each task includes 1–10 pass/fail criteria.
<a href="https://www.harvey.ai/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Harvey AI</a> provided early feedback on the design, scope, and realism of the corporate law worlds. They confirmed the complexity and value of the tasks in APEX-Agents, validating that they reflect the work of top lawyers at Fortune 500 enterprises and law firms.
Evals: We deployed AI agents inside these worlds using our open source evaluation infrastructure, Archipelago, to measure whether they finish the work correctly.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7bc182cda45d843f72fa787ae2fc5e49c2bcbd63-2756x1230.png" alt="APEX-Agents" style="width: 100%; border-radius: 10px;" />
<h2>The findings</h2>
Frontier models successfully complete less than 25% of tasks that would typically take professionals hours. With multiple attempts, performance improves but the gap remains large. Even with 8 tries, the best agents can only complete 40% of the tasks.
Many agents fail not due to lack of capability, but because they can’t manage ambiguity, find the right file, or hold context across the entire workflow.
No model is ready to replace a professional end-to-end.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f0ac6afea54245c059dc1cacb8fbe316c936e850-3840x2142.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Open source</h2>
We have released the entire benchmark open source via <a href="http://huggingface.co/datasets/mercor/apex-agents" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> with a CC-BY license and Archipelago, our infra, eval service, and set of apps, is available as an open source repo on <a href="https://github.com/Mercor-Intelligence/archipelago" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GitHub</a>.
Read the full research paper on <a href="https://arxiv.org/pdf/2601.14242" target="_blank" rel="noopener noreferrer" class="text-indigo-600">arXiv</a>.

Osvald Nitski

Mercor APEX-Agents is a benchmark designed to test how well AI agents complete real, long-horizon tasks in investment banking, consulting, and corporate law.

Introducing APEX-Agents

We created the first version of the AI Productivity Index (APEX) to assess whether frontier models can perform high-value work in Investment banking, Management consulting, Law, and Medicine. We found that even the very best models struggle on complex real-world tasks, failing to meet the production bar.
Mercor has now doubled the size of the heldout evaluation set in APEX from n=200 to n=400. This larger eval set allows us to more consistently evaluate frontier models’ ability to perform tasks that create economic value. The design of the cases remains the same (comprising prompts and source documents, and a grading rubric) but we have increased their complexity and variety. On average, tasks in APEX take over two-and-half hours for seasoned professionals to complete in the real-world. The contributors to APEX typically had over 7 years of experience.
We have also refined the evaluation methodology in APEX, increasing the number of runs we execute to 8, simplifying the grading process, and adding confidence intervals to our results. Read the <a href="https://arxiv.org/abs/2509.25721" target="_blank" rel="noopener noreferrer" class="text-indigo-600">technical report</a> to find out more about our approach to evaluation and how the dataset was created.
<h2>Results</h2>
The best performing model (GPT 5, Thinking = High) has the highest mean score at 67%, followed by Gemini 3 Pro (Thinking = High) at 64.3% and Grok 4 at 63.5%. Models&#39; scores are lowest on Investment banking, with the highest scoring model achieving 63.0%, followed by Management consulting (top score = 64%), Medicine (top score = 65.5%), and, with substantially higher scores, Law (top score = 77.9%). APEX shows that models struggle with real-world tasks that professionals undertake everyday. The new leaderboard is available <a href="https://www.mercor.com/apex/apex-v1-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a>.
Gemini 3 Pro (Thinking = High), released in November 2025, is a substantial improvement on 2.5, beating it out by over 5 percentage points overall, and on Investment banking by 9 percentage points. Similarly, we see Opus 4.5 (Thinking = On), also released in November, beating Sonnet 4.5 and Opus 4.1 by 6 percentage points and 12 percentage points respectively. These substantial improvements speak to a meaningful step forward in models’ ability to perform high-value tasks. GPT 5.1 is an exception and does not improve on GPT 5 (which remains the leaderboard champion), but this is perhaps not surprising given that 5.1 is primarily meant to be more conversational and explanative, rather than better at complex reasoning. Finally, given it is the only model without an explicit reasoning setting, Grok 4, is remarkably strong and comes 4th overall.
If you want to add your model to the leaderboard or run a loss analysis, contact the Mercor Applied AI Research team at apex@mercor.com.
<h2>Open source eval set</h2>
We are releasing n=100 open-source cases (APEX-v1-devset) on <a href="https://huggingface.co/datasets/mercor/APEX-v1-extended" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> with a CC-BY licence for anyone to train on, evaluate against, and research. The open-source cases were created by the same annotators through the same production pipeline. We are also open-sourcing our <a href="https://github.com/Mercor-Intelligence/apex-evals/tree/main/apex-oss-eval" target="_blank" rel="noopener noreferrer" class="text-indigo-600">evaluation harness</a> so you can exactly match our grading approach. We are looking forward to seeing what the community builds and would love to hear from any researchers using our data.

Doubling the size of the AI Productivity Index to better measure AI models’ economic value.
