Offline Evaluations in Practice: Improving Mercor’s SRE Agent

At Mercor, we run Site Reliability Engineering (SRE) agents that respond to production alerts. We had two agents, Medic and Momo, handling the same alerts, but no reliable way to tell which performed better. Live traffic made comparisons difficult because every alert was different, and there was not a consistent way to grade the results. To solve this, we built an automated process for creating offline evaluations from real incidents. Using the same underlying model, Medic scored 0.79 on average compared with Momo’s 0.55. We retired Momo and retuned our alert system, reducing cost per alert from $7.18 to $2.26, making it more than 3x cheaper. Today, each closed incident becomes a new evaluation task that helps us improve Medic over time.

What an offline evaluation is

An offline evaluation measures an agent’s performance under a fixed dataset and a defined scoring method. The world is a frozen copy of the information the agent could have used. The task is the agent instruction and the outputs required. The grader scores those outputs against an answer key that humans have confirmed afterward.

Figure 1. An offline evaluation: world, task, agent, grader

Building offline evaluations from real incidents

As our platform’s DAU increased so did the number of incidents we were receiving. This prompted us to rely on agents for initial incident triaging and opportunities to improve them.

Each task for our offline agent starts as a real incident that on-call engineers closed with a written root-cause analysis (RCA) and post-mortem. The agent gathers the available evidence, and the on-call engineer investigates the incident and confirms the root cause.

After an incident, the on-call engineer replies to Medic’s own alert thread with a single request including the incident number, root cause, post-mortem, and mitigation strategies. From there Medic builds the offline evaluation task in six steps.

Figure 2. From a closed incident to a graded trial

How a score is computed

Once the task is created, we task the agent to write three files for grading: a structured decision, the observations it would make to Medic’s coordination board, and a briefing for the on-call engineer. The grader scores them in four steps.

Figure 3. How a trial is scored

An incident’s score is the average of its graded checks. Not every incident grades every check: if the review never establishes who should have been paged, that check is left out rather than guessed.

Keeping the offline evaluation honest

A score is only as good as the task behind it, so before an incident becomes a task, we ask four main questions about it.

Figure 4. Four checks before an incident becomes a task

World integrity. To prevent answer leakage and hindsight bias, we freeze the evidence at the responder’s last action and exclude anything learned afterward. We include all data available to the agent at that point, even if it wasn’t queried during the original investigation. This lets us evaluate not just how the agent interprets evidence, but whether it identified the right evidence in the first place.

Self-contained and hard to game. The agent must complete the investigation using only the evidence provided. It cannot access the answer key, grader, or live systems. To prevent shortcuts, the grader assigns a zero if required outputs are missing or if the agent cites numbers not found in the evidence. This ensures the agent is scored on its investigation rather than its ability to guess or avoid answering.

A robust grader. Different investigations that reach the correct conclusion should receive the same score. To ensure consistent grading, we evaluate the agent’s final outputs using deterministic checks wherever possible, with an LLM judge reserved for root-cause analysis. Before adding a task to the evaluation suite, we test the grader against known correct and incorrect answers. The correct answer must score 1.0, and each incorrect answer must fail only the check it was designed to test.

An unambiguous task. The agent needs clear instructions so that differences in performance reflect its ability to investigate, not confusion about the task. We use the agent's production prompt and specify the exact output files, format and schemas required for grading.

Evaluation Suite: The five incidents

Our evaluation suite started with five recent incidents, each closed by on-call engineers with a written review. Together they cover a range of failures and severities, from a full outage to a false alarm:

• Shared storage full (Sev 1): A storage volume shared by every runner hit its size limit, and all runs stopped for about an hour.
• Feature flag rollout (Sev 2): A flag turned on for the whole organization added a new launch check, and agents built on custom images failed it.
• Runs stuck in the queue (Sev 2): A new enrollment hold refused every new run, so runs were accepted but never started.
• Records a page couldn’t read (Sev 2): Runs moved to a new storage format were missing a base record, and the page showing them refused to load them.
• False alarm (Sev 4): One customer queued over 400 runs that drained normally under their own limit, but the monitor treated it as a platform outage.

Five incidents with one attempt each is a small sample, so we read the results below as a direction, not a final measurement.

What we found

Figure 5. Score by incident, Medic vs Momo

The performance gap came largely from differences in the agent’s available skills and integrations, rather than from the underlying model itself. Medic scored higher on four of five incidents and tied on the fifth. The largest differences were in incident classification and mitigation recommendations, where access to the right tools and context mattered most.

Figure 6. Share of incidents where each check passed

Access to the right skills and integrations made a significant difference. Even when an agent identified the correct issue, missing tools could prevent it from recommending an effective mitigation. For example, Medic lacked access to PostHog, which prevented it from recommending a feature flag change to resolve one incident.

Ways to improve an agent

A failed check tells us where an agent fell short, but there is more than one way to respond. Because the tasks are frozen, we can make a change and rerun the same incidents, so we know the change caused the difference and not a quieter week of alerts.

Figure 7. Ways to change an agent, each retested on the same tasks

Taking action on evaluations

Beyond comparing performance, evaluations helped us identify opportunities to reduce costs. The two agents had similar operating costs per alert: $1.86 for Momo and $2.06 for Medic.

Figure 8. Cost of the on-call agents

With Medic outperforming Momo, we retired Momo and reduced weekly costs from $8,295 to $345. Building the evaluations also helped us identify noisy alert rules, which we retuned to reduce Medic’s monthly alert volume from an average 1,156 to 153. Together, these changes brought cost per alert down from $7.18 to $2.26, making it more than 3x cheaper.

From evaluations to continuous improvement

Every incident our engineers close now becomes a new evaluation task. The results help us identify where Medic needs improvement, test changes, and improve its performance on future alerts. Those alerts generate new incidents, continuing the cycle.

Figure 9. The loop: incidents become evals, evals improve agents, agents handle new incidents

Conclusion

When a new model comes out, the first question is usually how it ranks. But a score alone tells us little about how to improve an agent. That’s where offline evaluations become useful. By building evaluations from real incidents, we can test agents against verified root causes and the evidence available at the time. When an agent fails, we can identify whether it lacked the right skills, instructions or access to evidence, then test changes in the same incidents before deploying them. As we add new incidents to the evaluation suite, we can continue improving Medic’s ability to respond to production alerts and keep our platform reliable.