How Mercor uses Codex to build a production SRE agent with 1Password
Building an AI agent is no longer the hard part. Foundation models can already perform many of the individual tasks required in enterprise workflows. The challenge is deploying systems that can be trusted in production by understanding how work is performed, defining what success looks like for that organization, integrating with existing enterprise environments, and continuously measuring performance over time.
That was the challenge 1Password and Mercor set out to solve when building an SRE agent for 1Password’s engineering organization. Mercor used Codex as the execution layer for an SRE agent that investigates alerts across monitoring systems, deployment history, source code, feature flags, documentation, and past incidents, then surfaces findings for engineers to review. Using Codex, Mercor built the customer-specific context, integrations, and evaluation framework needed to deploy the agent through Mercor Enterprise AI Suite.
An SRE agent operates across boundaries when it traverses monitors, repositories, deployment history, and incident records. The leadership task is not simply to ask whether the model is capable. It is to define who owns the workflow, which systems the agent can reach, what evidence it must return, and which decisions remain human. With Mercor and Codex, we are accelerating incident investigation while keeping consequential production actions human-owned and evidence-backed.
Nancy Wang, CTO, 1Password
Understanding 1Password’s Environment
SRE spans monitoring systems, deployment history, source code, documentation, historical incidents, and cross-team engineering conversations. Much of the required context exists across multiple systems, making it time-consuming to find the relevant information, connect it across sources, and quickly assemble a reliable picture of an incident. Mercor’s Forward Deployed Engineers worked with the 1Password team to map investigation patterns and capture relationships between systems, documentation, and services in a context graph reflecting how engineers actually investigated incidents.
1Password and Mercor collaborated to define how the agent would integrate into 1Password’s existing engineering environment. Mercor Enterprise AI Suite supports SaaS, customer-managed, in-VPC, on-premises, BYOC, and BYOK deployments, allowing the deployment architecture to match 1Password’s infrastructure without changing the underlying evaluation methodology.
Developing the Evaluation Suite
Many AI projects treat evaluation as the final step. An agent is built first, tested against a handful of examples, and refined only after problems begin to appear. That approach can work for demonstrations, but it breaks down in production, where success depends on consistent performance across a broad range of real-world scenarios.
Before optimizing the agent, Mercor built a Mercor Enterprise AI Evaluation Suite based on real engineering workflows. The teams drew on more than 300 historical incidents to develop evaluation material, connecting the original alerts with their root causes and remediation. Every scenario defined the task, expected behavior, and success criteria, creating a benchmark against which every subsequent change was measured.
Exhibit 1: Mercor Representative AI Eval Suite
Every change to the agent was evaluated against the current regression suite. Missed context, incomplete recommendations, and retrieval failures became new evaluation cases, allowing the system to improve through measurable iteration rather than intuition.
With the evaluation framework in place, development became a continuous optimization loop. The teams evaluated each iteration against the current regression suite and used the results to decide which configurations to promote to production.
This SRE agent is not just an agent that can investigate incidents. It has to investigate incidents the way 1Password does, across its monitors, deployment history, code, and incident systems. We evaluated the task, the trajectory, and the output: did it look at the right evidence? Reason across systems? Produce a recommendation an engineer could act on? Codex gave us the execution layer, and the eval suite gave us a quantitative way to improve it before each change reached production.
Adarsh Hiremath, Co-founder and Co-CEO, Mercor
Building the SRE Agent
Most of an agent’s incident investigation is the same work an engineer performs at a terminal: pulling up the monitor that fired, reviewing any recent PRs, checking which feature flags changed, and often revisiting the postmortem from the last time something similar broke. Codex is built for that work: long-horizon, tool-heavy, code and terminal environments. Mercor used Codex as the execution layer for the SRE agent.
When a monitoring alert fired or an engineer declared an incident, Mercor’s event integration started a Codex investigation. The agent used read-only access to systems including Datadog for metrics, logs, events, monitors, and service ownership; GitHub for diffs across roughly 840 active code repositories, FireHydrant for past incidents and timelines, Unleash for feature flags, PagerDuty for alert context, and sometimes Sentry for errors. It also consulted engineering documentation and used local shell tools to correlate evidence across these systems and investigate recent changes. The assessment identified the likely affected service or failure mode, the responsible team, the most likely cause, and the first action an engineer should consider. Engineers retained responsibility for every production decision and action.
Exhibit 2: How the SRE agent turns incident signals into evidence-backed findings for human review, and how it continuously improves.
Codex was a strong fit for the SRE agent because it supports long-running, tool-intensive investigations across code and terminal environments. Mercor kept the orchestration, 1Password-specific context, integrations, and skills in a versioned layer around the Codex executor. Mercor and 1Password evaluated each configuration against the same evaluation suite before advancing it to production. This design also gave both companies’ engineering teams a repeatable way to evaluate new Codex releases and model versions before adopting them. Both companies found that Codex provided an easy-to-use foundation to evolve the agent quickly and safely.
Deploying Into 1Password’s Environment
Enterprise deployment means the system must integrate with existing infrastructure, governance, and security controls.
For 1Password, credentials remained securely managed within the organization’s existing credential management system and were retrieved dynamically at runtime rather than embedded in the agent. The agent produced a structured assessment of the likely affected service or failure mode, the responsible team, the most likely cause, and the first action an engineer should consider. Engineers could review every recommendation before taking action, while the evaluation framework continued validating improvements after deployment.
Exhibit 3: Mercor Enterprise AI Suite in Production
The evaluation suite started with a subset of real historical incident scenarios. Every scenario defined the task, expected behavior, and success criteria. Failures and newly encountered scenarios became new evals in the suite, and both teams treated development as a continuous optimization loop.
Evaluation results and early production observations
The goal was not to prove that the agent could solve one incident. It was to measure whether the same configuration performed consistently across different failure patterns. The shipped configuration was evaluated in three independent trials against each of 26 historical-incident tasks, for 78 runs total, earning 95% of the grading criteria. The rubric measured evidence gathering, root-cause reasoning, affected-service identification, distractor avoidance, and output quality. This was an evaluation score, not a measure of production accuracy.
Trust depended on more than a plausible answer. Every cited metric, pull request, and log query in the evaluated reports could be traced to a tool call the agent had actually made. No fabricated citations were observed in the defined evaluation set.
In a retrospective before-and-after comparison using historical production incident data, estimated median time to mitigate was 57% lower in the post-deployment cohort. The comparison included 199 earlier incidents and 32 post-deployment incidents and provides a directional comparison, not evidence of causation.
The clearest early production observation was speed to its findings: a median of under 4 minutes across 30 live agent-generated incident reports. These results are preliminary and not a controlled measure of production impact.
1Password moving the agent to production
Over a median time of under 4 minutes, for an average cost of under $2 the agent runs roughly 50 queries across multiple systems covering metrics, logs, monitors, past incidents, on-call, feature flags and code changes across some 840 repositories, then posts a written report in the incident channel.
During an incident, the agent greets the responder with a first pass across metrics, logs, monitors, code changes, flags and past incidents, or provides it within minutes. Most of the responder’s resolution time is spent finding the change, and that is the part the agent front-loads.
Setting authority before granting access
The 1Password SRE team defined the agent’s authority before launch. It could read the systems responders use to investigate an incident, but engineers retained control over declaring incidents, paging people, rolling back releases, running fixes, and merging code. We also run the agent in a secure container that prevents the agent from changing its own environment.
To illustrate one access decision, our team declined write access to the incident-management tool because it would have also allowed the agent to page people. The tool offered no way to grant write access without that paging capability, so we kept the agent’s access read-only.
Every investigation is also bound by a $3 budget.
Taking ownership of the service
When the agent entered production at 1Password, it began investigating declared incidents and monitor alerts routed to it, posting findings directly into the incident’s Slack channel. Six days later, we opened access to the entire engineering organization.
When Mercor’s engagement ended in late August, 1Password’s Observability team took full ownership. Between July 30 and September 24, our merged 85 changes spanning accuracy, output, evaluation, and operations. The handover made the ongoing commitment clear: the agent needed an accountable team to maintain it, inspect its work, and improve it as the environment changed.
Turning wrong answers into tests
Reviewing production investigations exposed a recurring weakness. We compared the agent’s conclusions with the findings in subsequent incident reviews. The agent too readily blamed a recent code change, including cases where nothing had changed.
A plausible but wrong explanation can delay responders. Upon noticing the pattern, we required every report to state explicitly whether a change caused the incident and graded that answer against the incident review. Our team turned a recurring reasoning failure into a specific requirement it could test. That gave engineers a way to track the behavior across future changes.
Continuous improvement after launch
We built three layers of review into ongoing operations to answer three different questions:
- A benchmark drawn from past incidents
- Could the agent repeat its performance?
- Retrospective grading against completed incident reviews
- Did its conclusion match what actually happened?
- Engineers reading the agent’s output:
- Could a responder use the report?
Since September 9, we traced every production investigation end to end and scored it automatically. Those traces let engineers inspect how the agent reached an answer, including which evidence it gathered. Incident reviews provide a later check against the eventual findings. Together, these mechanisms give us a way to identify misses and turn them into the next evaluation case.
Writing for all levels of responders
Our SRE team also rewrote the output (delivered in Slack) in plain English so responders could act on it without needing deep familiarity with the affected system. Reports include the agent’s level of confidence and identify any evidence that would change its conclusion. The rewrite makes the agent’s reports more accessible to all engineers, including those who might be less familiar with internal names.
When the responder starts reading, a first pass across metrics, logs, monitors, code changes, flags and past incidents is already there or arrives within minutes. Finding the change is where most of the resolution time goes, and that is the part the agent front-loads.
What’s Next
As of September 24, the agent remained in beta while handling real investigations on routed incidents. The team reported a cost of approximately $1.90 per investigation. Operating it also required the continuing engineering work behind those 85 changes. The team’s next priorities were to widen live grading and sustain the accuracy gains.
More broadly, 1Password will continue using agents in production. 1Password’s standard for agentic incident response is clear: respond quickly, show the evidence, stay within the permissions, keep review and action in human hands, and improve when engineers find a miss. That standard gives the team a basis for deciding where the agent should take on more work.
With Codex as the execution layer and evaluations guiding each production update, 1Password now has a repeatable way to improve the agent, validate new configurations, and adopt future model upgrades without rebuilding the surrounding system.