SWE-Marathon-Ext: Can AI build a whole SaaS product?

Most existing coding benchmarks do not test the full scope of work required to ship a product; they give models an existing codebase and ask for a localized change, rather than requiring them to infer and implement the system’s behavior from scratch. We wanted to extend this scope: what happens when a model has to build an entire software product itself?

SWE-Marathon-Ext builds on SWE-Marathon, a benchmark with 20 long-horizon software engineering tasks spanning library reproduction, full-stack application cloning, and algorithmic optimization. SWE-Marathon includes five product-clone tasks, but the paper explicitly defers analysis of those product clones to follow-up work.

SWE-Marathon-Ext adds to this unfinished part of the benchmark with a dedicated suite of 12 SaaS product clones. We chose widely used enterprise applications spanning scheduling, CRM, project management, HR, commerce, analytics, automation, support, design, and collaboration, capturing both common day-to-day business workflows and a diverse set of product-specific engineering challenges. Each model starts from an empty workspace and constructs a complete application from a product specification. The suite covers Calendly, Figma, HubSpot, Linear, Notion, Rippling (two workflows), Shopify, Tableau, Zapier, Zendesk, and Google Docs.

We evaluate eight frontier models and find several critical failure modes. Models often build the basic surface of a product, including authentication, CRUD, and workspaces, but struggle to implement the product-specific logic that makes those features work correctly together, including concurrency, time logic, validation and rejection paths, API behavior, and interface wiring. The gap shows up across test categories for all model rollouts: authentication features pass 79% of tests, validation 63%, CRUD 53%, concurrency 50%, time logic 43%, and UX tests at 32%.

Our findings suggest that strong performance on repository-based coding benchmarks may overstate how reliably today’s models can build complete products end-to-end.

While models are capable of delivering polished user interfaces, their performance at building out core logic varies by model family.

Setup

We focus on product clones because they test one capability that repository-based SWE benchmarks largely avoid: building the product entirely from scratch. A mature repository already contains much of the product substrate, including timezone handling, uniqueness constraints, and error behavior, so an agent editing existing code can inherit those decisions. A model starting from an empty workspace has to infer and implement that logic from the specification alone, making this a useful test of whether frontier models can handle long-horizon software work autonomously.

We evaluate eight frontier models from five developers using the Terminus agent on the Harbor harness, the same containerized infrastructure and grading system used by SWE-Marathon. Every model completes three independent rollouts per task, producing 288 rollouts in total, all of which pass a four-stage audit covering anti-cheat checks, timeout versus infrastructure-failure classification, token-floor sanity, and grader validity.

What a task looks like

As in SWE-Marathon, the agent works autonomously in a Linux container with shell access, starting from an empty workspace, and its final build is the graded artifact. Each task consists of a product specification, a golden solution, and two independent test suites, all authored by domain experts. The specification names a well-known product, fixes the operational contract, including the port, health check, persistence requirements, and start script, specifies the API surface in detail, and leaves architectural decisions to the model.

The test suite contains both deterministic and computer-use agent (CUA) tests.

• The deterministic test suite executes a pytest suite (39 to 187 tests per task) written against a golden solution: each test passes or fails. Deterministic tests are grouped into themed suites (test_auth, test_convergence) as per the SWE-Marathon convention, which localizes where within the product that performance is lost.
• The CUA test suite evaluates each model rollout through the user interface. For each rollout, Harbor’s Computer1 agent opens the completed application in a live browser and works through the user journeys defined in the task’s rubric.json, using screenshots to perceive the interface and click, type, and navigation actions to interact with it. Each weighted criterion receives a pass or fail based on whether the required functionality can actually be completed through the interface, with screenshots and the agent’s interaction trace recorded as evidence.

Before a task is accepted, experts run the golden solution through the same Harbor harness used for model evaluation, and it must score 100% on both suites across three independent grading runs on the same model rollout.

Anatomy of one task: the model sees only the spec (instruction.md). The hidden pytest suites, computer-use agent scripts, and rubric score the completed rollout.

Each task product also embeds one hard axis: the deliberately difficult capability that defines the product’s core value. The remainder is standard SaaS scaffolding, including authentication and sessions, CRUD on core entities, listing and pagination, and workspace setup. The table below lists the hard axis for each of the twelve tasks

The hard axis embedded in each of the twelve tasks.

Results

Figure 1: Leaderboard

Mean overall pass rate per model, averaged over three rollouts per task (mean@3). Higher is better.

By mean@3 score, GPT-5.6 Sol leads at 74%, with GPT-5.5 close behind at 73%, Grok 4.5 at 67%, and Claude Opus 4.8 at 65%. Performance then drops sharply to Gemini 3.5 Flash at 42%, Gemini 3.1 Pro at 34%, GPT-5.6 Luna at 28%, and GLM-5.2 at 27%.

Even the strongest models leave important product features incomplete or incorrect. GPT-5.6 Sol’s 74% mean@3 means that, on average, about one quarter of the evaluated behavior still fails. Our later analysis shows that these misses correspond to product failures that can make an application unreliable or unusable for its intended business workflow, such as double-booked time slots, oversold inventory, or invalid requests being accepted.

The scores also span a wide range, from 74% to 27%. That separation is important because frontier models often score much closer to one another on established benchmarks such as SWE-bench Verified, making it harder to distinguish models that may behave differently in practice. Hence, SWE-Marathon-Ext 47 point spread makes the benchmark useful for categorizing model capabilities on long-horizon, end-to-end software construction.

Figure 2: Deterministic vs CUA Pass Rates per model

For each model, the mean deterministic score (backend API correctness) against the mean CUA score (whether a computer-use agent can use the user interface), both as percentages.

For seven of eight models, mean CUA scores are lower than mean deterministic scores. Across all eight models, the difference is statistically significant (paired t-test, p = 0.026), with CUA lower by 7.6 percentage points on average. This suggests that the two evaluations may capture different properties of the same build, a pattern that also appears at the level of individual models and tasks. On HubSpot, for example, GPT-5.5 passes roughly 85% of deterministic checks, correctly implementing much of the underlying CRM logic for contacts, deals, pipeline stages, and lead scores, but only about 14% of CUA criteria because those capabilities were not reachable through the interface. GPT-5.6 Sol shows the reverse pattern: workflows appear to succeed in the UI while deterministic checks indicate that the underlying state change is incorrect, incomplete, or sometimes does not occur at all. Taken together, these cases suggest that using both evaluations can provide a fuller picture of whether functionality is both correctly implemented and usable through the interface.

Figure 3: Basic functionality vs hard axis, pass rate per task

For each task, we compare the test group models handled best (basic surface, grey) with the task-specific capability they handled worst (hard axis, indigo), across all graded rollouts.

Each task has two associated bars, aggregated across all model rollouts. The grey bar shows performance on the basic surface: login, workspaces, and CRUD. The indigo bar shows the hard axis, the task-specific capability central to the product’s core business function. The gap between the two is substantial across tasks. Calendly builds pass 84% of basic-surface tests but only 33% of scheduling tests; Shopify drops from 72% to 8% on concurrent checkout; Notion drops from 75% to 2% on validation. Zapier and HubSpot fall to 0% on their hard-axis tests. Overall, models are much more reliable on standard SaaS functionality than on the task-specific capability the product is designed around.

Where builds break: the difficulty gradient

Pass rate (%) by app and verifier axis, averaged across eight models. Apps ranked easiest to hardest (top to bottom); axes easiest to hardest (left to right). The darker-to-lighter drift shows that models clear Login and Validation nearly everywhere and fail Concurrency, Time logic, and UX, where most SaaS complexity lives. Notion and Shopify are the floor of all eight apps; login is the one column strong top to bottom.

Three failure modes drive the loss

When we dug into why builds fail the hard axis so consistently, three patterns emerged. Each cell in the table below depicts the number of failing deterministic checks over the total number of deterministic tests for that mode, aggregated across a model’s rollouts (3 rollouts per each of 12 tasks):

Concurrency and idempotency

Concurrency and idempotency failures happen when two requests modify the same state at once, or when the same request is processed twice. Across model rollouts, we observed builds failing to protect against these cases with transactions, locks, or idempotency keys.

In Shopify, for example, two concurrent checkout requests race for the last unit and both orders are confirmed, thus overselling the unit, as observed in 22/24 Shopify rollouts (8 models, 3 rollouts each). Calendly shows the same pattern, with two invitees booking the same time slot in 16/24 rollouts.

We saw this failure mode frequently in tasks like Shopify and Zapier, where inventory and durable job execution are core functionality, and much less often in Figma, at 4/24 rollouts, where most operations do not contend over the same state.

Time, timezone, and scheduling

Time-based logic is the most prevalent failure mode across all model rollouts because correctness depends on product-specific rules: timezone conversion, daylight-saving changes, business hours, tenure thresholds, etc,

In Calendly, model rollouts compute open slots through local clock times without checking a timezone database, thus offering a slot during an hour that does not exist on the spring-forward day (in 18/24 rollouts). In Rippling, small errors in the accrual rate at a tenure threshold affected later tests that depend on the employee’s balance, appearing in 17/24 rollouts.

These failures were most common in products with substantial time-based logic, such as Calendly, Rippling, and Zendesk, with 15/24 Zendesk rollouts also showing time-related failures, and were rare in products like Figma, at 3/24 rollouts, where time plays little role in core functionality.

Unhappy-path contracts

The third failure mode is incorrect handling of invalid or disallowed requests. Most builds can handle valid input. The harder question is whether they return the correct error, enforce limits and permissions, and reject malformed data.

For web APIs, client errors should usually return a 4xx status code: 404 for a missing record, 409 for a conflict, or 422 for a validly formed request that violates a constraint. Weak builds instead return 200 as though the request succeeded, or 500 as though the server failed.

In Notion, for example, we saw model rollouts accepting a malformed date or invalid multi-select value instead of rejecting it in 23/24 rollouts. Figma exposed a failure where accounts in model rollouts could access private workspaces belonging to other accounts (in 18/24 rollouts). Unlike the first two failure modes, this pattern appeared across nearly every product because every API has an error contract.

Why this matters

Across twelve zero-to-one product tasks, frontier models reliably build standard SaaS functionality such as authentication, workspaces, and CRUD flows, but struggle with the task-specific behaviors that make each product correct and usable.

These results suggest a next step for evaluating long-horizon SWE tasks: stress-testing completed products under sustained, multi-user interaction, where concurrent actions and conflicting state changes can expose failures that fixed test cases may not. Future evaluations could use agentic swarms or simulated users with varied goals and behaviors to generate this traffic, for example by having two shoppers compete for the same inventory, multiple users update the same record, or overlapping bookings contend for the same time slot. Thus, as coding agents take on increasingly complete software projects, the decisive question for evaluation shifts from whether they can build a product to whether that product holds up under real conditions of load and use, the point at which it becomes usable and capable of delivering real business value.

Acknowledgments: Shoutout to our friends at Abundant for coming up with a novel benchmark. Thank you to Bertie Vidgen and the broader Research team for their thoughtful feedback and guidance throughout the review process. Thank you also to Sameer Bhadouria and Aravind Selvan for their editing and feedback, and to the broader Code Data Team for helping ideate and shape the task design behind SWE-Marathon-Ext. We are grateful for the many iterations and discussions that helped refine both the benchmark and this analysis.