Introducing Physical-Engineering’s Last Exam (PLE) with Google Cloud AI Research

Humanity's Last Exam tested knowledge. Agents' Last Exam tested digital work. The next question is whether AI agents can do the work that builds the physical world.

That’s why Mercor partnered with Google’s AI & Infrastructure research team to release Physical-Engineering's Last Exam (PLE), a benchmark of 85 long-horizon tasks in mechanical, electrical, civil, and structural engineering. Licensed engineers from Mercor's network of experts wrote and validated each task end to end, and every task runs in an environment built on Archipelago, Mercor's open-source environment framework.

The best scoring agent configuration, Claude Opus 5.5 in Claude Code, reaches 48.3% on the hidden test set, at an average of 27 million tokens and 63 minutes per task.

Where Physical Engineering and AI meet

Most agentic benchmarks measure office work like building a financial model, reviewing a contract, drafting a deck, or working through an inbox. Our APEX family of benchmarks, including APEX-Agents, APEX-Accounting, and APEX-SWE cover these kinds of professional tasks.

Physical engineering is harder to measure. Models that score well on knowledge-work evals don't necessarily know how to size a switchboard, check a roof deck against code, or lay out a campus. Academic benchmarks don't cover these kinds of engineering tasks well. Frontier models may score above 90% on benchmarks that test textbook questions or pose self-contained problems. However, real engineering jobs are more complex and require a different kind of benchmark.

We're excited to keep building PLE with Google because AI will help design the grids, fabs, and infrastructure the next century runs on. Until now, the industry has measured AI on office work closely and on physical engineering barely at all.

Engineering work has unique requirements

Engineers spend most of the day at a desk. They read PDFs, work in spreadsheets, answer email, and mark up drawings. On the surface it's the same office work that existing benchmarks like APEX-Agents already cover.

What's different is the output, for example, a consultant's deck gets judged in a meeting whereas an engineer's design gets stamped, built, and carries load for decades. It has to obey physics and building codes, plus a licensed reviewer has to be willing to sign it. In software, a 90% solution can ship as a beta. In physical engineering, a 90% solution fails review, or worse.

PLE grades against this standard. An agent earns points for cost efficiency only after its design clears every hard safety and feasibility check. Our results suggest that even frontier models cannot yet be fully trusted with physical engineering work. The top agent's designs met 91% of individual constraints on average, but only 41% cleared every hard check. Where accuracy and reliability are both important, frontier models do not yet pass the par.

That’s because while the tools are similar, dealing with the physical world increases stakes and complexity.

Benchmarking across four disciplines

PLE covers four core engineering disciplines, so a model can't score well by being strong in just one specialty:

• Mechanical (30 tasks): design a cooling loop and a clash-free 3D equipment hall
• Electrical (27 tasks): size a switchboard and its cable schedule
• Civil (18 tasks): lay out a full campus, including grading and utilities
• Structural (10 tasks): design a roof deck

Each discipline has its own codes, conventions, and failure modes. This means that an agent that's strong on electrical load calculations might still struggle on a site grading plan.

Tasks written by real engineers

We asked licensed engineers from Mercor's expert network to build tasks from their daily workflows. For the 44 expert-authored tasks, these experts built everything themselves including the project folder, the prompt, and the grading rubric.

Google built the other 41 tasks with an adversarial loop. First, a Creator agent drafts a task, then a Solver agent tries to break it, and the Creator hardens it in response. Engineers validated every task from both paths, so each one has been confirmed by a human expert as realistic and hard enough.

Building worlds that look like a real project

Producing quality engineering work with AI requires more than a good prompt. When an engineer joins a project, nobody hands them a clean problem statement. They’re likely to inherit a folder of earlier revisions, half-finished calculations, client emails with shifting requirements, and drawings that disagree with the spec.

PLE's environments are built to match this kind of complexity. Each task runs in a containerized project folder with about 61 files, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models, and outdated revisions planted as traps. The agent has no internet access and up to six hours to find the governing codes and physical limits, run the calculations, and deliver something a licensed reviewer would accept. Much of the difficulty is in figuring out what the actual problem is.

What the results show

Working with Google, we ran 14 frontier models across seven agent harnesses. Claude Opus 5.5 leads at 48.3%, followed by GPT 6 Astra at 45.8% and Gemini 4 Argon at 40.6%. Choice of model matters far more than choice of harness. For example, scores across models span up to 50 points, while the same model on different harnesses moves a median of about 3.

Get involved

We developed PLE as an open benchmark, and invite the community to participate. Google has released 29 public tasks, the grading tools, and the evaluation container. The open competition runs on the hidden test set. Thirteen Google-sponsored models can be run for free, and prizes will be announced soon. Bring a harness, read the paper, or contribute a task.

Agents are getting good at office work. Engineering office work holds them to a harder standard, and we're proud to have helped build the benchmark that measures it. Read more about this collaboration in Google's blog post.