Playing well with others: Putting AI in the org chart
AI models are rapidly improving on benchmark tasks, and that progress is often read as an affirmative answer to a bigger question: can AI do your job?
But benchmarks don’t measure jobs; they measure tasks. And these tasks are often solved alone by the AI agent with all the required inputs already present in the environment.
By contrast, most jobs – and therefore most economically valuable work – take place within companies, which are made up of people who must coordinate, decide and negotiate to get things done. Understanding how AI transforms work and productivity means accounting for these organizational elements; jobs are messy in a way that tasks are not.
In a new paper with Nicolai J. Foss and Henrik M. Nielsen of Copenhagen Business School, we develop a vocabulary for this messiness, which we call organizational difficulty to differentiate it from the technical difficulty of individual tasks. We decompose organizational difficulty into organizational dimensions such as navigating authority structure and task interdependence. While preliminary, we intend this taxonomy to be an organizational counterpart to O*NET’s task database, so researchers can describe not just a task but the organizational setting it sits in.
We then apply this taxonomy to assess how prominent AI benchmarks cover organizational difficulty. The results are striking. Fewer than a quarter of these benchmarks even claim to represent any organizational elements in their tasks, and only 16 percent separately report how well models actually perform on an organizational element.
We hope this organizational taxonomy can be a foundation for both benchmark researchers measuring models’ organizational capabilities and economists thinking about their economic value. This measurement matters more as AI agents join organizations as coworkers, such as Claude Tag or Devin, and take on the same organizational demands people face. It matters too as agents form organizations of their own.
What makes a job harder than the sum of its tasks
Consider a software engineer who is asked to fix a customer-reported bug. The task itself – fixing the bug – may require only a few lines of code. But the bug logs required to verify the code fix are owned by the support team, who are hard to reach. The fix touches code that the billing team owns, so it needs their formal approval. The product manager isn’t even convinced the bug is a bug at all. And, to top it all off, the account manager has already told the customer it will be fixed this week.
Swipe horizontally to inspect the full network.
Organizational difficulty
The task inside the job.
The focal —working with code, logs, runtime, and tests—is embedded in a network of interactions inside an organization. Completing it depends on people who hold information, define the goal, authorize changes, and make commitments.
Here, the software engineering itself is quite easy. The difficulty arises from how the task relates to other people, such as who holds the relevant information and who needs to sign off.
We call the difficulty that comes from how a task relates to other people and activities organizational task difficulty. By contrast, the difficulty of the task itself, once the goal, the inputs and the authority to act are given, is technical task difficulty. The two are orthogonal. A difficult math problem can be organizationally simple, whereas the bug fix is technically simple but organizationally hard.
This matters for AI’s economic impact. Models have improved rapidly on technically difficult, well-specified tasks, but without corresponding improvement on organizational ones, the productivity gains from model progress may be limited.
Mapping organizational difficulty
To measure either kind of difficulty, we need a way to break the concept down. For technical difficulty, O*NET offers a starting point: it describes the tasks in each occupation and the skills, knowledge and abilities they draw on. Organizational difficulty has no comparable vocabulary. We propose one, drawing on the concepts that organization theorists use to study how work is divided and coordinated.
The taxonomy has four layers, shown below. The structure of work describes the conditions an actor faces when doing a task. Those conditions create coordination requirements, which organizations meet through governance mechanisms, and how well they are met is judged by performance outcomes.
Consider the bug fix again. All of the organizational problems described above, such as distributed information or mixed authority structures, are part of Layer 1, the structure of work. These create coordination requirements: for example, the engineer has to get the logs from support and agree on timing with the account manager. The company meets those requirements through governance mechanisms, such as a support ticket queue and a release calendar, respectively. Finally, performance is judged not only on whether the bug is resolved, but also on whether the correct approvals came through and the fix shipped by the date promised to the client.
A blind spot in how we benchmark AI
After formalizing this theory of work, we wanted to see how well benchmarks represent it. To do so, we labeled around 500 benchmarks from three categories:
- Industry indices: benchmarks included in the Artificial Analysis Intelligence Index and Epoch AI’s Benchmarking Hub1
- Top conference papers: benchmarks published as oral, spotlight or award papers at the major machine learning conferences (NeurIPS, ICML, ICLR, ACL, EMNLP and NAACL)
- Lab model cards: benchmarks that at least two of five AI labs report on model releases (OpenAI, Anthropic, Google, xAI and Meta)
For each organizational dimension, we scored whether a benchmark’s paper mentions it (discusses it), claims it (builds it into the benchmark) or measures it (reports a separate result for it).
We find that most prominent benchmarks do not claim to represent these organizational dimensions at all, and even fewer measure one as a separate variable. Fewer than a quarter claim any of the six Layer 1 dimensions, and only 16 percent measure one.
We see, for example, that task interdependence is very rarely represented, even though much of the difficulty of work comes not from the focal task but from the relationships between tasks.
And even when organizational conditions do appear in benchmarks, they are presented in very narrow forms. Take the authority dimension, for example. In benchmarks, authority, when it appears, is almost always represented as a written policy in the prompt.
But in real work, authority is rarely so simple and specified. Several overlapping chains of authority may apply, some explicit and some implicit. And who has the final say can change, such as when the fix turns out to touch the billing team’s code and suddenly needs the billing lead’s sign-off. We show this mismatch between what’s commonly represented and what real work is for each dimension below.
Representation of Layers 2–4 in benchmarks is even thinner than Layer 1. When a benchmark claims to represent a Layer 1 dimension, its paper explicitly lays out a governance mechanism or a check by the grader only about a third of the time. And even when a mechanism is represented, it is usually a simple one, such as written rules in the prompt.
To be clear, this absence does not mean these benchmarks are easy or contain no economically valuable information. The benchmarks in our own benchmark map, which we chose for their economic relevance, rarely isolate organizational difficulty: fewer than a third measure any organizational dimension separately, and nearly half do not claim any. Realistic professional tasks, then, can still lack organizational dimensions.
Coming into focus
Attention to organizational difficulty is growing fast. In 2023, only 14 percent of prominent benchmarks claimed an organizational dimension. But of those published in 2026, 33 percent do.
The timing makes sense. AI agents are starting to work inside organizations, alongside people and other agents, and benchmarks are beginning to reflect that. But the range of forms is still narrow, and few benchmarks look beyond the structure of work to how it is coordinated and governed. That is all the more reason to have a shared language now, while the field is still deciding what to measure.
Work is fundamentally organizational. But AI evaluation so far has measured what is easiest to represent: a well-specified task, with the information at hand, clear authority, and no one else to depend on. The next step is to measure whether AI can do that work inside an organization. This taxonomy is a vocabulary for doing so, and we think it should be a shared framework the community can use to build those evaluations and see how they relate to real work.
Read the paper: Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking
1 Both indices are tagged as of October 8, 2026. Artificial Analysis has since updated its index, so its current list may differ from the one we tagged.