The AI Consumer Index

The ACE leaderboard

We created ACE because consumer applications are among the most widespread and fastest-growing uses of AI—yet existing benchmarks have paid too little attention. ACE evaluates whether AI models can help people with everyday activities such as finding products, enjoying their hobbies, fixing items at home, and cooking. It comprises a hidden held-out set of 400 prompts.

ACE tests frontier models with web search enabled, mimicking real-world consumer behavior. It uses a novel evaluation methodology that combines expert-crafted rubrics with hurdle criteria and grounding checks. Read the paper to learn more about our approach.

To support open research, we have open-sourced 80 in-distribution cases on Hugging Face, including all metadata labels. We have also released our evaluation harness for full reproducibility.