How evals drive the next chapter in AI for businesses

Over one million businesses⁠ around the world are leveraging AI to drive greater efficiency and value creation. But some organizations have struggled to get the results they are expecting. What is causing the gap?

At OpenAI we are leveraging AI internally to achieve our ambitious goals. One key set of tools we use are evals, methods to measure and improve the ability of an AI system to meet expectations. 

Similar to product requirement documents, evals make fuzzy goals and abstract ideas specific and explicit. Using evals strategically can make a customer-facing product or internal tool more reliable at scale, decrease high-severity errors, protect against downside risk, and give an organization a measurable path to higher ROI. 

At OpenAI, our models are our products, so our researchers use rigorous frontier evals⁠(opens in a new window) ¹ to measure how well the models perform in different domains. While frontier evals help us ship better models faster, they cannot reveal all the nuances required to ensure the model will perform on a specific workflow in a specific business setting. That is why internal teams have also created dozens of contextual evals designed to assess performance within a specific product or internal workflow. It is also why business leaders should learn how to create contextual evals specific to their organization’s needs and operating environment. 

How evals work: Specify → Measure → Improve

1. Specify: Define what “great” means

Start with a small, empowered team that can write down the purpose of your AI system in plain terms, for example: “Convert qualified inbound emails into scheduled demos while staying on brand.”

2. Measure: Test against real-world conditions

To actually test the system, use examples drawn from real-world situations whenever possible, and include or invent edge cases that are rare but costly if mishandled.