AI Evaluation Becomes New Product Blueprint, Says Expedia Executive

Defining Products Through Testing

A top Expedia Group official recently stated a shift in product development. Xavi Amatriain, the company's chief AI and data officer, spoke at VB Transform 2026. He explained that evaluations are now replacing traditional product requirement documents. This change reflects a new approach to building AI-driven products.

This means that the desired product behavior is defined through these evaluations. These evalscan include various testing methods. They might incorporate red teamingexercises, for instance.

Why Are Evaluations Replacing Traditional Documents?

Amatriain highlighted that these evaluations effectively encode product specifications. Instead of lengthy written documents, the testing process itself outlines what the product should achieve. This method inherently includes security considerations from the start. Security requirements are built into the evaluation framework. This ensures that safety and performance are assessed together.

The focus moves from theoretical descriptions to practical testing. This ensures that the AI product meets specific criteria. It also helps identify potential issues early in the development cycle.

# What is a PRD?

The move to evaluation-driven development is significant. It suggests a more agile and responsive way to build AI products. Traditional product requirement documents can be rigid. They may not fully capture the complexities of AI systems. Evaluations, however, allow for continuous refinement. They provide direct feedback on how the AI performs. This approach can lead to more robust and secure AI applications. It integrates testing as a core part of the design process.

# What does red teamingmean in this context?

This new methodology could streamline development. It also fosters a deeper understanding of the AI's capabilities and limitations. The industry may see more companies adopt this evaluation-first strategy.

A PRD, or Product Requirement Document, is a traditional document. It outlines the features, functions, and purpose of a product. It serves as a guide for development teams.

# How does this change benefit product development?

Red teaming involves simulating adversarial attacks or challenging scenarios. It helps identify vulnerabilities and potential biases in AI systems. This strengthens security and robustness.

This shift integrates security and performance testing earlier. It defines product behavior through practical evaluation. This can lead to more secure and effective AI products.