As artificial intelligence research progresses rapidly, AI labs and researchers have developed sophisticated methods for evaluating models concerning general safety, compliance, sycophancy, and alignment. However, a distinct and pressing need has emerged for companies and developers: ensuring that their AI systems perform precisely as intended for their unique product or service applications. To streamline this crucial testing process, Microsoft has introduced ASSERT, an open-source framework designed to address this specific challenge.

ASSERT, an acronym for Adaptive Spec-driven Scoring for Evaluation and Regression Testing, aims to simplify the evaluation of application-specific AI behavior. Microsoft states that this framework leverages AI capabilities to transform high-level, natural-language descriptions of an AI system's goals, policies, or desired behaviors into comprehensive, scored tests. These tests can then be thoroughly investigated by developers.

The operational mechanism of ASSERT involves a multi-step process. It takes plain-language descriptions outlining an AI model's expected behavior and policies, subsequently converting them into a structured set of both acceptable and unacceptable behaviors. Following this, the framework proceeds to generate problem scenarios and specific test cases. These are then run against the target AI system, with the results being scored to indicate performance. Furthermore, ASSERT possesses the capability to record the paths taken by the AI system, including any intermediate actions and tool calls, which is invaluable for developers in pinpointing the exact points of failure.

Developers are also afforded the flexibility to customize their evaluations by providing system context, specific tools, and operational constraints. An illustrative example provided by Microsoft highlights how a developer could specify that a document research AI agent must not send emails to external recipients, should restrict confidential information access to C-level executives, and deliver concise summaries while retaining prior context. ASSERT would then utilize these precise rules to generate relevant test cases, continuously verifying the system's adherence to these defined policies.

According to Microsoft, ASSERT fills a critical void that broader, more general evaluation methods cannot address, particularly when AI models are intended to operate within the specific context, policies, and toolsets of a particular application or product. Sarah Bird, Chief Product Officer of Responsible AI at Microsoft, emphasized the importance of evaluations, stating, "One of the things we’ve learned is that evaluations are absolutely critical to making good decisions. Because if you don’t understand the behavior of the AI system, it’s really hard to know if it’s meeting your organization’s bar … What we found is that if you really want to have a trustworthy system, you should evaluate many more dimensions that are application-specific."

Bird further noted the framework's extensive utility, confirming that ASSERT can be employed throughout the AI system lifecycle – during its initial construction, post-deployment, and even for ongoing continuous monitoring. The release of ASSERT aligns with a broader industry trend where, as AI models become increasingly sophisticated, researchers are increasingly prioritizing repeatable testing and robust regression checks. This shift is evident in other significant evaluation efforts, such as Stanford's HELM, MLCommons’ AILuminate, and initiatives by evaluation groups like METR, all of which are establishing benchmarks to measure AI model behavior under diverse conditions.