Rogue AI Alert: Ex-Anthropic & METR Experts Uncover Groundbreaking Control Method

The increasing intelligence of artificial intelligence (AI) systems has sparked growing concerns about their control and safety, leading to significant developments in the field of AI safety. These concerns were recently highlighted when Anthropic researcher Jacob Coxon resigned over fears that AI could pose existential risks by the decade's end. In response to such critical challenges, two founders, Rune Kvist and Rajiv Dattani, have launched a new venture aimed at mitigating these risks within corporate environments.

Kvist, an early employee at Anthropic, and Dattani, formerly the COO of the AI safety research organization METR, co-founded Artificial Intelligence Underwriting Company (AIUC). They contend that as AI becomes smarter, its adoption and control paradoxically become more difficult. AIUC's mission is to introduce robust AI safety protocols for enterprises and companies engaged in building AI models and agents. The startup has already attracted several prominent customers, including Cursor, Lovable, Harvey, and ElevenLabs, indicating a strong market need for its services.

AIUC recently secured substantial funding, announcing a $40 million Series A round led by Ribbit Capital with participation from First Harmonic. This follows an earlier $15 million seed round from investors such as Nat Friedman’s NFDG, Emergence, Terrain, and Anthropic co-founder Ben Mann, bringing their total funding to an impressive $55 million. This considerable investment underscores investor confidence in AIUC’s innovative approach to addressing AI risks.

At the core of AIUC’s strategy is the application of a familiar cybersecurity paradigm—specifically, a third-party audit and certification layer—to the emerging landscape of AI agent risks. Kvist explained that institutions like banks, hospitals, governments, and militaries are hesitant to deploy AI not because of a lack of intelligence, but because they cannot guarantee that AI systems will adhere to predefined commitments made to their own customers. To address this, AIUC has developed a new standard, termed AIUC-1, inspired by the widely adopted cybersecurity standard SOC 2, along with a comprehensive testing service to validate AI agents against this benchmark.

The AIUC-1 standard was meticulously crafted in collaboration with a consortium of approximately 250 security and risk leaders—the very individuals responsible for procuring AI agents within their organizations. Dattani elaborated that ongoing monthly consultations with these leaders help shape the standard, focusing on the critical questions and assurances they seek when acquiring AI agents. This collaborative development ensures the standard is directly responsive to enterprise needs.

Following the standard's development, AIUC puts AI agents through an extensive suite of around 5,000 tests. These rigorous evaluations assess agent behavior across various critical scenarios, including potential jailbreaks, instances of hallucination, and risks of data leaks. The outcome of this testing process is a detailed, approximately 100-page report, which meticulously outlines an agent’s performance in terms of safety and reliability, clearly identifying both strengths and areas of concern. Intriguingly, AIUC leverages AI agents themselves to conduct these tests and further employs AI to analyze the vast datasets generated, though human experts ultimately verify the final audit reports, ensuring a crucial layer of human oversight.

The concept of independent AI evaluation is gaining traction within the broader AI community. Dattani's previous employer, METR, where he served as COO and remains a board member, conducts similar testing for frontier AI labs, though their historical focus has been more on performance than pure safety. METR notably played a role in investigating OpenAI's Hugging Face incident. Furthermore, Anthropic CEO Dario Amodei has publicly advocated for a more measured pace in frontier AI development, citing a rise in problematic AI behaviors. Amodei has even suggested the necessity of embedding third-party evaluators, such as METR, within frontier labs to observe and verify safety protocols. While AIUC does not propose embedded evaluators, its model of providing an independent assessment of AI agent safety directly to enterprises aligns with this broader industry call for increased scrutiny and accountability. Dattani summarized AIUC's value proposition by stating, "Here’s where it passes and where you can trust it. And here’s where there’s concerns. You should be aware of those references before you make the decision to buy," emphasizing informed decision-making for businesses adopting AI.