California Governor Gavin Newsom has signed a groundbreaking law aimed at establishing stringent safeguards against the potentially catastrophic misuse of powerful artificial intelligence models. This legislative action positions California as a leader in AI regulation, with Newsom highlighting the state's proactive approach while simultaneously critiquing the lack of federal action in this critical area. The new law is designed to implement some of the nation's first comprehensive regulations on large-scale AI models, carefully crafted to protect communities without hindering the state's burgeoning homegrown AI industry, which houses many of the world’s top AI companies.

The core of the legislation mandates that AI companies operating with "frontier" models—those running on significant computing power—must establish and publicly disclose safety protocols. These protocols are specifically intended to prevent their most advanced systems from being exploited to cause widespread harm, such as constructing a bioweapon, orchestrating a bank system shutdown, or hacking into vital infrastructure like a power grid. While the numerical thresholds for computing power are acknowledged as an imperfect initial metric, they serve as a starting point to differentiate today's high-performing generative AI systems from the next generation of even more potent technologies. Companies like Anthropic, Google, Meta Platforms, and OpenAI, many of which are California-based, will be directly impacted by these requirements.

Under the new law, a "catastrophic risk" is precisely defined as an event causing at least $1 billion in damage or resulting in more than 50 injuries or deaths. To ensure accountability, companies are required to report any critical safety incidents to the state within 15 days of their occurrence. Furthermore, the legislation introduces crucial whistleblower protections for AI workers and facilitates research by establishing a public cloud for researchers. Non-compliance carries a significant penalty, with fines set at $1 million per violation.

The path to this legislation was not without its challenges and debates. While some tech companies expressed opposition, arguing that AI regulation should ideally be handled at the federal level, others, like Anthropic, offered support. Jack Clark, co-founder and head of policy at Anthropic, described the regulations as "practical safeguards" that formalize many safety practices already voluntarily adopted by companies. He emphasized that California has created a robust framework that successfully balances public safety with continued innovation, even as federal standards remain essential to prevent a fragmented regulatory landscape.

This recently signed bill follows Newsom's veto of a broader version of AI legislation last year, where he sided with tech companies concerned that the earlier requirements were too rigid and could impede innovation. Subsequently, Newsom convened a group of industry experts, including renowned AI pioneer Fei-Fei Li, to develop recommendations for AI model guardrails. Supporters confirm that the new law effectively integrates the feedback and recommendations from this expert group and the industry, particularly by avoiding the imposition of the same level of reporting requirements on startups to safeguard nascent innovation, as highlighted by state Sen. Scott Wiener, the bill’s author. Senator Wiener lauded the legislation, stating, "With this law, California is stepping up, once again, as a global leader on both technology innovation and safety."

California's move stands in contrast to approaches proposed at the federal level, such as former President Donald Trump's plan to eliminate "onerous" regulations to accelerate AI innovation, or unsuccessful attempts by Republicans in Congress to ban states from regulating AI for a decade. In the absence of stronger federal oversight, states nationwide have been independently addressing AI concerns, from deepfakes in elections to AI "therapy." California itself has passed several additional bills this year concerning AI chatbots for children and AI's role in the workplace. Beyond regulation, California has also been an early adopter of AI technologies, deploying generative AI tools for practical applications like detecting wildfires and managing highway congestion and road safety.