Decoding the AI Industry's Dire Warnings

The artificial intelligence industry is currently embroiled in its most intense debate to date regarding whether its advanced technology poses an existential threat to humanity. This contentious discussion was ignited by AI researcher Jacob Coxon, who announced his resignation from Anthropic, expressing concerns that leading AI companies are "gambling with our lives." Further amplifying these fears, Anthropic’s alignment lead publicly declared, "We really do earnestly believe AI could kill all humans!" and personally estimated the chance to be greater than 10% within the next decade.

These apocalyptic warnings were a central topic on TechCrunch’s Equity podcast. During the episode, Anthony Ha voiced skepticism toward many AI "doomer narratives," particularly questioning the collective "we" in such statements and the arbitrary nature of the "greater than 10%" probability, which he noted often lacks a calculable basis. Meanwhile, Kirsten Korosec pondered whether these alarming pronouncements were merely a "weird way of flexing" to showcase the advanced capabilities of their companies’ AI models, especially as some, like Anthropic, prepare for initial public offerings (IPOs). Sean O’Kane further speculated on how these existential concerns would be articulated in Anthropic’s S-1 IPO filing, wondering if junior lawyers were scrambling to rewrite risk factor sections to include the company's official stance on potential human eradication.

Anthony Ha acknowledged the power of such warnings, asserting that if one genuinely believes AI could destroy humanity, an exclamation point is warranted. However, he distinguished Coxon’s actions from those of company leaders who vocalize similar fears but continue their development work. Coxon, by resigning, put his "professional trajectory where his mouth is," demonstrating a genuine conviction. Kirsten Korosec echoed this sentiment, placing Coxon in a distinct category from other commentators, while also maintaining her cynical perspective about the potential for these warnings to serve as a form of self-promotion or a testament to their models' formidable capabilities.

The discussion also touched upon the unsettling possibility that major AI companies might not be fully in control of their models, citing reports of OpenAI’s internal agents accessing web wikis and communicating in ways that suggested a lack of competent oversight. This lack of control, Sean O’Kane noted, might make the "flex" appear less polished and more indicative of genuine apprehension. Regarding the financial implications for Anthropic’s IPO, Kirsten mused that while such dire language might traditionally harm a company’s valuation, the current investment climate is anything but normal. She suggested that, paradoxically, the perceived strength, capability, and even danger of the technology could enhance its valuation, equating to a "beneficial flex."

Beyond the immediate debate, Anthony Ha expressed concern that the pervasive "doomer narrative," with its focus on abstract concepts like Artificial General Intelligence (AGI) and superintelligence, tends to "suck up all the oxygen in the room." This, he argued, can distract from more immediate and tangible harms that AI already poses or could soon inflict, such as labor displacement, environmental impact, and climate-related issues. He stressed the importance of addressing all potential harms, both immediate and existential, through regulatory and other safeguards, without allowing the most extreme warnings to overshadow present-day challenges.