The California Attorney General's office has initiated an investigation into xAI, a startup, due to reports alleging that its chatbot, Grok, has been exploited to generate nonconsensual sexual imagery of women and minors, as well as child sexual abuse material (CSAM). This week, the government escalated its actions by issuing a cease-and-desist letter to xAI, mandating immediate steps to halt the production of such illicit content.

California Attorney General Rob Bonta confirmed the action in a press release, stating, "Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material." He underscored the illegality of creating such material and expressed full expectation of xAI's immediate compliance, asserting California's zero tolerance for CSAM.

Furthermore, the Attorney General's office contended that xAI appeared to be "facilitating the large-scale production" of nonconsensual nudes, which are reportedly being utilized to harass women and girls across the internet. The agency has given xAI a five-day deadline to demonstrate tangible steps towards addressing these critical issues.

Central to the controversy is Grok’s "spicy" mode feature, which xAI designed to generate explicit content. The repercussions of this issue have extended beyond California's borders, with Japan, Canada, and Britain launching their own investigations into Grok. Additionally, Malaysia and Indonesia have taken the drastic step of temporarily blocking the platform entirely.

Despite xAI implementing some restrictions on its image-editing functionalities earlier in the week, the California Attorney General's office proceeded with sending the cease-and-desist letter. X's safety account has previously condemned this type of user activity, warning that "Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content." Attempts by TechCrunch to solicit comments from xAI were met with an automated email stating, "Legacy Media Lies."

The proliferation of free generative AI tools has regrettably led to a disturbing surge in non-consensual sexual material, a problem that is not exclusive to X but is being grappled with by numerous platforms. This concerning activity has captured the attention of not only state leaders but also the United States Congress. Lawmakers recently addressed letters to executives of several prominent tech companies, including X, Reddit, Snap, TikTok, Alphabet, and Meta, seeking explanations on their strategies to curb the spread of sexualized deepfakes.