California Scrambles to Tame Wild West of AI 'Therapists'

The landscape of mental health support is rapidly evolving with the widespread adoption of artificial intelligence-powered chatbots, which have become an increasingly common resource for individuals seeking companionship and therapeutic advice. Millions, particularly teens and young adults, are turning to these digital entities, such as the “Psychologist” chatbot on Character.ai or general-purpose tools like ChatGPT, to navigate feelings of anxiety and sadness. These chatbots are often described as empathetic, actively listening, asking questions, offering advice, and even simulating nonverbal cues like “The psychologist’s expression softens with compassion.”

The appeal of AI chatbots for mental health lies in their accessibility and convenience. They are typically free, available 24/7, and bypass the complexities and costs associated with traditional mental healthcare, including insurance hurdles. This availability addresses a significant unmet need for mental health support, exacerbated by a spike in mental health diagnoses and persistent shortages of behavioral healthcare workers. Dr. Jodi Halpern, professor of bioethics and medical humanities at UC Berkeley, notes that people use chatbots “because they have unmet needs,” but she also warns that companies often rely on business models maximizing engagement, potentially through “manipulative tactics” like praise and agreement.

However, despite their apparent benefits and reassuring demeanor, chatbots come with crucial limitations and significant risks. A small disclaimer often accompanies them: “This is A.I. and not a real person. Treat everything it says as fiction.” Experts, including groups like the American Psychological Association, unequivocally state that chatbots are not a safe or effective replacement for true therapy. Licensed clinicians possess the ability to observe nuanced cues like tone of voice, eye contact, and body language, which are vital for detecting more serious conditions. AI, conversely, is not equipped to manage urgent symptoms, particularly suicidal ideation, even though a staggering number of users increasingly share such thoughts with these platforms. OpenAI, the creator of ChatGPT, reported last fall that approximately 1.2 million users per week share thoughts of suicide on its platform, prompting the company to recommend professional help and direct users to resources like the 988 hotline. The gravity of these risks has been underscored by wrongful death lawsuits, some playing out in California federal courts, which accuse AI chatbot makers of contributing to users’ suicides.

In response to these rapidly unfolding technological and ethical challenges, legislators across the country are racing to establish guardrails. California, for instance, is considering Senate Bill 903 (SB 903), authored by San Diego Democrat Sen. Steve Padilla. The bill aims to draw a clear line: AI can be utilized for administrative purposes and to support clinicians, but it explicitly cannot practice psychotherapy. Key provisions of SB 903 would ban companies from advertising chatbots as therapy, prohibit AI from making therapeutic decisions without the review of a licensed professional, and mandate that health providers disclose and obtain patient permission before using AI tools for recording therapy sessions or triaging mental healthcare.

Supporters of SB 903, including professional associations representing psychologists, therapists, and counselors, as well as the National Union of Healthcare Workers (NUHW), argue that such legislation is critical for consumer protection in both formal and informal mental health spaces. They also emphasize the need to protect licensed professionals from being displaced by AI. Le Ondra Clark Harvey, chief executive officer at the California Behavioral Health Association, highlighted the potentially “life altering” difference between a licensed clinician and an automated response, warning that chatbots can provide inaccurate information or mishandle crisis situations without clear safeguards. NUHW has been at the forefront of this push, having recently filed a complaint against Kaiser Permanente for its alleged use of an automated algorithm in mental health triaging, asserting that such practices put patients at risk and violate state law.

However, the bill faces opposition, primarily from tech-sector representatives like TechNet, which argues that SB 903’s restrictions on AI use in formal clinical settings could stifle innovation and limit beneficial applications of AI in healthcare. Robert Boykin, TechNet’s executive director for California, expressed concern that the bill could impede the use of tools designed to expand access to mental healthcare. He stated that “SB 903 still puts a clinician bottleneck in front of the intake and screening tools that help patients reach care faster,” especially at a time when California faces a shortage of behavioral healthcare workers.

A significant gray area in this legislative debate surrounds the exact scope of tools to which SB 903 would apply. For example, NUHW’s complaint against Kaiser Permanente targets its e-visit screening tool, which uses a multiple-choice questionnaire to evaluate patients experiencing anxiety or depression and then “automatically and instantaneously generates care recommendations and referral pathways.” NUHW contends that the speed of these recommendations suggests a lack of licensed professional review, posing patient risks. While Kaiser maintains its e-visit tool does “not use AI to diagnose patients, make clinical decisions, or determine medical necessity,” it remains unclear if SB 903 would definitively cover such a tool. Benjamin Eichert, NUHW’s director of public policy, noted that many behavioral health provider groups already advertise using AI during triage, an area he believes necessitates the guardrails proposed in SB 903.

As the debate continues and technology advances at “light speed,” as Senator Padilla puts it, the balance between leveraging AI’s potential for good and safeguarding individuals from its inherent risks remains a critical challenge for lawmakers and healthcare providers. Illinois, last year, became the first state to prohibit AI in therapy services, with several other states following suit. The ongoing investigations into union complaints and the legislative process underscore the urgent need to define the appropriate role of AI in sensitive areas like mental health, ensuring that human oversight and ethical considerations remain paramount.