New California Laws Regulate AI Use in Health Care

Three of the over twenty new AI-related laws that were signed into law in California last week will have significant impacts on the use of AI for health care involving California residents. The laws govern the development and deployment of AI-based clinical decision support systems, the confidentiality of medical information shared with health care chatbots, and the obligation of large businesses to provide access to human customer service agents, with exceptions for certain hospital communications.

All three of the new laws take effect on January 1, 2027.

Regulation of Clinical Decision Support Systems

Two of the new laws touch on clinical decision support (CDS) systems, which provide notifications or alerts to clinicians to assist them in making clinical decisions based on the contents of a patient’s electronic health record. 

Both laws define a CDS system as “an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decision-making related to timing of care, diagnosis, or treatment.” The definition specifically excludes systems that provide appointment management (such as booking, canceling, and rescheduling appointments and appointment reminders) and payment processing, to the extent that performance of those activities by the system does not require professional licensing.

The new laws tackle both the potential for bias in AI models and the need for physicians to maintain their professional judgment when treating patients.

Bias in CDS Systems

Senate Bill 503 focuses on eliminating any “biased impact” from the use of CDS systems, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.

The law requires developers and deployers (including health facilities, clinics, physicians’ offices or office of a group practice) to make reasonable efforts to identify CDS systems that are known to have or have a reasonably foreseeable risk of having biased impacts in health programs or activities. Both developers and deployers of these systems are also required to take reasonable steps to mitigate the bias impacts from the CDS systems. 

Developers must also provide deployers with a statement describing the system’s intended uses and known or reasonably foreseeable risks, along with documentation – available upon request or at the time of initial sale, whichever is earlier, and upon material updates – disclosing each of the following:

• High-level summaries of the type or types of data used to train the CDS system, including a description of its demographic representativeness when demographic data is available.

• How the CDS system was evaluated for performance, limitations, and mitigation of biased impacts.

• High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases.

• Intended benefits and expected outputs of the CDS system.

• Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks.

• Recommendations regarding how the CDS system should be used and monitored, and how risks should be mitigated.

• Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the CDS system for biased impacts.

We expect that the new developer obligation to provide this information and documentation will be welcome to deployers, because developers are generally not willing to provide it. However, both deployers and developers will need to potentially enhance their AI governance programs to incorporate this bias analysis and potentially ongoing monitoring of CDS systems.

Medical Judgment in CDS Systems

Assembly Bill 1979 attempts to address the perception or fear that health care providers could offload medical decision-making to CDS systems. It requires a health facility, clinic, physician’s office or office of a group practice to take reasonable steps to ensure that a licensed provider retains independent professional judgment in the scope of their practice when care is informed by the output from a CDS system.

Further, under the new law, any of the identified health care providers may not be allowed to use or deploy any tool or device that uses AI to either independently perform or to direct, guide, supervise, or instruct unlicensed personnel in performing clinical functions that are required by law to be performed by a licensed professional. The law includes exceptions for tools that are used as part of a supervised course of training while working toward licensure and for automated decision tools that do not require the application of professional judgment.

Violations of the section are within the jurisdiction of the appropriate licensing body and could result in determinations of the unlawful practice of medicine.

Privacy of Medical Information in Health Care chatbots

Assembly Bill 1979 also amends the state’s Confidentiality of Medical Information Act (CMIA) to automatically deem a business to be a health care provider for the purposes of CMIA applicability if that business provides a health care chatbot to consumers. According to the law, a health care chatbot is one that is marketed as and is used for facilitating or supporting health services. This includes either a bot allowing an individual to manage the individual’s health care information, or one that can be used by either the business or individual for the diagnosis, treatment, or management of a medical condition of the individual.

The CMIA confidentiality obligations will now apply to businesses that use these health care chatbots. The CMIA includes a prohibition against health care providers intentionally sharing, selling, using for marketing, or otherwise using any medical information, for any purpose not necessary to provide health care services to a patient. Violation of the law is a misdemeanor criminal offense.

Importantly, the new law only deems those businesses as health care providers with regard to the CMIA and does not automatically import those businesses into any other provision of state law that regulates health care providers. These businesses must maintain the same standards of confidentiality required of a provider of health care and are subject to the CMIA’s penalties for improper use and disclosure, and a violation that results in economic loss or personal injury is punishable as a misdemeanor.

Use of Customer Service AI Chatbots

The third of the three laws more broadly regulates the use of AI chatbots in a customer service environment, including in relation to the provision of health care. Assembly Bill 1609 restricts large private businesses, as defined in the statute, from representing that AI chatbots are human and requires the businesses to make a good-faith effort to connect customers to a human customer service agent upon the request of the customer during regular business hours within 15 minutes.

The term “large private business” is defined to include a business with more than $500 million in gross annual revenue nationwide and that provides goods and services to customers. Many health care companies fall well within this category. The law does, however, exempt hospital communications relating to the provision, coordination, management, administration, payment, or operation of health care services.

Violations of this law are subject to civil fines of up to $5,000 for an initial violation and $10,000 for each subsequent violation. However, the law does not create a private right of action for aggrieved consumers of a “large private business” and creates exceptions for unforeseen circumstances, emergencies and exclusive lines of business (meaning communications channels not intended for general customer use). The law also does not require a business to set up a telephone communications system for access to a live customer service representative if the business did not already have an existing system.

Failed Attempts to Regulate AI

The California legislature had passed two other bills relating to AI that would have impacted businesses involved in health care, but the governor vetoed both. The first, Assembly Bill 2575, would have prohibited retaliation or discrimination against a worker who chooses to either rely or not on a CDS system when using their professional judgment to make an assessment or decision.

Governor Newsom, in his veto message, indicated that he thought this bill actually tied the hands of the Labor Commissioner in establishing a violation of the state’s anti-retaliation provisions by linking the anti-retaliation provisions to the scope of practice requirements.

The second bill, Senate Bill 903, which was the subject of a Reed Smith Client Alert earlier this year, would have restricted the use of AI in providing mental health services. The bill would not only have limited the use of AI chatbots in the actual provision of mental health services, it would have also required mental health providers to get detailed consent from patients when using AI to transcribe or record mental health sessions.

In vetoing this bill, Governor Newsom called the prohibitions overly broad and expressed concerns that the provisions would improperly limit the tools that mental health professionals could use.

What Does This Mean?

Despite warnings from the White House about state-by-state regulation of AI, California is one of many states pressing forward in passing laws that regulate the use of AI in the health care context. Notably, in both of the veto notices for the failed bills, Governor Newsom commended the efforts to regulate AI in that space and urged the legislature to return to the subject at a later time.

As AI’s clinical decision-making role increases, and as the federal government’s express regulation and preemption on this topic continues to lag, health care providers, AI developers, and other businesses that use AI systems need to look to their state laws for regulation and confirm that they are in compliance.

Reed Smith will continue to follow developments in state law and in the regulation of AI. If you have any questions about these laws or about topics related to AI in health care, please reach out to the authors of this post or to your health care lawyers at Reed Smith.