Google Tests AI That Can Conduct Real-Time Video Medical Consultations

Google researchers have taken their medical AI system a step closer to the experience of an actual doctor's appointment, testing a version that can speak with patients, observe them through video and reason about their symptoms in real time.

The system, called AMIE — Articulate Medical Intelligence Explorer — remains experimental and is not available as a replacement for a doctor.

But new research suggests it can perform surprisingly well in controlled video consultations.

In a study involving patient actors and primary care physicians, researchers tested a video-enabled version of AMIE across medical scenarios involving cardiopulmonary, abdominal, neurological, psychiatric, musculoskeletal and other conditions.

Clinical evaluators found that AMIE's video system performed on par with or better than participating doctors across measures including history-taking, diagnosis, management and physical observation.

The results represent a significant change from earlier versions of AMIE.

Google originally developed the system around text conversations. Its researchers have progressively expanded those capabilities to include medical images, clinical documents and longer-term disease management.

The new approach attempts something more difficult: holding a natural video conversation while simultaneously watching, listening and reasoning about what the patient is presenting.

To accomplish that, AMIE divides the work between several AI agents.

One handles the conversation with the patient, while another works in the background on possible diagnoses and management decisions. A separate perception component processes visual and audio information from the consultation.

That separation matters because complex medical reasoning takes time. Making one AI system perform every task sequentially could result in awkward delays while a patient waits for it to respond.

Video also gives AMIE access to information that text cannot easily capture.

The system can observe certain physical signs and instruct a patient to perform simple examination manoeuvres during the call. Researchers found this to be one area where the video version performed particularly strongly.

The study nevertheless produced an important reminder about the difference between clinical performance and human interaction.

Patient actors rated AMIE highly in several areas, but participating physicians were preferred when it came to rapport and partnership-building — an important qualification when considering how AI might eventually fit into healthcare.

And Google isn't claiming that AMIE is ready to replace doctors.

Earlier research published in Nature has shown promising results for AMIE in diagnostic conversations, but researchers have repeatedly stressed that controlled studies cannot establish whether such a system can safely operate independently with real patients. The published research on conversational diagnostic AI explicitly identified real-world clinical translation as an area requiring further study.

Some real-patient testing has begun.

A separate prospective study placed AMIE into an actual primary-care setting, where 100 adult patients interacted with the text-based system before attending urgent-care appointments. Human safety supervisors monitored the conversations, while doctors remained responsible for patient care.

That study found AMIE's diagnostic and management-plan quality broadly comparable with clinicians on several measures, although doctors performed better when researchers assessed the practicality and cost-effectiveness of their management plans.

Google has also said it is conducting a nationwide study examining AI in real-world virtual care.

The potential implications extend well beyond Silicon Valley.

Countries struggling with shortages of doctors, overloaded hospitals and limited access to specialists could eventually benefit from AI capable of assisting clinicians remotely. For parts of Africa, where access to healthcare can vary dramatically between major cities and underserved communities, that possibility is particularly significant.

But AMIE has not reached that point yet.

The latest research demonstrates that an AI system can increasingly see, hear, converse and reason during a simulated medical consultation. Whether it can do all of that safely with real patients at scale is the much bigger test still ahead.