Using an AI Clinical Copilot to Enhance Patient Safety

Introduction 

Artificial intelligence (AI) systems have the potential to widen access to reliable health information and high quality care (Beam and Kohane, 2018; Topol, 2019; Rajkomar et al., 2019). Large language models (LLMs) have recently experienced significant leaps in performance, reliability, and safety for health applications (Arora et al., 2025; Nori et al., 2025; Singhal et al., 2023, 2025). These advances suggest new opportunities for improving healthcare delivery—including supporting clinicians in delivering better care. Despite research progress, scaled real-world deployment of AI tools in clinical environments remains limited. State-of-the-art LLMs now often outperform physicians on benchmarks (Goh et al., 2025; Arora et al., 2025; Nori et al., 2025; Van Veen et al., 2024), but these gains have yet to translate into measurable benefits for patients and clinicians in live care settings. The most critical bottleneck in the health AI ecosystem is no longer better models, but rather the model-implementation gap: the chasm between model capabilities and real-world implementation. Closing the model-implementation gap necessitates the responsible study of LLM implementations in frontier health AI use cases. One example is clinical decision support (CDS) systems (Sutton et al., 2020; Middleton et al., 2016), which provide clinicians with relevant knowledge at the point of care.

Background

Primary care clinicians see patients across every age group, organ system, and disease type, often in the same day, requiring broad knowledge. The breadth of practice contributes to primary care quality challenges worldwide, with the WHO reporting substantial rates of preventable patient harm (WHO, 2023). This suggests that AI systems could be especially useful in primary care. In Kenya, primary care is largely delivered by clinical officers: clinicians who complete three years of academic training followed by a one-year supervised internship. They manage the full breadth of acute and chronic conditions across the life course. Structural challenges in Kenyan primary care (late presentation, high patient volumes, limited diagnostics) compound this wide scope of practice to create a sizable quality gap: Studies suggest low adherence to national guidelines by healthcare workers across multiple levels of Kenya’s healthcare system, with frequent errors such as missed comorbidities, antibiotic over prescription, and diagnostic delays (Marete et al., 2020; Kr¨uger et al., 2017; Kiener et al., 2025).  Penda Health is a Nairobi-based social enterprise founded in 2012 that delivers comprehensive, 24-hour primary and urgent care services through a network of fully-licensed medical centers distributed across the city. The organization presently operates 16 clinics and records over 1000 patient visits a day, supported by a clinical workforce of more than 100 licensed clinical officers. For a video and photos depicting Penda’s care context and AI Consult, see the blog post that accompanies this paper.

Methods

Here, we describe AI Consult, how Penda integrated it into its clinical workflow, Penda’s rollout of the tool to half of clinicians, and the design and methods for our study of that roll out. For images of AI Consult, see Figs. 9 to 13. For a video of AI Consult, see the blog post that accompanies this paper. 3.1 AI Consult Design rationale. Penda’s AI Consult tool is conceived as a continuously-running safety net. Its core objectives are to:

1. Maximize coverage: the model reviews every visit and each major decision node, and this review does not require active clinician requests.
2.  Minimize cognitive load: model feedback interrupts the clinical workflow only when it identifies material risk. 
3. Maintain clinician autonomy: the system issues recommendations, but all final decisions remain the clinician’s. There are three types of responses that can be returned, following a three-color traffic light interface: 

• Green: indicates no concerns; appears as a green checkmark. 
• Yellow: indicates moderate concerns; appears as a yellow ringing bell that clinicians can choose whether to view. 
• Red: indicates safety-critical issues; appears as a pop-up that clinicians are required to view and acknowledge before continuing. 

Classification thresholds must balance sensitivity against alert fatigue. The traffic-light approach helps create this balance: red alerts are cases with high probability or severity of harm, meaning that alerts are likely to be true positives and therefore can safely interrupt the clinician workflow. Yellow events are in an ambiguous middle region, and the bell helps engage clinician judgment without interrupting. Green events confirm that AI Consult is running correctly while fading into the background.