July 6, 2026 · Lex Hamilton

Why you can't make clinical AI accurate enough to skip governance

A higher accuracy score doesn't close the clinical AI safety gap. Here's what governance does that model training cannot, and why the FDA's 2026 CDS deregulation makes this urgent.

The 2026 NEJM AI randomized trial on ambient scribes produced two sets of findings. The efficiency and burnout numbers got most of the attention. The hallucination rate is the one worth sitting with.

Across 238 outpatient physicians and 14 specialties, the trial found ambient scribes hallucinate in roughly 7% of clinical encounters. One large health system logged over 2.5 million AI-assisted encounters in 14 months. At 7%, that's 175,000 notes with fabricated clinical content in a single year. Train the model to 99% accurate and the number drops to about 25,000. The math doesn't stop, it scales.

Most governance conversations in health systems focus on accuracy, and that's a reasonable place to start. The question this article explores is what accuracy leaves unresolved, and what governance handles instead.

Why accuracy alone doesn't complete the safety picture

Accuracy measures how often a model is right under evaluation conditions. It's useful for vendor selection and useful for tracking model improvement over time. What it doesn't measure is how the system performs across the specific conditions of a given deployment, on a given patient population, in the hands of a given set of clinicians.

Deployment conditions always diverge from evaluation conditions. A simulation study comparing five ambient scribe platforms found a mean error rate of 26.3% in real clinical notes, against vendor-reported rates of 1–3% under structured testing. That gap reflects what happens when a model trained on one distribution of data encounters a different one, and it's a consistent pattern across clinical AI types and specialties.

The Epic Sepsis Model is one of the most studied examples. The model had strong internal validation numbers before widespread deployment across hundreds of U.S. hospitals. An independent external analysis found 33% sensitivity at the recommended threshold in real-world use, meaning the system missed two-thirds of actual sepsis cases while generating 109 alerts per true positive. The training data looked different from the deployment reality, and there was no systematic mechanism to detect that until researchers published the gap.

Accuracy improvement matters and it's worth asking vendors about. The part governance adds is the ability to know when performance is drifting in your specific context, not the aggregate.

What kinds of failure don't appear in accuracy scores

Clinical AI fails in ways that standard accuracy metrics don't surface cleanly, and understanding those failure modes changes what a governance program needs to catch.

Hallucination in ambient documentation is worth naming specifically. It isn't a note that looks obviously wrong. It's a note that looks right, generated from content that was never said. Topaz, Peltonen, and Zhang's analysis in npj Digital Medicine identified several concrete patterns including fabricated medication entries, physical examinations documented without an exam having occurred, and symptoms misattributed from a family member to the patient. The clinical concern with hallucinated content is that reviewer detection relies on the reviewer knowing what to look for, and fabricated content that's formatted correctly doesn't signal that it's wrong.

Critical omissions work differently but create a similar challenge. A note that accurately records what the AI captured can still be missing a medication disclosure, a symptom mentioned in passing, or a follow-up instruction. The note isn't incorrect by accuracy standards, and the missing content isn't visible on its face.

Drift is the failure mode that accuracy scores miss most systematically. A model deployed across thousands of clinicians and patient encounters will encounter conditions its benchmark didn't include, and performance degrades in ways that aren't always visible to the clinician using the tool. In cardiovascular AI, a 2026 lifecycle analysis found that over 40% of FDA-cleared devices were authorized without clinical validation data, and recall rates for AI/ML-enabled devices in radiology and cardiovascular categories exceed 15%. Clearance established regulatory compliance, but performance across the deployment's specific conditions was a separate question.

What changed with the FDA's January 2026 CDS guidance

In January 2026, the FDA issued revised guidance on clinical decision support software. Under the updated framework, AI tools that assist clinicians with recommendations the clinician can independently evaluate may not require premarket 510(k) clearance. Many ambient scribes and LLM-based clinical tools now operate outside mandatory regulatory review.

The practical effect is a shift in where governance responsibility sits. The regulatory layer that previously carried some of that responsibility has moved, and health systems now own more of the post-market surveillance function than they did before. That's manageable with the right infrastructure in place, and it's worth naming clearly so resourcing decisions reflect the actual accountability structure.

What governance adds to the picture

Governance provides infrastructure that accuracy improvement doesn't, and the distinction is worth being specific about.

The first thing it adds is detection that doesn't require an error to surface through incident reporting. Most performance issues in clinical AI aren't discovered through complaints or safety events. They're discovered when someone looks systematically at output across the full encounter volume and finds patterns that wouldn't appear in any single case review.

The second is an audit record at the point of care. Most health systems can report how many AI-assisted encounters were processed. Fewer can report which notes were reviewed before signing, which contained errors, what the clinical context was, and how errors were handled. That record matters for quality improvement, and it matters considerably in legal or regulatory settings.

The third is independence from the system being monitored. When the system generating the recommendation is also the system checking it, the check and the output share the same underlying model and the same potential failure modes. An independent verification layer that operates outside the ambient scribe or CDS tool is architecturally different from the vendor's own quality monitoring, even when both are working from the same data.

What this looks like in practice

A few governance decisions that don't require any specific product can shift the risk profile meaningfully.

Defining error thresholds by clinical use case rather than overall is one of them. A 7% hallucination rate carries different stakes in emergency medicine than in outpatient primary care, and governance policies that treat all contexts the same way leave the highest-risk settings underspecified.

Requiring deployment-condition validation data from vendors is another. Evaluation accuracy from a vendor's benchmark dataset is a starting point, but performance on the actual patient population, the actual specialty workflows, and the actual edge cases of a given deployment is a different and more relevant question.

Building post-market monitoring into contracts before deployment is a third. Drift develops over months, often long after the initial deployment review, and monitoring architecture put in place after the first incident is monitoring architecture that missed whatever led to the incident.

The deeper work of runtime verification, independent audit logs, and interception at the point of care is where most health systems currently have a gap, and it's the work that the accuracy conversation tends to skip past.