← Back to blog

· Lex Hamilton

Automation Bias in Clinical AI: What Happens When Humans Stop Checking

What automation bias is, the evidence that human review atrophies when AI checks AI, and how hospitals can keep verification real. Sourced for CMIOs.

Last reviewed: August 2026

Automation bias is the tendency of human users to accept a computer system's output in place of their own vigilant checking. It shows up as two kinds of error: commission, where a person follows incorrect machine advice, and omission, where a person misses a problem because the system never flagged it. It is the main reason "a clinician reviews everything the AI writes" is a weaker safeguard than it sounds. In 2025 and 2026, software engineering — a field further along the same adoption curve — produced measurable evidence of what happens to human review once an AI tool takes over the checking. This article lays out that evidence and what it means for hospitals deploying ambient documentation and clinical AI.

Table of contents

What is automation bias, and how often does it change clinical decisions?

Automation bias was documented in clinical systems long before large language models. A 2012 systematic review in JAMIA by Goddard, Roudsari, and Wyatt defined it as users over-accepting computer output "as a heuristic replacement of vigilant information seeking and processing." Their meta-analysis of clinical decision support studies found that erroneous advice was more likely to be followed when the system was present, with a risk ratio of 1.26 (95% CI 1.11 to 1.44). In four prospective studies, between 6% and 11% of cases saw a clinician change a correct decision to an incorrect one after seeing the computer's advice.

Two things about that finding deserve attention. First, it predates generative AI: these were rule-based decision support tools, far less fluent and far less persuasive than a model that writes complete clinical prose. Second, the bias operated on trained clinicians who believed they were exercising judgment. Diligence and intelligence do not immunize against it; automation bias is a predictable property of humans working alongside systems that are usually right.

The open question for clinical AI governance has been how strong this effect becomes at scale, when the AI produces most of the work product and the human role shifts to review. Healthcare does not yet have that number. Software engineering now does.

What happened when software engineering let AI review AI?

Software teams adopted AI on both sides of their quality process: models that write code, and AI review tools that check it. The measured result is that human review collapsed where the AI reviewer was present.

Qodo's 2025 State of AI Code Quality report states that when an AI code review tool is enabled, 80% of pull requests receive no human comment or review at all. That figure is Qodo's own product telemetry, not a result from its 609-developer survey, and Qodo sells AI code review — both caveats should travel with the number. But it is the most direct measurement available of the substitution effect: the AI reviewer was added as a supplement, and in four of five cases it became a replacement.

The output being waved through was not fine. CodeRabbit's December 2025 analysis of 470 open-source pull requests found AI-co-authored code carried about 1.7x more issues than human-only code (10.83 versus 6.45 per PR), with security issues up to 2.74x higher (CodeRabbit also sells AI review; the study's authorship labels were inferred, not confirmed). Cortex's 2026 engineering benchmark, drawn from a small sample of about 50 engineering leaders plus development metrics, found incidents per pull request rose 23.5% while output per author rose 20% — more work shipped, and proportionally more of it failed in production.

None of these vendors' numbers should be read as precise. Read together with the caveats attached, they describe one consistent pattern: when an AI system occupies the review seat, human verification does not stay constant. It atrophies.

Why does fluent output make oversight worse?

Because fluency and correctness improved at different rates. Veracode's 2025 GenAI Code Security Report, which tested over 100 large language models, found that 45% of AI-generated code samples failed security tests and introduced OWASP Top 10 vulnerabilities. Its central finding, stated verbatim: "While the models got better at writing functional or syntactically correct code, they were no better at writing secure code. Security performance remained flat, regardless of model size or training sophistication."

The clinical translation is direct. An ambient scribe's note can be complete, well-organized, and written in flawless clinical register while carrying a wrong laterality, an inverted negation, or a medication the patient never mentioned. The polish of the artifact is evidence of the model's language ability, not of the content's accuracy — and polish is precisely what a rushed reviewer's eye uses as a proxy for accuracy. We examined this failure mode in detail in why a transcript-faithful note can still be clinically wrong, and the broader point — that model accuracy improvements do not eliminate the need for verification — in why you can't make clinical AI accurate enough to skip governance.

Why is verifying AI output harder than producing it?

Automation bias gets stronger as verification gets harder. Lyell and Coiera's 2017 JAMIA systematic review found automation bias concentrated in tasks "typically involving diagnosis rather than monitoring, and with high verification complexity," and associated with the cognitive load of the decision task. In plain terms: the harder it is to check the machine, the more likely the human is to stop checking.

Verifying an ambient note is a high-verification-complexity task. Confirming that a generated note faithfully represents a twenty-minute encounter requires holding the whole encounter in memory, or re-listening to it — work that can approach the effort of writing the note manually. The economics of the deployment assume the clinician will not do that work, because if they did, the time savings that justified the purchase would vanish. The verification step survives in policy and disappears in practice, one plausible-looking note at a time.

This is why counting signatures tells a hospital nothing. A signed note proves attestation occurred; it does not prove verification occurred. The code-review data above shows what the gap between those two can look like when someone finally measures it.

Can the AI check its own work?

The tempting fix is a second model that reviews the first. Self-review has a structural ceiling, which we cover fully in why an AI system can't audit its own output: a checker that shares the generator's architecture, training data, or vendor incentives tends to share its blind spots, and its approval is unfalsifiable from inside the system.

The 2025 DORA report — presented by Google Cloud, and the closest thing to a disinterested source in this evidence set — frames the underlying dynamic: "AI's primary role is as an amplifier, magnifying an organization's existing strengths and weaknesses." An organization with weak verification practices that adds AI acquires speed, and the weak practices travel along at the new pace. What gets amplified is whatever was already there.

Measurement of the checking layer itself has to be independent of the tool being checked — the reason benchmarking of agentic clinical AI withholds its case sets from vendors.

What can a hospital do about oversight atrophy?

Regardless of which vendors are involved, four practices address the failure mode described above:

  • Measure review rates, not attestation rates. Track edit distance between draft and signed note, time-in-note, and the fraction of notes signed without modification. A sustained rise in unedited sign-offs is the clinical analogue of the 80% figure — and it is measurable today from data most EHRs already hold.
  • Budget verification time explicitly. If the deployment's ROI case assumes clinicians spend near-zero time reviewing, the deployment's safety case cannot simultaneously assume they review carefully. Make the two cases use the same number.
  • Put deterministic checks where attention fails. Automation bias is a property of human attention; rule-based checks do not tire, trust, or satisfice. Structured verification of high-severity items — medications, allergies, laterality, negations — catches the specific errors a fluency-lulled reviewer misses.
  • Keep the checking layer independent of the generating layer. As ambient tools expand from documentation toward order entry, the stakes of this separation rise. As one health-system data architect put it this month: "Documentation is clerical work. Clinical decision support is medicine. The liability exposure changes, the regulatory requirements change."

Orinyx exists because of the structural version of this argument: verification that sits inside the system it verifies inherits that system's failure modes, so the safety layer has to be architecturally separate. But the four practices above stand on their own, whoever builds them.

FAQ

Is automation bias just complacency or poor training?

No. The Goddard review found it in trained clinicians using systems they understood, and Lyell and Coiera tie it to task structure — verification complexity and cognitive load — rather than user attitude. Training reduces it modestly; redesigning the verification task reduces it more.

Does the software engineering evidence really transfer to healthcare?

The domains differ, and the numbers should not be quoted as healthcare statistics. What transfers is the mechanism: a human reviewer, a plausible machine artifact, and an AI tool occupying the checking role. Healthcare already documented the same mechanism at smaller scale in the pre-LLM decision-support literature.

If review rates are falling, isn't the answer simply requiring clinicians to review more thoroughly?

Mandates that ignore verification complexity produce attestation, not verification. The reliable levers are making verification cheaper (structured, targeted checks) and making its absence visible (measuring unedited sign-off rates), not adding a policy line that assumes attention is free.

Further reading

Related in this series