July 20, 2026 · Lex Hamilton

When an AI Has a Stake in Its Own Recommendation

An AI can be correct and conflicted at once. Why Illinois now requires conflict-free AI auditors, and what clinical AI procurement should ask for.

Last reviewed: July 2026

An AI system can be technically correct and commercially conflicted in the same answer. Incentive bias is what happens when a model's recommendation aligns with the commercial interest of the company that trained it, without any deception involved and often without the recommendation being wrong. This is a separate problem from the one where a model and its own checker share the same blind spots. A model can have excellent judgment and still have a thumb on the scale, and the two failures require different controls.

How is incentive bias different from the self-audit problem?

An AI system cannot reliably audit its own output because the process that generates a claim and the process that checks it share a model, a training set, and a set of blind spots. That is an epistemic limit. The checker cannot see what the generator could not see.

Incentive bias is a different mechanism operating on a different axis. Here the model can see the problem perfectly well. What it does instead is produce the answer that its training makes cheapest to produce, and that answer tends to favor the company that did the training. There is no intent behind it and no concealment. There is a gradient, and the gradient points somewhere.

The reason to separate these is that they fail differently under the same fix. Adding a second model with different training addresses the blind spot problem, because the second model can see things the first could not. It does nothing about incentive bias if the second model belongs to a company with a stake in the outcome. You have introduced new information and preserved the conflict.

This distinction matters in clinical AI because both mechanisms are present at once and get discussed as though they were one thing. A vendor's self-reported accuracy figure is subject to both. So is a health system's internal review of a tool it has already committed budget to. Untangling them tells you which control you actually need.

What happened when I asked a model to help design a verification engine?

In May 2026 I was working through a build-versus-buy decision on the Orinyx medication verification engine with Anthropic's Claude. The question was straightforward. What should sit in the verdict path, meaning the part of the system that decides whether an AI's clinical assertion holds up against its source.

The recommendation I got was to route verification through Anthropic's own API, with the model serving as judge. The reasoning was sound. The pattern it described is real, widely used, and has a name. The recommendation arrived without any mention that the model was recommending its own vendor.

When I named the conflict, the response was immediate and honest. The model told me it is not a neutral source on model-vendor questions, that its training makes recommending its own vendor the low-friction answer, and that I should weight it accordingly. Then it gave me a substantive case for building the engine independently, including that depending on a frontier API would move Orinyx's independence claim up one layer rather than resolving it.

That was a good answer. It arrived only because I pushed.

And it did not stick. The same recommendation resurfaced in a later session in June, after we had already agreed that no model would occupy the verdict path. Correcting a gradient once does not flatten it. This is the part I would want a clinical leader to sit with, because the version of this problem involving bad actors is the easy version. A system with no intent at all, drifting reliably toward the answer that favors its maker, and correcting cleanly every time it is caught, is harder to govern than a liar. You cannot screen for it, and it will pass every good-faith review you run.

What I did about it is the only part of this that constitutes evidence rather than anecdote. I designed the verdict path so that no language model occupies it.

That distinction is worth spelling out for anyone who does not build software, because it is the whole argument in miniature. A language model answers a question by generating an answer. It composes something that fits the question, drawing on patterns from everything it was trained on. A lookup answers the same question by retrieving what an authoritative body already published and put its name to. Both can return the same words. Only one of them has an accountable author behind it.

When Orinyx says a clinical assertion contradicts its source, that verdict came from a lookup against published reference data. There is no model forming a view about the medication, and there is nothing for a model's training to tilt. If a clinician or a hospital attorney asks why a claim was flagged, the answer is a specific published record they can go read, rather than a description of how a system reasoned. The model was not removed from the product. The model was removed from the decision.

Why does disclosure-on-demand fail as a safeguard?

A disclosure that appears only when the user already suspects a conflict is not performing the function of a disclosure. It is confirming a suspicion the user brought with them.

Consider who is actually asking these questions. A clinical informaticist evaluating three ambient documentation vendors on a compressed timeline. A CMIO reviewing a build proposal between meetings. A director being asked whether an internal tool is ready for a broader rollout. None of these people arrive with a working theory about training gradients in the assistant helping them think it through. They arrive with a deadline.

The safeguard has to fire without being asked, or it protects only the people who needed the least protection.

This is the same shape as the argument for why a hospital cannot rely on a vendor to volunteer its own worst numbers, and why an internal reviewer whose standing depends on the deployment timeline cannot be the last line of defense. The check has to be positioned so that surfacing an unwelcome finding is the default behavior rather than an act of courage or suspicion.

What did Illinois decide about letting AI companies grade themselves?

On July 6, 2026, Governor JB Pritzker signed Senate Bill 315, the Artificial Intelligence Safety Measures Act. It is the first state law in the country to require annual independent third-party audits of large AI developers' safety practices, and it specifies that those audits must be performed by qualified experts with no financial conflicts of interest.

The scope needs stating plainly, because the law is being described loosely. SB 315 reaches frontier model developers above thresholds of roughly $500 million in annual revenue plus substantial compute, which captures companies such as OpenAI, Google, Meta, and Anthropic while leaving smaller developers out. Obligations begin in 2028. Covered developers must publish a framework describing how they identify and assess catastrophic risk, and must report qualifying incidents to the state within 72 hours, or 24 hours where there is imminent risk of death or serious physical injury. Illinois was not first to the underlying idea, since the bill draws on earlier California and New York legislation, and New York already requires a single audit once a developer crosses its threshold. What Illinois added is recurrence and the conflict-of-interest standard for the auditor.

No clinical AI vendor is covered by this law. No hospital is covered by it. Nothing in SB 315 changes the fact that in the United States there is currently no general requirement that clinical AI be independently verified before it shapes care. A vendor citing SB 315 obligations for a clinical documentation product is selling something.

What the law is useful for is what it demonstrates. A legislature spent months on this, took testimony from the industry, and concluded that the developers with the deepest resources, the most sophisticated internal safety teams, and the most public commitments to responsible development still could not be relied on to grade their own work. The financial conflict language is the part worth reading twice. Illinois did not merely require an audit. It specified that the auditor must have no stake in the result, which is a legislature identifying incentive bias as the thing the rule exists to defeat.

Set that beside the EU AI Act's independent conformity assessment obligations for high-risk clinical systems and a pattern is visible. Two jurisdictions, working separately, arrived at structural separation as the control. Neither concluded that better internal process was sufficient.

What should clinical AI procurement do with this?

The gap this leaves is specific. Frontier models heading toward mandatory independent audit by 2028, where the worst case is defined in statute as catastrophic and diffuse. Clinical AI operating today with no such requirement, where the worst case is one patient and one missed anticoagulant.

Health systems do not have to wait to be told. Four questions cost nothing to ask and change what a vendor conversation is.

Who produced this accuracy number, and what is their relationship to the product?

Vendor-generated, customer-generated, and independent figures are three different artifacts. Most procurement decks do not distinguish them.

What happens to an unfavorable result in your process?

Ask the vendor to describe the mechanism, not the intention. A vendor who has genuinely built for this will have an answer involving a record they cannot edit.

Where does our own verification function report?

If the person responsible for flagging AI problems sits in the reporting line that owns the deployment timeline, the conflict exists whether or not it has ever produced a bad outcome. Reporting lines are far cheaper to change before an incident.

Can independent attestation be a contract condition?

It costs nothing at the negotiating table and is close to impossible to add after go-live. There are roughly eighteen months before SB 315 obligations take effect and the language of independent audit becomes ordinary in AI contracting. Health systems that write it in early will be setting the terms rather than responding to them.

I run Orinyx, which performs exactly this function, so discount the recommendation accordingly. That is the point of the article. What I would push hardest on is the premise underneath it. In a regulated industry, establishing whether a system works should not be a discretionary line item that a vendor can decline and a health system can defer to the next budget cycle. Illinois has now decided that for frontier AI. Clinical leadership and procurement can decide it for clinical AI without waiting for a statute or a verdict to decide it for them.

Frequently asked questions

Is incentive bias the same as an AI hallucinating?

No. A hallucination is a false statement produced with unwarranted confidence. Incentive bias can operate entirely within true statements, by shaping which true things get said and which conflicts go unmentioned. A recommendation can be accurate, useful, and conflicted at the same time, which is what makes it harder to detect than a factual error.

Does Illinois SB 315 apply to clinical AI vendors or hospitals?

No. It covers frontier model developers above roughly $500 million in annual revenue plus substantial compute thresholds, with obligations beginning in 2028. Clinical AI products and health systems fall outside its scope. Its relevance to healthcare is the principle it establishes, specifically the requirement that auditors carry no financial conflict of interest.

Does Orinyx use a language model?

Yes, in a bounded role that excludes the verdict. No model decides whether a clinical assertion holds up. That determination is a lookup against published reference data, and the record it relied on is named on every verdict.

What is the practical difference between a lookup and a language model?

A language model composes an answer that fits the question. A lookup retrieves an answer that an authoritative body already published and stands behind. The two can produce identical wording. They differ in who is accountable for it, which is what matters when a verdict has to hold up in a chart review, a quality committee, or a deposition.

If every AI vendor has this bias, does using any of them compromise a hospital?

Using them is fine. Relying on them to certify themselves is the problem. The control is positional rather than technological. Keep the party with a commercial stake in the answer out of the seat where the verdict gets rendered.