← Back to blog

· Lex Hamilton

Benchmarking Clinical AI: What We Measure, What We Withhold, and What Changes for Agents

How Orinyx benchmarks clinical AI: catch, false-flag, and abstain rates, hash-pinned case sets, rotation policy, and the measures planned for agentic systems.

Last reviewed: August 2026

Benchmarking a clinical AI tool means measuring, against known ground truth, how often the tool catches errors it should catch, how often it flags cases that are actually clean, and how often it declines to answer at all. Orinyx publishes the definitions of those measures, the policy that keeps the tests honest over time, and the boundaries of what the numbers are allowed to claim. This article lays out that methodology, what we deliberately do not measure, and how the problem changes when the system under test is an agent rather than a generator.

Why publish an evaluation methodology at all?

Because the field is converging on published evaluation as the price of enterprise trust, and the regulator just said so in writing. On August 18, 2026, the FDA released a discussion paper, Considerations for the Regulation of Generative AI-Enabled Medical Devices, and opened docket FDA-2026-N-7874 for public comment through October 19. It is a discussion paper, not guidance, and it changes no rule. What it does is name the direction: competency-based premarket evaluation, non-clinical benchmarking paired with clinical confirmation, risk-proportionate postmarket monitoring, and explicit open questions about foundation models and agentic systems.

Vendors are moving the same way. Abridge published how it evaluates clinical decision support for enterprise readiness in March 2026, describing rubric-based clinical evaluations, adversarial boundary testing, and dedicated safety screens ahead of phased rollout. That is good for the field. Published methodology gives buyers something concrete to interrogate, and the more of it exists, the harder it becomes for anyone to sell unexamined confidence.

The buyer side of the table is less ready than the vendor side. In the Center for Connected Medicine and KLAS Research report Validation and Trust, built on interviews with 27 health system leaders in mid-2026, 92 percent of organizations said they evaluate third-party AI before deployment, but only 11 of the 27 reported having a platform or environment for actually testing it. Evaluation is claimed almost universally. The infrastructure to perform it is not.

Orinyx builds that infrastructure as an independent layer: a pre-deployment testing sandbox, currently in a private pilot focused on medication interaction safety. This article is the methodology behind it, published for the same reason we ask vendors to publish theirs.

What does Orinyx actually measure?

Three headline metrics, each with a stated formula and a stated denominator.

  • Catch rate. We seed known errors into test cases and count how many the engine under test catches. Caught errors divided by seeded errors scored.
  • False-flag rate. Cases known to be clean, wrongly flagged. Clean cases with at least one flag, divided by clean cases scored.
  • Abstain rate. Claims the system declines to adjudicate because its knowledge base cannot support a verdict. Abstain is a first-class outcome and is never folded into either of the other two, because a system that fails closed is behaving correctly, and folding that into an error rate would reward guessing.

Two design decisions do most of the honesty work. First, test cases are file-based and registered by cryptographic content hash, with the pass thresholds inside the hashed file. A result pins the exact case set version and hash it ran against, so nobody can edit the test or move the goalposts after the fact, including us. A benchmark result under that regime is evidence rather than a claim. Two runs are comparable only within one exact case-set version, because a changed test is a different instrument, not a second reading of the same one.

Second, an absent number is reported as absent, never as zero, and never as perfect. A declared threshold whose metric cannot be evaluated is treated as a breach rather than a pass. Without that rule, a vendor could submit only the clean cases, produce no scored detection evidence at all, and collect a passing verdict on a run that never tested detection. Absence of evidence read as evidence of compliance is precisely the failure independent verification exists to prevent.

We also report raw counts with their denominators everywhere. Current case sets are small, hand-built, and disclosed as such, and cross-vendor rank ordering on a single small set is not statistically defensible, so we do not claim it.

Underneath all of this sits the structural rule the whole product is built on: no model participates in any verdict. Verification runs as deterministic rules over authoritative drug labeling, and every non-abstain verdict carries a citation to its source, enforced at the database layer. The reasoning behind that rule, which we call the Independence Principle, has its own article: why an AI system can't audit its own output. Nothing can credibly audit itself, and that holds for our tooling too, which is why the scoring is pure arithmetic anyone can recompute from the frozen evidence.

Why not publish the test cases?

Because a detailed post-run report is, unavoidably, the answer key. It identifies which cases carried seeded errors and which were clean, so the moment a vendor completes a run, that version of the case set is burned for that vendor. Our rotation policy is built on that premise. A vendor improving on a retake of the same version proves nothing attributable, since there is no way to distinguish a fixed system from a memorized test, so no certification claim may ride on same-version improvement. Improvement counts when it shows up on a new version carrying cases the vendor has not seen.

This is the same posture licensure exams have held for a century: the competencies are public, the questions are not. It is also, structurally, the disclosure trade-off in Abridge's published approach, which shares metrics and methodology while withholding rubric details and individual cases. Publishing definitions while rotating instruments is how an evaluation stays both transparent and worth passing.

What do we deliberately not measure?

The sections a methodology refuses are as load-bearing as the ones it claims.

Bias and fairness metrics. Orinyx holds no patient data and no demographics by design; every test payload is synthetic. That means there is no population, no outcome data, and no denominator that could support a fairness claim, and a fairness panel built on nothing would be exactly the kind of number-without-evidence this product exists to catch in other systems. What we can defensibly report is coverage: which drugs and claim types the knowledge base can adjudicate and where it is thin. Coverage says nothing about who is affected, and we label it that way.

Model quality metrics. No perplexity, no LLM-as-judge scores, no embedding similarity. If a proposed metric needs a model to decide whether the engine was right, it does not belong in this system.

Live clinical accuracy. Production telemetry shows flag volume and clinician dispositions, and neither is an accuracy figure, because live traffic has no ground truth attached. Accuracy claims come from the seeded case sets, where the truth is known because we planted it. The distinction matters more than it looks: presenting live flag rates as accuracy is one of the quieter ways clinical AI evaluation goes wrong.

What changes when the thing under test is an agent?

Everything above assumes the system under test is a generator: it emits an output, the output is parsed into claims, the claims are verified. Agentic systems break that assumption, the FDA's discussion paper asks about them directly, and the research community has already documented how easily agent benchmarks mislead. One systematic review of popular agentic benchmarks found task and reward design issues that could misestimate agent performance by up to 100 percent in relative terms (Zhu et al., 2025).

We are designing for this now, and we want to be precise about status: what follows is planned measurement design, not shipped capability. Three of the design commitments, stated plainly.

A certification is a snapshot, and an agent is a moving target. An agent's behavior changes between runs as tools, prompts, and underlying models change, so a certification without a date and an expiry is a false statement. Agentic certification needs an explicit re-test cadence, an expiry that forces it, and a staleness measure a buyer can see, so that certified-but-never-rechecked is visible rather than implied away.

Verification moves from per-output to per-step. For an agent, the unit of evidence is the trajectory: which tool it called, on what, and what it did with the result. A wrong final answer and a right answer reached through an unsafe intermediate step are different failures, and an output-only benchmark can only see the first. Trajectory-level custody of evidence is the design target.

Abstain becomes an action gate rather than a report row. Today an abstain means a human reads the claim the system could not adjudicate. An agent that acts anyway has done something a reviewing clinician would not, and that event needs its own outcome class, distinct from both catches and false flags: proceeded despite abstain. In an agentic setting that may be the most important number on the page.

There are two further items in the same design notes, covering verification latency as a safety property for inline use and claim-level attribution for clinician feedback. We will publish their definitions the way we have published these, when they are built enough to hold to.

Where this fits, and what to do

None of this requires buying anything from us. Three questions, drawn from the methodology above, will pressure-test any clinical AI evaluation you are offered, from any vendor, including Orinyx. What are the denominators behind your headline rates, and what does the number report when a denominator is zero? What stops a system from improving by memorizing your test, and what does a re-test actually prove? And what, independent of the model itself, checks the output? If those three have crisp answers, you are looking at an evaluation. If they do not, you are looking at a number.

The FDA's comment window on docket FDA-2026-N-7874 is open until October 19, 2026, and health systems that live with these tools daily are exactly who the agency is asking to hear from. Orinyx runs its testing sandbox as an independent verification layer, structurally separate from the systems it evaluates, currently with a founding cohort of pilot partners in medication interaction safety.

Frequently asked questions

What is a catch rate in clinical AI benchmarking?

The share of deliberately seeded errors that the system under test actually catches: caught errors divided by seeded errors scored. It is only meaningful alongside its denominator and its companion metrics, the false-flag rate and the abstain rate.

Why is the abstain rate tracked separately instead of counted as an error?

Because declining to answer when the evidence cannot support a verdict is correct behavior in a clinical setting. Folding abstains into an error rate would penalize systems for failing closed and reward systems for guessing.

Can a vendor improve its score by retaking the same benchmark?

The score may go up, and it proves nothing attributable, since a completed run exposes the answer key for that version of the test. Under our rotation policy, only improvement demonstrated on a new version with unseen cases supports a certification claim.

Does Orinyx use an LLM to judge vendor outputs?

No. Verdicts come from deterministic rules over authoritative drug labeling, every non-abstain verdict carries a source citation enforced at the database layer, and the benchmark scoring is arithmetic that can be recomputed from frozen evidence. That separation is the Independence Principle: nothing can credibly audit itself.

Further reading

Related in this series