← Back to blog

· Lex Hamilton

Who Verifies the AI That Verifies Healthcare AI?

Orinyx told FDA in docket FDA-2026-N-7874 that independent verification of healthcare AI must reach the AI judge, the agent's trajectory, and the supervisory agent.

Last reviewed: October 2026

An AI system should not be solely responsible for verifying its own reliability. Independent verification means the checker is separate from the thing being checked at two levels: the organization running the test, and the evaluator model doing the judging. In October 2026 I submitted this argument, with four related recommendations on postmarket monitoring and agentic systems, to the FDA's public docket on generative AI-enabled medical devices.

On this page

Background: what I commented on

In August 2026, FDA's Center for Devices and Radiological Health published a discussion paper, "Considerations for the Regulation of Generative AI-Enabled Medical Devices," and opened docket FDA-2026-N-7874 for feedback until October 19, 2026. It asks whether independent third parties should take part in benchmarking, how postmarket monitoring should work, and what changes when a device is agentic rather than a single model call. FDA states that the paper is for discussion only and is not draft or final guidance.

I answered five of its questions (16, 19, 20, 22, and 26). You can read my full public comment on regulations.gov. This is the plain-language version and the first of five Engineering Notes, one per question.

Orinyx is not currently a manufacturer of generative AI-enabled devices. I'm building independent verification infrastructure for clinical AI: the crash-test lab, not the car manufacturer. That is why I commented, and it is an interest you should weigh as you read.

What does independent verification mean for a clinical AI system?

Independent verification means the party evaluating a clinical AI system has no stake in the result and no shared dependence on the system's own model. In Question 16 I argued this matters most for safety-critical recognition, escalation, and calibration, because the entity best positioned to know where its device is weak also has the strongest incentive not to surface that weakness in a benchmark it wrote.

I proposed defining independence at two levels:

  • Organizational independence. The third party has no financial relationship with the sponsor tied to benchmarking outcomes, and no equity, licensing, or revenue-sharing arrangement with the foundation model developer behind the device.
  • Evaluator independence. If the adjudication method uses an AI judge, that judge should not be the same model family, or a fine-tuned variant of it, as the device under test. A GPT-based device benchmarked by a GPT-based judge is not independent in any way that matters, even if the organizations running the test are unrelated.

The second level is the one that gets skipped, so here is what the evidence does and does not show. Panickssery and colleagues found that an LLM judge scored its own generations higher than human annotators did, and that the bias grew with the model's ability to recognize its own text. Wataoka and colleagues reported significant self-preference in GPT-4 and tied it to familiarity: judges rated lower-perplexity text higher than humans did, whether or not they wrote it. That is consistent with my concern about same-family pairs. Neither paper tested a deployed clinical device, so the step from "judges favor familiar text" to "same-family judging is unsafe for medical devices" is my argument, not an established result.

Judges are still useful: the MT-Bench paper reports strong LLM judges agreeing with human preferences more than 80% of the time, about the rate at which humans agree with each other. The aim is not to ban judges but to keep them out of the one seat where their bias matters most.

On competition, I argued against a closed panel of pre-approved evaluators. Published qualification criteria any structurally independent entity can meet, like the accreditation model the paper references, keep entry open while enforcing the standard. The general version of this problem is in why an AI system cannot audit its own output.

Who verifies the supervisory agent?

A machine-based supervisory agent can help with postmarket monitoring, but it needs its own governance, or it only moves the trust problem up a level. That was my answer to Question 20. A supervisory agent's reliability cannot be established from its own logs. It needs the same two-level independence standard.

From my own build, I described three loops that may be a useful frame:

  • Point-of-action safeguards. Deterministic, rule-based checks before and after every output: malformed-input rejection before, grounding verification after (is every claim traceable to a source the agent actually had access to). I keep this layer deterministic because it has to be auditable and fail predictably.
  • Upstream governance. A deployment gate: a golden test set the workflow must pass before going live, with a defined hallucination-rate threshold, re-run on any prompt or model change.
  • Post-deployment monitoring. The supervisor's own outputs are sampled and scored by a separate evaluator, using external telemetry. That means logging tool-call results independently of the agent's self-reported status, because agents under-report their own failures.

I also suggested FDA name turn budgets and per-session action caps as a circuit breaker, since an unbounded supervisor is harder to audit and to contain mid-session.

The chain has to end somewhere. If a supervisory structure benchmarks a worker agent, something still has to benchmark the supervisory structure, and that chain has to terminate in a structurally independent, non-agentic check. Otherwise accountability regresses one level at a time and never resolves. That is the question in this article's title. I do not think it has a clever answer. It has a boundary: somewhere, a check that is not itself an agent.

Why is the final answer not enough to evaluate an agent?

A correct final answer does not show that the process behind it was sound. I'm developing and testing a version of this pattern that splits the supervisory job into narrower evaluators running against the same output, instead of one model judging another:

  • A deterministic evaluator checks what a machine can verify without any model: did the output cite a source that exists, does a calculation match, was a required field present.
  • A semantic evaluator assesses the quality and appropriateness of the reasoning.
  • A policy evaluator checks whether the agent stayed within its authorized scope and tools.
  • A trace evaluator scores the full execution path, not just the final output.

The trace evaluator matters more than it first appears. Evaluating only input against output misses mid-process failures, such as an agent that reaches a correct-looking conclusion through a reasoning chain that should have triggered escalation earlier, or through a tool it was not authorized to use. I suggested FDA consider requiring postmarket evaluation of agentic systems to assess the trajectory, not only the endpoint.

Agent-evaluation research points the same way. A 2025 paper on the reasoning trajectories of tool-augmented agents found that agents with similar answer accuracy differed significantly once their trajectories were scored for efficiency, hallucination, and adaptivity. Splitting the judge also narrows what has to be trusted: where a check can be deterministic, it is, and only the judgment-dependent part is left to a model, which must meet the same independence standard. It does not remove the question of who checks the checker.

Why is sampling not enough after launch?

Each of the three postmarket approaches in FDA's paper is necessary and none is sufficient, because they run at different latencies and catch different failures. That was my answer to Question 19.

  • Periodic re-benchmarking catches regressions against known cases but cannot, by construction, catch a failure mode the benchmark did not anticipate.
  • Sample-based clinician review catches what a domain expert can see after the fact, but is bounded by sample size and reviewer availability and will miss low-frequency, high-severity events by design.
  • Performance degradation monitoring catches aggregate drift, but a threshold breach is a trailing indicator that typically arrives after the drift has become clinically material.

I proposed a fourth approach that Question 20 gestures toward: continuous per-output verification, run on every clinical assertion instead of a sample or a cadence. It does not test against a fixed set of cases. It checks each live output against the current authoritative source (FDA labeling, clinical guidelines, federal safety guidance) at the moment of generation and logs a citation for every claim it clears or flags. Sampled review answers "was this device safe in the cases we happened to sample." Continuous verification asks "was this specific output defensible, right now," and leaves a complete audit trail instead of a point-in-time snapshot.

A related point: a single passed benchmark run proves very little, because a device can clear a fixed test set once and still perform inconsistently across repeated attempts. A reliability profile from repeated runs of the same prespecified task is more informative: success rate across attempts, consistency under semantically equivalent inputs, and how often the device needed human intervention. Reusable test bundles rerun on a cadence make results comparable across reassessment cycles.

How should re-benchmarking scale with a change?

Re-benchmarking should scale with the reversibility and blast radius of a change, not only its technical nature. In Question 22 I suggested a risk-tiered gate parallel to the two-axis framework the paper already proposes for initial risk assessment. Blast radius means how many patients, how many device functions, and how quickly the change could compound before detection.

A prompt template change that touches a safety-critical escalation pathway may deserve more scrutiny than a model swap that only affects a low-consequence function; the two are different in kind. Changes that are low on both axes could reasonably sit inside a predetermined change control plan, documented in the sponsor's quality management system. Changes high on either axis, especially to safety-critical recognition, scope maintenance, or calibration, should trigger re-benchmarking against the original competency baseline before deployment, not after, however small the technical change looks.

What do agentic systems add?

Two things that single-call evaluation does not cover. These were my answer to Question 26.

Escalation of authority within a single session. An agent's authority can expand mid-task. A system that starts with read-only access to a chart may, over a multi-step task, invoke a tool that grants write access it did not have at session start. Evaluation should test whether the agent recognizes that its own available scope has changed, not only whether it recognizes out-of-scope actions.

Tool-output trust. An agent that treats a tool's output as ground truth has the same self-affirmation problem as a device benchmarking itself. The tool call looks like an external check, but if the tool and the agent share a developer, a training lineage, or an incentive structure, it is not one. I suggested FDA's evaluation criteria explicitly distinguish tool outputs from structurally independent sources from tool outputs that are not.

Where this fits and what to do

FDA has not answered these questions; its paper is a request for feedback. But if you build clinical AI agents, these habits are cheap to start now and hard to retrofit:

  • Do not grade your agent with the same model family that powers it.
  • Treat a single benchmark pass as a starting point. Build reliability profiles from repeated runs.
  • Log tool calls independently of your agent's self-reported status.
  • Evaluate the execution trace, not only the final output.
  • Where an authoritative source exists, check outputs against it, and treat "cannot verify" as a valid, reportable result.

For the broader frame, see why you cannot make clinical AI accurate enough to skip governance and why an AI system cannot audit its own output. I plan to write up the other four questions as their own notes.

What this is and is not

My comment to FDA describes a longer-term architecture: continuous per-output verification of live clinical output against current authoritative sources, integrated through standards such as CDS Hooks. That is the direction, not what exists today.

The Orinyx MCP is smaller. It is free for developers building healthcare AI agents and launches October 29, 2026. It runs on synthetic data only, uses simplified bounty agents, has no dashboard, has no CDS Hooks or EHR integration, and does not monitor live clinical output. It lets you test your agent against adversarial scenarios before it goes anywhere near a patient. An API tier for test environments, and an enterprise tier that depends on completing SOC 2, are plans and not products. The developer waitlist is where launch notices go. The verdict path in the MCP is deterministic rules, not a language model. The semantic evaluator described above belongs to the pattern I am developing and proposed to FDA; it is not part of how the MCP produces a verdict.

This article is not legal or regulatory advice, and it does not predict what FDA will do.

Common questions about independent verification

What is independent verification of a generative AI medical device?

Evaluation by a party with no financial stake in the result and no shared dependence on the device's own model. My comment defines it at two levels: organizational independence and evaluator independence, the second covering any AI judge used in adjudication.

Can an LLM judge another LLM in a clinical setting?

It can, with a constraint. Published research shows LLM judges can favor their own outputs, so I recommend the judge not be the same model family, or a fine-tuned variant, as the device under test. Where a check can be deterministic, it should be.

What is trajectory evaluation?

Scoring an agent's full execution path, including reasoning steps and tool calls, instead of only its final output. It catches a correct-looking answer reached through a path that should have triggered escalation earlier.

Does FDA require any of this today?

No. The discussion paper is for discussion only and is not draft or final guidance.

Further reading

Related in this series