How to Evaluate a Clinical AI
Five questions to put to any clinical AI vendor, from a keynote to South African healthcare leaders — and our own published answers to every one of them.
Respocare Connect AI Team — Respocare (PTY) Ltd · Reg. 2018/411829/07 · Practice No. 9990900010775614. Licensed healthcare practice operating since 2018.
· 10 min read
How to Evaluate a Clinical AI: Five Questions to Ask Any Vendor
Five questions, usable in any vendor meeting by anyone, with no technical background required:
- What evidence tests fabrication, and what are its limits?
- Does it preserve a safety fact, an allergy, across a fragmented history?
- Does it know when to defer?
- Who is the author of record?
- Can you show the messy case?
They were put to a room of South African healthcare leaders at HealthTech Connect 2026 in October, with the invitation to use them on every vendor in the building — this one included. This page is that list, written down, with our own answers to all five at the end.
Why a demonstration is the wrong evidence
A demonstration measures the ceiling. The record is clean, the question is clear, the person driving knows exactly which features to touch, and everyone is watching.
Medicine happens on the floor. The history is scattered across providers. Somebody states something confidently and is wrong. There are no results on file. Nobody is watching.
These are not two different products. They are the same product under two sets of conditions, and only one of those sets is medicine. A system can pass the first and fail the second, which is why the ceiling tells you so little. Ask for the floor.
What each question is actually testing
| The question | What it tests |
|---|---|
| What evidence tests fabrication, and what are its limits? | Whether the answers are grounded in the record — and whether the vendor will state the limits of its own evidence rather than only its strengths. |
| Does it preserve a safety fact across a fragmented history? | Whether the system surfaces an allergy when you did not ask about allergies, and whether an empty field is reported as unknown rather than as none. |
| Does it know when to defer? | Whether it pauses when the evidence is thin, instead of producing something plausible. |
| Who is the author of record? | Who carries clinical responsibility at the moment a draft becomes part of the patient's record. |
| Can you show the messy case? | All four of the above at once, on a record nobody tidied first. |
The two behaviours that earn trust
A clinical agent can be described by five behaviours. It listens, it reasons, and it surfaces what it found — and those three are what every vendor will show you, because they demonstrate well.
The two that earn trust are the ones most often left out.
It defers. When the evidence is thin, it pauses rather than guessing. The refusal is the safe output; a confident answer drawn from a thin record is the hazard, not the help.
It returns control. Every decision stays with the clinician. The system produces a draft, and only a clinician decides what becomes the record.
That is the whole of it, in one line: you are not chatting with it, you are governing it. The principle we repeat until it is boring is the same one: the AI drafts, the record verifies, the clinician decides.
Our answers to our own five questions
A list of questions a vendor will not answer about itself is marketing. So here are ours, each pointing at something already published rather than at a promise.
1. What evidence tests fabrication, and what are its limits?
Our methodology is published in full as the Clinical AI Validation Framework, and our findings as the Clinical Evaluation Report. Both are dated and versioned, so you can see when they were written and what they did and did not cover.
The limit, stated plainly: clinical validation is ongoing, every output requires independent clinician review, and any patient example we publish is a synthetic test record built for evaluation and labelled as such. We would rather you knew the boundary than inferred one.
2. Does it preserve a safety fact across a fragmented history?
Our published position is that no clinician should have to remember to ask whether a patient will be harmed. A trustworthy system surfaces allergies, interactions and critical results whatever the question. If a tool only mentions an allergy when you ask about allergies, it is relying on your memory.
The corollary matters as much: where the allergy field is empty, the honest output is that the status is unknown, not that there are none. Missing and absent are different facts, and a record that blurs them is dangerous in a way that looks tidy.
3. Does it know when to defer?
Where the record does not support a safe answer, the assistant is built to decline and flag the gap rather than fill it with something plausible. That is why we describe the discipline to defer as a behaviour to be measured rather than claimed, and why "the evidence for that is not here" is a correct answer rather than a failure.
4. Who is the author of record?
The clinician, always. The assistant prepares and flags; it does not act. Nothing enters or changes a patient's record without the clinician's explicit action, and we do not build prescribing, diagnosis, or any behaviour the person responsible for the patient cannot see and reverse. These are design commitments, not limitations waiting on a better model.
5. Can you show the messy case?
This is the one we would rather be asked than the other four. Our validation framework is explicitly built around records that are fragmented, information that is incomplete, documentation that evolves across visits and questions that are natural and unscripted — because testing on clean prompts measures the ceiling again.
We also publish The Agentic Report weekly, faults included. A company that only reports its good weeks is asking you to do the verification it should be doing itself.
When the hard part is time
The five questions above are the general test. One area needs its own, because it is where plausible systems most often come apart: reasoning across a record that spans years, where the date attached to a result changes what the result means. If that is your situation, the five narrower questions in longitudinal clinical reasoning go underneath these ones — which date is used, whether the source can be opened, what happens when two records disagree, how you know a document was read, and what the system says when it does not know.
For the mechanics of how an answer is tied back to the document it came from, see traceability in clinical AI. For what separates a model from the system built around it, see the model is not the product.
Where these questions came from
HealthTech Connect returned for its second edition on 28 October 2026 at the Courtyard Hotel in Waterfall City, Midrand. It is organised by Tech Review Africa, which bills it as "South Africa's Premier Digital Health, HealthTech and Healthcare Innovation Summit".
The room was the one these questions were written for: healthcare executives and hospital administrators, Department of Health officials, policymakers and regulators, medical scheme and clinical leaders, chief information and technology officers, founders of South African health technology companies, and academia. Artificial intelligence in healthcare was one of ten themes on the programme, alongside interoperability, data governance, connected care and organisational readiness.
Speakers included the South African Medical Association's head of policy and research and its adviser on AI in healthcare, together with the founders of EMGuidance, Guardian Health and CipherGenix, and clinical and risk leadership from across the private sector. Respocare was a partner of the programme and gave the keynote on agentic AI.
The argument of that keynote was that healthcare is still judging a new kind of clinical AI by the rules of the old kind. A chat window and a governed clinical system can look identical on screen, and the interface alone will never tell you which one you have. The working definition offered to the room was deliberately plain: an intelligence that can reason across context and use tools to act toward a goal. Two words carry the weight. Reason, meaning it works something out from the patient's record rather than fetching a stored answer. Act, meaning it can search, read and draft on its own. That is the freedom, and in healthcare the freedom needs governance.
Which is what the five questions are for. They are not a test of how clever a system is. They are a test of whether anyone can show how it behaves when the conditions stop being favourable.
The two questions to take with you
For any system, from any vendor, including this one:
Which decisions am I governing? And can I prove how it behaves?
Trust is a behaviour, not a claim.
Respocare Connect AI is a clinical assistant and documentation platform, built by Respocare (PTY) Ltd, a licensed South African healthcare practice operating since 2018. It is not a medical device, it is not diagnostic, and it is not a substitute for clinical judgement; every output requires independent clinician review.
Frequently asked questions
What questions should I ask a clinical AI vendor?
Five, and none of them requires a technical background. What evidence tests fabrication, and what are its limits? Does it preserve a safety fact, such as an allergy, across a fragmented history? Does it know when to defer? Who is the author of record? And can you show the messy case — a real record, not a tidied one?
Why is a product demonstration not enough to evaluate clinical AI?
A demonstration measures the ceiling: one clean record, one clear question, everyone watching. Medicine happens on the floor — a history scattered across providers, a confident assertion that is wrong, no results on file, nobody watching. A system can pass the first set of conditions and fail the second, so ask to see the floor.
What does it mean for a clinical AI to defer?
Deferring means pausing when the evidence is thin instead of producing a plausible answer anyway. Where the record does not support a safe answer, the honest output is not a fluent paragraph but a statement that the evidence is not there, with the gap flagged for the clinician.
Who is responsible for an AI-generated clinical note?
The clinician. In our design the assistant drafts, the record verifies, and the clinician decides. Nothing enters or changes a patient record without the clinician acting, and every output requires independent clinician review.
Should a clinical AI mention an allergy if I did not ask about allergies?
Yes. A trustworthy system surfaces allergies, interactions and critical results whatever the question. If a tool only mentions an allergy when you ask about allergies, it is relying on your memory. An empty allergy field should also be reported as unknown rather than as none — missing is never the same as absent.
Is Respocare Connect AI a medical device?
No. Respocare Connect AI is a clinical assistant and documentation platform. It is not a medical device, it does not diagnose or prescribe, and it is not a substitute for clinical judgement. Every output requires independent review by the treating clinician.
Keep reading
Built by Respocare (PTY) Ltd — a licensed South African healthcare practice operating since 2018.