Eight failures to look for, with an example and a way to catch each
1. Made-up facts
What it looks like: The summary mentions a test or diagnosis that isn’t in the chart.
How to catch it: Check every claim against its source passage.
2. Stale or wrong source
What it looks like: The answer follows a withdrawn policy, or a national guideline that contradicts your protocol.
How to catch it: Test with draft, withdrawn and outdated documents loaded on purpose.
3. Missed negation or certainty
What it looks like: “Ruled out pneumonia” is read as “has pneumonia,” or “suspected” becomes “confirmed.”
How to catch it: Test pairs of near-identical statements with opposite meanings.
4. Copied-forward facts
What it looks like: An old medication keeps appearing because every note repeats it.
How to catch it: Compare AI answers with structured data such as the medication list.
5. Citation that doesn’t support the claim
What it looks like: The answer links to a real document, but the passage says something else.
How to catch it: Audit a sample of citations each month, reading the passage itself.
6. Wrong “I don’t know”
What it looks like: It declines when the answer is in the chart, or answers confidently when it isn’t.
How to catch it: Track both kinds of mistakes, not just overall accuracy.
7. Wrong patient or restricted data
What it looks like: Information from another encounter, a duplicate record, or a chart the user shouldn’t see.
How to catch it: Test with restricted users and duplicate or merged records.
8. Missing documentation treated as “no”
What it looks like: “Not documented” is read as “doesn’t have it,” so a summary says there is no history of a condition.
How to catch it: Ask questions where the chart is silent and check the wording of the answer.