Eight numbers to review each month after go-live
1. Clinician flag rate
What to track: How often clinicians report a wrong or unhelpful answer.
Warning sign: A sudden rise, or a rate near zero (which may mean people have stopped reporting).
2. Flags by severity tier
What to track: Flags sorted using your severity scale.
Warning sign: Any Tier 1 (Unsafe) case, or Tier 2 cases rising.
3. “I don’t know” rate
What to track: How often it declines, and whether those declines were right.
Warning sign: A big shift in either direction after a content or system change.
4. Citation support rate
What to track: In a monthly sample, how many cited passages truly support the claim.
Warning sign: Any decline from your baseline.
5. Test set results
What to track: Pass rate on your clinician-graded cases, re-run after every EHR, template, content or model change.
Warning sign: Fewer passes after a change nobody thought would matter.
6. Time to fix
What to track: Days from a flag to a resolved fix, by tier.
Warning sign: Flags sitting with no named owner.
7. Content freshness
What to track: Sources past their review date, and how fast approved updates reach the AI.
Warning sign: Overdue or withdrawn documents still in use.
8. Active use
What to track: Number of clinicians using it, and how often they override or ignore answers.
Warning sign: Use dropping after a bad experience.
Pro Tips:
Set your own thresholds from your first 90 days of data. A generic target won’t fit your setting.
Review the scorecard with a clinician and the content owner in the room, not just IT.