How it stays safe & self-improving

‹Draft — replace with a real story from our floor.› The first question our own pharmacists asked about an AI that suggests substitutes was the right one: “what stops it offering the wrong medicine?” An assistant that’s clever but occasionally dangerous is worse than no assistant. So the safety locks came first, and the selling came second.

The problem

An AI that suggests medicines can’t be “usually right.” Offer an injection in place of a tablet, or sustained-release where immediate-release was meant, and you’ve created a clinical risk — for drugs like carbamazepine or nitroglycerin that’s serious. A recommender tuned only for sales will, sooner or later, make that mistake.

How it stays safe

• Hard clinical locks. Route, form, strength, and release are never relaxed for the auto-pushed up-sell. A tablet is never traded for a suppository; SR is never mixed with IR.
• Tiered substitution. Equivalents are ranked Same molecule → Diff. strength → Same class, and the riskiest tier — same class, different molecule — is a Swap* that requires pharmacist approval, recorded on the sale. The pharmacist at the counter is always the final clinical check.
• Ingredient veto. Misclassified products are normally vetoed automatically when their recorded ingredients disagree with the AI.

How it improves itself

• Flag → suppress. Any cashier can flag a wrong suggestion; it vanishes from their screen at once and is suppressed pharmacy-wide once a second cashier flags the same pair — a guard against one mistaken flag. Admins can resolve flags directly in the Pharmacist Center.
• Honest about uncertainty. With no cost price, an item is unknown-margin and never claimed “higher margin.” With thin sales data, cross-sell holds back rather than guessing — it degrades gracefully instead of over-restricting. Thresholds: Data requirements & thresholds.

What makes it different

A generic recommender treats a wrong suggestion as a missed click. We treat it as a clinical event with a kill-switch and a human in the loop — because at our own counters, the pharmacist who’d catch the mistake is us.

When it doesn’t apply

• Cross-sell has no flag — it’s from real sales, not AI, so there’s no model to correct.
• The locks bound the AI’s suggestions; they don’t replace the pharmacist’s judgement, which remains the final check.