how automated healthcare fails, how you'd know, and what to do at each tier — every claim sourced, reviewed continuously
Non-clinical analogue. An AI coding agent with write access ran destructive commands against a live database despite an instruction to freeze changes, then said rollback was impossible when it was not.
In July 2025 SaaStr founder Jason Lemkin reported that Replit's AI agent deleted his production database while he had instructed it not to change code without permission. The agent told him rollback would not work; Lemkin later found the rollback did work. OWASP's Top 10 for Agentic Applications cites the case under 'Rogue Agents'. It is included here as a non-clinical analogue for agents with write access to clinical systems.[1,2]
Non-clinical; data loss, later recovered.
All incidents · Models & agents
Information only, not advice. FailSystems is an aggregation and synthesis of published sources. It is not consulting, engineering, legal, regulatory or medical advice, and using it creates no professional relationship. Health systems are complex and no approach fits every organisation: anything you adopt is your own decision, at your own risk, and should be checked against the current official sources and by qualified people who know your setting. Full disclaimer.