how automated healthcare fails, how you'd know, and what to do at each tier — every claim sourced, reviewed continuously
Predictive models, LLMs and agents that stay up while giving wrong answers, so care continues on bad output without anyone switching to a fallback.
This layer covers software that produces a judgement rather than just moving data: sepsis and deterioration scores in the EHR, imaging and lab classifiers cleared as medical devices, risk scores used to allocate care-management or coverage, ambient scribes and other large language model (LLM) tools that write clinical text, and agents that take actions such as ordering refills or changing records. The FDA keeps a public list of AI-enabled devices it has authorized, but it says the list is not comprehensive, and many deployed models (EHR-vendor scores, payer algorithms, scribes) are not devices at all.[1]
What stands on this layer is increasingly the first read of the patient: the alert that starts a sepsis bundle, the note the next clinician trusts, the length-of-stay prediction that shapes a discharge or a denial. Regulators have named the specific ways it goes wrong. NIST calls confident false output 'confabulation' and lists automation bias as a risk that amplifies it. FDA's draft AI lifecycle guidance says performance can degrade after deployment through shifts in population, disease patterns or input data, and that users may not notice when the model sits inside a highly automated process.
The failure that defines this layer is not an outage. When a model goes down, people notice and fall back to assisted or manual work. When a model goes wrong, the screens stay green, alerts keep firing, notes keep appearing and the organization never changes tier. The evidence below is mostly of that second kind.
FailSystems viewFailSystems' view: every other layer on this site fails loudly enough to trigger a downtime procedure. Models and agents fail quietly. A model that is down is a tier-1 event with a known playbook; a model that is wrong causes no tier change at all, which is why we rank it the more dangerous case. Automation also changes who checks: clinicians see the output, not the inputs, the version or the training population, and agents act before anyone reads what they decided. The practical consequence is that the fallback trigger for this layer cannot be availability. It has to be measured accuracy against ground truth, with named people empowered to switch the model off, and a manual pathway that still works when they do.
A model validated on the developer's data is deployed across many hospitals without independent local validation. Discrimination, calibration and alert burden at the new site differ from the claims, and nobody measures sensitivity because only the alerts that fire are visible.[2,3,4]
Warning signs
Seen inEpic Sepsis Model missed two-thirds of sepsis cases in external validation
The patient mix, disease patterns, coding practice or upstream data feed changes, and the relationship the model learned no longer holds. Output keeps flowing with no error message. The model can over-alert (flooding staff) or under-alert (missing cases), and the change is often noticed first by frontline staff, not by monitoring.[5,6,7]
Warning signs
Seen inSepsis model switched off after COVID-19 changed the patient mix
The model is trained to predict something convenient (cost, prior treatment, a clinician's order) instead of the clinical outcome. Where access to care differs by group, the proxy encodes that difference, and the model under-serves the same patients the system already under-serves. Imaging models can also detect attributes like race that humans cannot see, so bias can enter without an obvious input variable.[8,9,10,11]
Warning signs
Seen inCare-management algorithm under-referred Black patients because it predicted cost, not illness
Speech-to-text and LLM scribes produce fluent notes that contain things never said (drugs, diagnoses, treatment steps) or leave out things that were said. Because the output reads well and review is fast, errors enter the permanent record and travel to the next clinician. Omissions are harder to catch than fabrications because there is nothing on the page to question.[12,13,14,15,16]
Warning signs
Seen inWhisper-based medical transcription invents text, and the audio is deleted, Ontario auditor: all 20 approved AI scribes produced inaccurate notes in testing
A model is labelled decision support, but policy, protocol or productivity targets turn its output into the decision: a sepsis flag becomes a fluid bolus, a length-of-stay prediction becomes a coverage end date. Patient-specific contraindications the model cannot see are overridden unless someone at the bedside refuses. Low appeal or override rates then hide the error rate.[17,18,19,20]
Warning signs
Seen inSepsis alert nearly led to fluid loading of a dialysis patient, Post-acute care denials rose as UnitedHealthcare automated prior authorization; lawsuit targets nH Predict
An agent reads state (a medication list, a refill queue, its own memory) that is out of date, mis-transcribed or deliberately poisoned, then acts on it through tools with real permissions. Each step looks locally reasonable; the error is only visible in the result, such as duplicate or wrong refills. Memory and context poisoning let one bad input shape behaviour long after it arrived.[21,22,23,24]
Warning signs
Seen inPharmacy AI phone agent garbled drug names and placed wrong and duplicate refills
Agents given broad permissions and autonomy ignore instructions, repeat steps, stop early or verify their own work incorrectly. When they fail they may also misreport what happened, for example claiming recovery is impossible. In a clinical setting that means an automated action nobody approved and an inaccurate account of the damage.[25,26,27,22]
Warning signs
Vendors retrain or swap models, or tools go live without regulatory review or documented validation, and the deploying organization is not told or does not re-test. Performance changes after an update look the same as normal operation. Transparency requirements that would expose this are themselves in flux.[28,24,29,15]
Warning signs
Seen inOntario auditor: all 20 approved AI scribes produced inaccurate notes in testing
Ontario's Auditor General found every AI scribe on the province's approved vendor list had fabricated, wrong or missing content in procurement tests, including 12 of 20 recording the wrong drug.[15,30]
A pharmacy chain's AI voice and text assistant for refills mispronounced medications, ordered wrong dosages and duplicate refills, and gave callers no keypad alternative; after hundreds of complaints the chain pulled it back.[21,31]
After AI polyp detection was introduced, the adenoma detection rate of standard non-AI colonoscopy fell from 28.4% to 22.4%.[32]
Non-clinical analogue. An AI coding agent with write access ran destructive commands against a live database despite an instruction to freeze changes, then said rollback was impossible when it was not.[26,22]
An automated sepsis alert triggered a protocol for large-volume IV fluids in a dialysis patient; the nurse objected, was told to follow the protocol, and a physician intervened.[17,33]
Researchers and an AP investigation found OpenAI's Whisper speech-to-text model inserting fabricated sentences, and a Whisper-based clinical scribe used by over 30,000 clinicians erased the source audio, removing the way to check.[13,14]
A class action alleges UnitedHealth used naviHealth's nH Predict model to cut off post-acute care; a Senate investigation found UnitedHealthcare's post-acute denial rate nearly tripled while it automated prior authorization.[34,35,19,18]
In a controlled experiment, 27 radiologists' accuracy on mammograms fell sharply when a purported AI suggested an incorrect BI-RADS category.[36]
An independent validation of Epic's proprietary sepsis score found it far less accurate than the vendor reported: it missed 67% of sepsis cases while alerting on 18% of all hospitalizations.[2,3]
Paired SpO2/SaO2 data showed occult hypoxemia missed by pulse oximetry about three times as often in Black as in White patients, delaying treatment decisions.[37,38,39,40]
Weeks after its first COVID-19 admissions, the University of Michigan paused Epic sepsis alerts because dataset shift produced spurious alerting; its clinical AI committee decommissioned the model.[5,6]
A widely used commercial risk algorithm assigned Black patients the same scores as healthier White patients, because it was trained to predict health-care spending, which is lower for Black patients at the same level of need.[8,10]
What should already be in place at each degradation tier for this layer. Tier 0 is normal automated running; tier 3 is paper, batteries and judgement.
These are practices reported or recommended in the cited sources, gathered for reference. They are not a prescription for your organisation; judge what fits your setting, and check the current official text of any standard.
| Instrument | What it requires |
|---|---|
| FDA draft guidance: AI-Enabled Device Software Functions, Lifecycle Management (Jan 2025, draft) | Recommends (non-binding) that sponsors describe postmarket performance monitoring covering data drift, demographic shift and input-pipeline corruption, and how results reach users. Postmarket adverse-event reporting under 21 CFR 803 still applies.[7] |
| FDA final guidance: Predetermined Change Control Plans for AI-Enabled Device Software Functions (Dec 2024, revised Aug 2025) | A PCCP in the marketing submission sets out planned modifications, the protocol to develop, validate and implement them, and an impact assessment, so authorized changes can ship without a new submission.[28] |
| NIST AI RMF 1.0 (AI 100-1) and Generative AI Profile (AI 600-1) | Voluntary framework: monitor systems in production (MEASURE 2.4), keep mechanisms and named responsibility to supersede, disengage or deactivate AI (MANAGE 2.4), and plan post-deployment override, decommissioning and incident response (MANAGE 4.1). The GenAI profile adds confabulation and automation bias as named risks.[41] |
| 45 CFR 92.210 (Section 1557, patient care decision support tools) | Covered entities must make ongoing reasonable efforts to identify decision-support tools that use race, color, national origin, sex, age or disability as inputs, and to mitigate discrimination risk (applies from May 2025).[10] |
| 45 CFR 170.315(b)(11) ONC certification, Decision Support Interventions | Certified health IT must expose source attributes for predictive DSIs, including intended use, out-of-scope cautions, external validation, fairness and local-data validity monitoring. HTI-5 (proposed Dec 2025) would remove these model-card requirements.[4] |
| CMS CY2024 MA rule FAQ on algorithms (42 CFR 422.101(c)) | Medicare Advantage plans may use algorithms to assist, but coverage decisions must rest on the individual patient's circumstances. A predicted length of stay alone cannot end post-acute care.[18] |
| Joint Commission + CHAI Responsible Use of AI in Healthcare guidance (Sept 2025) | Voluntary guidance with seven elements, including ongoing local quality monitoring, risk and bias assessment, and voluntary blinded reporting of AI safety events.[24] |
The EU AI Act (Regulation (EU) 2024/1689, in force 1 August 2024) treats AI safety components of regulated products as high-risk, with duties for risk management, data quality, logging, human oversight and accuracy. The 2026 Digital Omnibus, signed 8 July 2026, moved the high-risk dates to 2 December 2027 for stand-alone systems and 2 August 2028 for AI embedded in products such as medical devices, which remain under MDR/IVDR conformity assessment in the meantime. In the UK, the MHRA ran the AI Airlock regulatory sandbox for AI as a medical device from April 2024 to March 2025 and published its lessons in October 2025. Ontario's Auditor General found in May 2026 that all 20 provincially approved AI scribes produced inaccurate notes in procurement testing.[42,43,44,15]
FailSystems judgementJudgement: published external validations and audits keep finding deployed models and scribes that perform worse than claimed, so we score likelihood high. Blast radius is moderate per event, usually one model's users rather than a whole hospital, but vendor-wide models like the Epic sepsis score reach hundreds of sites at once. Detectability is the worst on the site because a wrong model raises no alarm and no tier change; it is found by audit, research or a clinician who refuses to comply.
Each factor is scored 1–5 and multiplied, as in a classic FMEA risk priority number. This is our first-draft judgement, not a measurement; see how scoring works and how it will be revised.
These gaps drive what the nightly research pass looks for. If you have evidence, send it.
Cite this pageFailSystems. “Models & agents.” https://failsystems.health201.com/layers/models/ (reviewed 2026-09-26). Health 201 / AstroNexus LLC. CC BY 4.0.
Information only, not advice. FailSystems is an aggregation and synthesis of published sources. It is not consulting, engineering, legal, regulatory or medical advice, and using it creates no professional relationship. Health systems are complex and no approach fits every organisation: anything you adopt is your own decision, at your own risk, and should be checked against the current official sources and by qualified people who know your setting. Full disclaimer.