{ "last_reviewed": "2026-09-26", "tiers": [ { "n": 0, "name": "Full automation", "definition": "The normal state of a digital hospital. Machines acquire and route data (monitors, interfaces, lab analysers with autoverification), analyse it (early-warning scores, results flags), recommend (CPOE decision support, sepsis or deterioration models) and in places act (barcode-gated administration, smart-pump limits, automated dispensing). Clinicians supervise and decide, but much of the checking has moved into software they no longer see. In levels-of-automation terms, most functions sit in the middle of the scale: the computer narrows or suggests and the human approves (Parasuraman, Sheridan and Wickens 2000). Tier 0 is only safe if you can tell when you have left it: the literature shows automation can fail silently while appearing to work (Wright 2016).", "what_must_be_true": [ "Power, network, EHR, interfaces and decision support are all up, and you have evidence they are working, not just the absence of complaints.", "Measure EHR response time for key tasks continuously; ONC's target is under 2 seconds (SAFER Contingency Planning 3.1).", "A read-only downtime viewer is refreshed at least hourly, tested at least weekly and can print (SAFER 1.4).", "Backups are daily, off-site, encrypted, air-gapped from normal storage, and full restores are tested (SAFER 1.4).", "Monitor decision-support alert firing rates so a rule that stops firing is noticed (Wright 2016).", "Downtime plans, paper forms and staff training exist before they are needed; the CMS rule and Joint Commission EM chapter require exercised plans (42 CFR 482.15(d); EM.13.01.01, EM.16.01.01)." ], "how_you_know_you_are_here": [ "Synthetic transactions (e.g., a test-patient order queried every minute) complete within target times (SAFER 3.2).", "Interface queues are empty or draining; no lab, pharmacy or imaging messages are stuck in buffers (SAFER 2.5).", "Alert volumes per rule stay within expected ranges; a sudden drop to zero is a failure, not good news (Wright 2016).", "The downtime viewer's last-refresh time is current." ], "how_you_get_back_up": [ "Re-enter tier 0 only after a formal uptime announcement sent through a channel independent of the EHR (SAFER 2.2).", "Restart interfaces in a defined order, confirm buffers are empty, and check that no in-transit data was lost (SAFER 2.5).", "Verify that decision-support rules fire on test patients before relying on them; an upgrade or code change can silently break or flood alerts (Wright 2016).", "Confirm back-entered data from lower tiers is complete before automation acts on it (TJC SEA 67; ASPR TRACIE 2022)." ], "source_ids": [ "parasuraman-sheridan-wickens-2000-ieee", "wright-2016-jamia-cds-malfunctions", "onc-safer-contingency-2025", "ecfr-42cfr482-15", "tjc-em-reference-guide-2022", "tjc-sea-67-cyberattack", "aspr-tracie-cyber-readiness-response" ], "short": "Normal digital operation: software acquires, routes and checks data, recommends, and in places acts; clinicians supervise and decide." }, { "n": 1, "name": "Assisted operation", "definition": "The systems are still running, but you can no longer trust them to do their part unsupervised. The EHR is slow enough to cause errors, one interface has stopped, a model or alert is misfiring, or an upgrade changed behaviour. Automation is demoted to an advisory role: humans make each decision and add the checks the software used to do. This is the most dangerous tier to miss, because screens still look normal. Automation bias and complacency push people to keep trusting output that has become wrong (Goddard 2012; Parasuraman, Sheridan and Wickens 2000), and CDS failures often go undetected for long periods (Wright 2016).", "what_must_be_true": [ "Someone has authority to declare that a specific system is untrusted, even while it is technically up, and to say which functions staff must now check manually.", "Clinicians know which safety checks the software normally performs (dose ranges, interactions, allergy alerts, result routing) so they can perform them deliberately.", "Pharmacists and lab staff have a rapid channel to flag results or orders that look wrong.", "Users can report slow response easily, and IT treats slowness as a safety event (SAFER 3.2)." ], "how_you_know_you_are_here": [ "Hourly mean response time above 5 seconds, or more than 3 standard deviations above the mean: ONC's definition of functional downtime (SAFER 3.2).", "An alert that normally fires daily has gone quiet, or a rule starts firing on patients it should not (Wright 2016).", "Results are missing, late or duplicated; clinicians report futile searches for results (Wang 2016).", "A vendor, peer hospital or public source reports a fault in a component you use (e.g., the July 2024 CrowdStrike update took patient-facing services offline at about a third of scanned US hospitals; Tully 2025).", "Clinicians find the software's advice is repeatedly contradicted by bedside findings." ], "how_you_get_back_up": [ "Fix the root cause, then prove the fix: run the affected rule or interface against test patients and compare output with expected values before withdrawing manual checks.", "Reconcile anything that passed through the degraded system during the window (orders, results, alerts that did or did not fire).", "Announce the return to normal explicitly, naming the system and the time the manual checks stop.", "Review the event: SAFER 3.1 asks that downtime events, including functional downtimes, be logged and reported to leadership." ], "source_ids": [ "goddard-2012-jamia-automation-bias", "parasuraman-sheridan-wickens-2000-ieee", "wright-2016-jamia-cds-malfunctions", "onc-safer-contingency-2025", "wang-2016-jbi-downtime-pathology", "tully-2025-jama-netw-open-crowdstrike" ], "short": "Automation degraded or distrusted — humans decide, machines advise." }, { "n": 2, "name": "Manual operation", "definition": "Electronics are up but the cognitive layer is down. Power, monitors, pumps, phones and devices work in standalone mode, but the EHR, CPOE, decision support, barcode verification or lab/pharmacy interfaces are unavailable. Care runs on paper forms, printouts from a read-only viewer, fax, runners and people's memory. This is the tier the downtime literature describes. It is slower and error-prone: lab result reporting was 62% slower on average during downtime in one study (Larsen 2019), a 17-minute results-system outage multiplied clinician read times several-fold (Wang 2016), and in 46% of downtime-related safety reports procedures were not followed or not in place (Larsen 2018). Ransomware can hold a hospital here for weeks. The Joint Commission tells hospitals to prepare for four weeks or longer (SEA 67).", "what_must_be_true": [ "Enough paper forms in every care area for at least 8 hours of orders, medication administration, lab and imaging (SAFER 1.3); ASPR TRACIE suggests pre-printing for a set period such as 48 hours.", "Paper forms match the current electronic workflow, not forms retired years ago (TJC SEA 67).", "A read-only downtime viewer or recent printed snapshot gives access to allergies, medications, problems and recent results (SAFER 1.4; ASPR TRACIE 2022).", "A way to register new patients with unique temporary record numbers that can be reconciled later (SAFER 1.5).", "Communication that does not depend on the EHR network: call trees, radios, overhead paging, runners (SAFER 2.2).", "Manual double-checks for high-alert medications in place of barcode and CDS checks (TJC SEA 67).", "Ordering is reduced to what is needed, because manual lab throughput is far below automated capacity (Larsen 2019).", "HICS or an equivalent is activated for anything beyond a short outage, with separate IT and clinical branches (HICS 2014; ASPR TRACIE facility assessment)." ], "how_you_know_you_are_here": [ "A designated person has declared downtime against pre-set thresholds, e.g., expected duration over two hours (ASPR TRACIE facility assessment); ONC advises activating the warm site before two hours of unplanned unavailability (SAFER 2.3).", "Downtime is classified by expected duration so the response scales, e.g., 12 hours or less, over a day, over three days (ASPR TRACIE 2022).", "When duration is unclear, the ASPR guidance says to assume the longer downtime (ASPR TRACIE 2022).", "Workarounds appear before a declaration: staff printing, phoning results, keeping private lists. Treat these as a signal that you are already here." ], "how_you_get_back_up": [ "For cyber events, confirm the attacker has been removed before restoring; restoring services without eradication can re-infect the network (ASPR TRACIE 2022).", "Restore in stages by clinical priority; systems will not come back at once (ASPR TRACIE 2022; Synnovis restored by clinical criticality over about five months).", "Keep downtime procedures running on each unit until its systems are verified; treat recovery as part of downtime (ASPR TRACIE 2022).", "Back-enter orders as coded data and scan other paper; authenticate every transcribed order (SAFER 1.3; TJC SEA 67).", "Reconcile medications with doctor-pharmacist pairs; one district needed 2-3 hours in high-use areas after an 8-hour downtime (Lyon 2023).", "Merge temporary patient IDs into permanent records (SAFER 1.5).", "Add staff for data entry so bedside staff are not doing catch-up documentation on top of care (ASPR TRACIE 2022)." ], "source_ids": [ "larsen-2019-aci-downtime-laboratory", "wang-2016-jbi-downtime-pathology", "larsen-2018-jamia-ehr-downtime-events", "tjc-sea-67-cyberattack", "onc-safer-contingency-2025", "aspr-tracie-facility-downtime-assessment", "aspr-tracie-cyber-readiness-response", "hics-guidebook-2014", "lyon-2023-aci-downtime-uptime", "synnovis-cyber-update", "hanuscak-2009-ajhp-downtime-med-errors", "martin-2019-lancet-dh-hit-failures" ], "short": "Electronics up, cognition down. Staff run the workflow on devices alone." }, { "n": 3, "name": "Analog fallback", "definition": "Power, the network or both are gone, or cannot be trusted. Generators have failed or do not cover the load, UPS batteries are running down, phones and paging may be out, and HVAC and lighting may be lost. What remains is paper, battery-powered devices while their charge lasts, manual techniques (bag-valve ventilation, gravity infusions, cylinder oxygen, direct observation instead of telemetry) and clinical judgement. At this tier the question is often whether to shelter in place or evacuate. NYU Langone evacuated 21 NICU patients in 4.5 hours after Hurricane Sandy's surge cut power in 2012 (Espiritu 2014).", "what_must_be_true": [ "Staff are trained for complete power failure: monitoring ventilator and device batteries, bagging, gravity drips, switching to cylinder oxygen, with paper checklists (ASPR TRACIE facility assessment).", "Critical equipment on emergency power is known; equipment not on backup power (e.g., call buttons) has a monitoring plan (ASPR TRACIE facility assessment).", "Emergency lighting and battery backups for key lights and access controls are in place (ASPR TRACIE facility assessment).", "Generators and fuel meet the hospital's plan; ONC suggests 2 days of fuel on site and monthly testing (SAFER 1.2); the Joint Commission requires a 96-hour sustainability plan (EM.12.02.11).", "Evacuation and transfer plans, and receiving-hospital agreements, exist and account for regional outages (42 CFR 482.15(b)(3),(b)(7); TJC SEA 67).", "Printed contact lists, on-call lists and the downtime policy are available on each unit and off site (SAFER 2.3; ASPR TRACIE facility assessment)." ], "how_you_know_you_are_here": [ "Loss of normal power with generator or transfer failure, or UPS alarms with no generator pickup.", "Loss of both data and voice communications at once.", "Loss of HVAC in areas that need temperature control (ORs, server rooms, pharmacy storage) (ASPR TRACIE facility assessment).", "Pre-defined triggers for diversion, transfer or evacuation have been met (ASPR TRACIE facility assessment 1.13.1)." ], "how_you_get_back_up": [ "Before power returns, turn off and unplug unused equipment to prevent surge damage (ASPR TRACIE facility assessment).", "Restore power and communications first, then climb to tier 2: run paper procedures while electronic systems are checked.", "Check life-support devices and biomedical equipment before putting them back in service; recovery may require reloading software on capital equipment (ASPR TRACIE 2022).", "If you evacuated, repatriate patients deliberately and rebuild their records from what travelled with them (ASPR TRACIE 2022; Espiritu 2014).", "Run a full after-action review; CMS counts a real activation as an exercise only if it is documented and analysed (42 CFR 482.15(d)(2); SOM Appendix Z)." ], "source_ids": [ "espiritu-2014-pediatrics-nicu-sandy", "aspr-tracie-facility-downtime-assessment", "onc-safer-contingency-2025", "tjc-em-reference-guide-2022", "ecfr-42cfr482-15", "tjc-sea-67-cyberattack", "aspr-tracie-cyber-readiness-response", "cms-som-appendix-z" ], "short": "Power or network gone. Paper, batteries, hand calculation, judgement — and a rehearsed plan." } ], "matrix": { "power": { "0": "The whole power chain (generators, fuel, transfer switches, cooling) is sited above flood level, and production and recovery IT do not share a cloud region or a grid.", "1": "Partial outages have clinical triggers: slow labs or order entry open downtime command even while systems are technically up.", "2": "Generators are load-tested, the EHR and downtime workstations ride on UPS, there is fuel for two days, and everyone knows which devices are on emergency outlets.", "3": "Paper operation for weeks and evacuation without lifts have been rehearsed, and every evacuated patient leaves with a paper summary." }, "connectivity": { "0": "Phishing-resistant MFA on all remote and vendor access, air-gapped backups, redundant network paths, and contracts that commit third parties to notification and recovery times.", "1": "A segmented network, a warm site that can take the whole EHR within hours, and a contracted alternate clearinghouse and reference lab.", "2": "A read-only EHR refreshed hourly and printable on backed-up power, a communication channel off the EHR network, and a decision to call downtime within 2 hours.", "3": "Current paper forms on every unit, runners to move orders and results, regional mutual aid for cyber incidents, and a planned recovery phase for back-entry." }, "devices": { "0": "An inventory with firmware and support dates, SBOMs and patch timelines in contract, staged update rings, and accuracy data by skin tone for oximeters.", "1": "Clinicians verify pumps against the current order and question readings that don't fit; clinical devices sit on their own network and can still monitor locally.", "2": "Standalone monitors and pumps stocked on critical units, offline recovery kits for endpoints, and manual infusion programming with a double-check.", "3": "Paper flowsheets, manual BP cuffs and gravity sets on hand, the skills to use them, and a decided list of what elective work stops." }, "models": { "0": "Every model is validated locally before go-live, has a named owner and monitoring plan, logs its version with each output, and agents act with least privilege.", "1": "Someone is authorised to switch a model off on a defined trigger, a visible 'model off' banner shows it, and override is explicit and unpenalised.", "2": "The pre-AI workflow is documented, staffed and rehearsed, and patients always have a non-AI route to care.", "3": "Paper versions of the criteria models encode (sepsis screens, early-warning scores), and drills where the EHR is up but a model is known to be wrong." }, "handoff": { "0": "Clinicians keep unaided performance measured, every alarm and alert has a named responder, and each AI tool's level of automation is written down.", "1": "Every mode change is shown on screen and says what the clinician now owns, with takeover procedures for each known failure signature.", "2": "Alarm limits are set per patient, and who may silence or widen them is written down and audited; critical alarms are audible wherever the responder is.", "3": "Unannounced downtime drills on every unit each year, with every clinician trained on paper ordering and on finding the read-only EHR." }, "cascades": { "0": "A dependency map from each clinical service to its suppliers, networks, devices and utilities, with primary and backup never sharing a single point of failure.", "1": "Alternates are enrolled for every concentrated supplier, and disconnection criteria are agreed with partners before an incident.", "2": "Neighbours and EMS hear early, services are restored in order of clinical criticality, and all staff (not only IT) can run an extended cyber downtime.", "3": "Signed transfer agreements, plans for region-wide events where every neighbour is down, and space, power and oxygen for the community's electricity-dependent patients." } }, "rehearsal": { "text": [ "US hospitals are required to exercise. Under the CMS emergency preparedness rule, a hospital must run two exercises a year: an annual full-scale community exercise or facility functional exercise, plus a second one that may be a facilitated tabletop. It must analyse every drill, tabletop and real event, and revise its plan (42 CFR 482.15(d)(2)). The Joint Commission's 2022 EM chapter mirrors this: two exercises a year, with after-action reports reviewed by a committee and sent to senior leaders (EM.16.01.01, EM.17.01.01). CMS guidance says not to test the same scenario every year (SOM Appendix Z). None of these rules names EHR downtime or automation failure as a required scenario. ONC's SAFER guide goes further: it recommends unannounced EHR downtime drills at least once a year, and the Joint Commission suggests drilling annually or quarterly depending on staff turnover (SAFER 2.1; SEA 67).", "The evidence that drills work is thin and mostly descriptive. A systematic review of hospital mass-casualty training found drills helped staff learn procedures and exposed weak points in command, communications and patient flow, but study quality was poor and no included study evaluated tabletop exercises (Hsu 2004). A 2025 scoping review found tabletops were the most common hospital training method but could not identify a best one (Malek 2025). Studies of downtime drills are single-site reports: unit drills audited against a checklist (Kashiwagi 2016), a live PACS-offline drill (Dhamija 2022), a quarterly exercise tied to EHR upgrades (Bulson 2024), six-monthly random-unit drills (Lyon 2023), and a nurse escape room with self-reported gains (Rossley 2022). None measures patient outcomes.", "Real downtimes show what drills should target. In Larsen 2018, 46% of downtime-related safety reports described procedures that were not followed or not in place. Clinicians kept ordering tests at normal rates during downtime (Larsen 2019). In a simulation, clinicians did not recognise that a patient's deterioration came from a compromised device (Dameff 2018). Staff surveyed after a well-planned downtime still did not know where to find resources (Lyon 2023). FailSystems' reading: exercises should test detection and the declaration decision, the recovery and back-entry phase, and tier 1, not only the switch to paper." ], "practices": [ { "text": "Run at least one unannounced EHR downtime drill a year on randomly chosen units. Check that staff can find the downtime kit, log in to the read-only viewer and print a medication record.", "source_ids": [ "onc-safer-contingency-2025", "lyon-2023-aci-downtime-uptime" ] }, { "text": "Use a different failure scenario each year. Rotate through EHR ransomware, a single interface failure, a silent CDS malfunction, a cloud or vendor outage, and generator failure.", "source_ids": [ "cms-som-appendix-z", "wright-2016-jamia-cds-malfunctions", "tully-2025-jama-netw-open-crowdstrike" ] }, { "text": "Include the uptime phase in every exercise: back-entry, medication reconciliation, merging temporary IDs, restarting interfaces and the uptime announcement.", "source_ids": [ "lyon-2023-aci-downtime-uptime", "onc-safer-contingency-2025", "nelson-2007-jcc-downtime-procedures" ] }, { "text": "Tabletop the declaration decision itself. Practise who declares downtime, on what threshold, and how units are told when the network is down.", "source_ids": [ "aspr-tracie-facility-downtime-assessment", "aspr-tracie-cyber-readiness-response", "tjc-sea-67-cyberattack" ] }, { "text": "Run clinical simulations in which the cause is a failed or compromised device or system, so clinicians practise suspecting the technology.", "source_ids": [ "dameff-2018-jem-cyber-simulation" ] }, { "text": "Include lab, pharmacy and blood bank in drills, and practise reducing orders, because manual capacity is a fraction of automated capacity.", "source_ids": [ "larsen-2019-aci-downtime-laboratory", "tjc-sea-67-cyberattack" ] }, { "text": "Exercise with neighbouring hospitals and your healthcare coalition. A multi-week outage at one system sends ambulances, stroke patients and blood demand to the others.", "source_ids": [ "dameff-2023-jama-netw-open-adjacent-eds", "nhsbt-2024-o-type-appeal", "tjc-sea-67-cyberattack" ] }, { "text": "Treat the published drill studies as single-site descriptions, not proof: drills improve familiarity and show up gaps, but no study links them to patient outcomes. Measure your own: time to find the kit, time to first paper order, back-entry backlog.", "source_ids": [ "hsu-2004-pdm-mci-training-review", "malek-2025-dmphp-cbrne-training-review", "kashiwagi-2016-ajmq-clear-toolkit", "dhamija-2022-pediatr-radiol-pacs-drill", "bulson-2024-jbcep-quarterly-exercises", "rossley-2022-jnpd-escape-room", "aspr-tracie-ehr-downtime-collection" ] }, { "text": "Write an after-action report for every drill and every real downtime: what was supposed to happen, what happened, what went well, what to change, and dates. Send it to senior leadership. Do a root-cause review of any unplanned downtime over 24 hours.", "source_ids": [ "cms-som-appendix-z", "tjc-em-reference-guide-2022", "onc-safer-contingency-2025" ] } ] }, "recovery": { "text": [ "Coming back up is its own hazardous phase, and guidance treats it as part of downtime rather than its end. Restore systems in stages by clinical priority, keep downtime procedures running on each unit until its systems are verified, and for cyberattacks, confirm the attacker is gone before restoring, since restoring without eradication can leave the network compromised (ASPR TRACIE 2022). Restart interfaces in order with empty buffers, because in-transit data can be lost without warning (SAFER 2.5). Before you withdraw manual checks, check that automation behaves as expected. Wright 2016 found that CDS rules can silently stop firing, or fire spuriously after an upgrade. That is why FailSystems suggests running test patients through key rules before announcing uptime.", "Back-loading paper is heavy, slow work and needs its own staff. SAFER asks for a process to enter orders as coded data and scan other paper after reactivation, and to merge temporary patient IDs (SAFER 1.3, 1.5). The Joint Commission says to authenticate every transcribed order and to assign staff for data entry (SEA 67). One district needed 2-3 hours of doctor-pharmacist medication reconciliation in high-use areas after an 8-hour planned downtime (Lyon 2023). Paper records made during real downtimes are often incomplete (Larsen 2019), and some manually recorded data may be deliberately left out of the EHR. ASPR advises documenting what that data is and where it is kept (ASPR TRACIE 2022). As early as 2000, LDS Hospital found that recovery from planned downtimes was not smooth, and it wrote down exactly which data had to be re-entered (Nelson 2007)." ], "source_ids": [ "aspr-tracie-cyber-readiness-response", "onc-safer-contingency-2025", "wright-2016-jamia-cds-malfunctions", "tjc-sea-67-cyberattack", "lyon-2023-aci-downtime-uptime", "larsen-2019-aci-downtime-laboratory", "nelson-2007-jcc-downtime-procedures", "hics-guidebook-2014", "hscc-occi-2022" ] }, "levels_of_automation": { "text": "The four tiers are FailSystems' own operational framework, not an established standard. They borrow from the levels-of-automation literature. Sheridan and Verplank's 1978 ten-level scale runs from the human doing everything to the computer acting alone. Parasuraman, Sheridan and Wickens (2000) applied it separately to four functions: information acquisition, analysis, decision selection and action. A hospital at tier 0 runs different functions at different levels. Tiers 1-3 describe what happens when those levels are forced down in a failure, rather than chosen at design time. Tier 1 lowers decision and action automation while keeping information automation. Tier 2 removes analysis and decision automation but keeps device-level acquisition and action. Tier 3 also loses most acquisition and action. The same literature warns that high automation brings reduced situation awareness, complacency and skill loss, which matter most when automation fails (Parasuraman et al. 2000). Bainbridge's 'ironies of automation' makes the same point: automating a task can make the human's remaining job harder. Automation bias in clinical decision support is well documented (Goddard 2012). SAE J3016's driving levels 0-5 are a useful vocabulary analogue only. Its 'fallback' and 'minimal risk condition' concepts roughly match a planned drop to a lower tier, but J3016 has no status in healthcare.", "source_ids": [ "sheridan-verplank-1978", "parasuraman-sheridan-wickens-2000-ieee", "bainbridge-1983-automatica-ironies", "goddard-2012-jamia-automation-bias", "sae-j3016-2021" ] }, "intro": [ { "text": "The tiers describe how much of the automation is still working and trusted, not how serious the incident is. A hospital can be at tier 1 for one layer (a sepsis model switched off) while every other layer runs at tier 0. The layer × tier matrix below is the design question FailSystems asks of every clinical workflow: for each layer, what must already be true if you drop to this tier today?", "source_ids": [] }, { "text": "US rules already require practice. The CMS emergency preparedness rule and the Joint Commission's emergency management standards require two exercises a year with an after-action review, but neither requires the scenario to be an EHR downtime or an automation failure. ONC's SAFER guide goes further and recommends unannounced downtime drills at least once a year.", "source_ids": [ "ecfr-42cfr482-15", "onc-safer-contingency-2025" ] } ], "last_audited": "2026-09-26", "audit_status": "verified" }