Care-management algorithm under-referred Black patients because it predicted cost, not illness

Incident

Care-management algorithm under-referred Black patients because it predicted cost, not illness

October 2019United States (commercial algorithm; the authors say it affects millions of patients)Study findingModels & agents

A widely used commercial risk algorithm assigned Black patients the same scores as healthier White patients, because it was trained to predict health-care spending, which is lower for Black patients at the same level of need.

Sources checked when written 26 September 2026

What happened

Obermeyer and colleagues analysed a commercial algorithm that health systems use to select patients for high-risk care-management programmes. At any given risk score, Black patients had more uncontrolled chronic illness than White patients. The cause was the training label: the model predicted future cost, and unequal access to care means less is spent on Black patients with the same needs. Correcting the disparity would have raised the share of Black patients receiving additional help from 17.7% to 46.5%. The paper does not name the vendor.^[1,2]

Documented harm

Systematic under-referral of Black patients to extra care; individual patient harm not quantified.

What it teaches

Check what the model is actually trained to predict; a convenient proxy can encode inequity.
Aggregate accuracy can look good while one group is badly served; report by subgroup.
Section 1557 now requires covered entities to look for and mitigate this risk in decision-support tools.

Sources

Dissecting racial bias in an algorithm used to manage the health of populations. Science (Obermeyer Z, Powers B, Vogeli C, Mullainathan S), 25 October 2019. Primary Peer-reviewed · link checked 2026-09-26
45 CFR 92.210 Nondiscrimination in the use of patient care decision support tools (Section 1557 rule). U.S. Department of Health and Human Services (eCFR), 6 May 2024. Primary Regulation · link checked 2026-09-26

All incidents · Models & agents

Information only, not advice. FailSystems is an aggregation and synthesis of published sources. It is not consulting, engineering, legal, regulatory or medical advice, and using it creates no professional relationship. Health systems are complex and no approach fits every organisation: anything you adopt is your own decision, at your own risk, and should be checked against the current official sources and by qualified people who know your setting. Full disclaimer.

Dealing with an incident right now? This site is a reference, not an incident-response service. Activate your organisation's emergency operations plan and incident command, and:

Life-threatening emergency: call 911.
Cyberattack or ransomware: report to CISA and the FBI field office near you (or IC3).
Power loss, disaster or resource needs: go through your local or county emergency management. They escalate to the state, and the state requests FEMA support; hospitals do not call FEMA directly.
A medical device problem: report it to the manufacturer and to FDA MedWatch.
Outside the US: your national emergency number and national cyber agency (in the UK, NCSC).

From Health 201FailSystems