A Risk Score on the Hotline
In one Pennsylvania county, every call to the child protection hotline gets a risk score built from public records. Researchers found racial disparities in its recommendations, and parents with disabilities say it flagged them.
When a system learns from public records, are the families who rely on public services the ones it sees most?
What happened
In Allegheny County, Pennsylvania, calls to the child protection hotline are given a risk score from 1 to 20 by the Allegheny Family Screening Tool. The score is built from county records such as Medicaid, mental health, jail, and probation data. Screeners use it when deciding whether to open an investigation. The tool advises; a person decides.
In April 2022 the Associated Press reported research from Carnegie Mellon University. Between August 2016 and May 2018, the tool’s scores would have sent 32.5 percent of Black children reported for neglect to a mandatory investigation, against 20.8 percent of white children. Screeners disagreed with the score about a third of the time, and the researchers found that on its own the tool would have been more racially disparate than the screeners were. The AP also reported that a technical glitch had shown screeners wrong scores for more than two years.
Who spoke up
Families and their advocates. In October 2022 an attorney, Robin Frank, filed a complaint on behalf of a father with an intellectual disability. In 2023 Andrew and Lauren Hackney, parents with developmental disabilities, told the AP that they lost custody of their infant daughter after taking her to hospital because she was not feeding well, and that they believed the tool had flagged them. The county confirmed the tool had used disability-related data. Families cannot see their scores, and the AP could not confirm whether a score affected the Hackneys.
What changed
The county called the 2022 research hypothetical, stressed that workers can override the tool, and acknowledged that the tool had not reduced racial disparities. In January 2023 the AP reported, citing anonymous sources, that lawyers in the US Justice Department’s Civil Rights Division were examining whether the tool discriminated against parents with disabilities. No outcome of that examination has been made public.
What to keep straight
- The tool does not make the decision. Screeners do, and the research found that they reduced the tool’s racial disparity rather than adding to it.
- The Justice Department’s interest was reported through anonymous sources. No findings have been published.
Talk about it
- Records like Medicaid or jail data exist mostly for families who use public services. What does that mean for who gets scored, and how?
- If screeners already disagreed with the score a third of the time, what is the score adding?
- Families cannot see their own score. Should they be able to?
Go further
Is AI Disciplining Society?
Examining how algorithmic systems discipline society—training populations to conform, punishing deviation, and reshaping behavior without consent.
Read the article Teach it Classroom Kits · Ages 10–14The Fairness Menu
Two groups design a fair rule for the same everyday decision. Both are reasonable. Both disagree — a first encounter with real fairness tradeoffs.
View the kitSources
- How an algorithm that screens for child neglect could harden racial disparities, Associated Press, via PBS NewsHour (29 April 2022).
- Imagining new futures beyond predictive systems in child welfare (Stapleton, Cheng and others), arXiv:2204.13872 (2022).
- AP report: DOJ examining AI screening tool used by Pa. child welfare agency, Associated Press, via PBS NewsHour (31 January 2023).
- Not magic: Opaque AI tool may flag parents with disabilities, Associated Press, via The Daily Record (15 March 2023).
More accounts to read
All Community Voices-
Families
The Benefits Scandal That Brought Down a Government
A Dutch risk model that weighed citizenship helped wrongly brand tens of thousands of parents as fraudsters. Parents pushed back, and the government resigned.
2021 · Netherlands Read the account -
Families
Debts by Averaging
Australia averaged people's yearly income to decide who had been overpaid welfare. People with irregular pay were chased for debts they did not owe. A Royal Commission called it neither fair nor legal.
2015–2019 · Australia Read the account -
Students
The Summer an Algorithm Set the Grades
With exams cancelled, a model lowered 39 percent of teacher-assessed A-level grades, and small classes fared best. Students marched; the government reversed within days.
2020 · England Read the account
Summarised from the public record. Follow the sources for the full story.