In January 2020, Detroit police arrested Robert Williams at his home in Farmington Hills, Michigan, in front of his wife and two young daughters. A facial recognition search had wrongly matched an old driver’s licence photo of him to surveillance footage of a 2018 theft from a Detroit shop. He was held for nearly 30 hours.
Robert Williams and his family. With the American Civil Liberties Union he sued the city of Detroit in 2021, and that July he gave testimony to a subcommittee of the US House Judiciary Committee about the arrest and what it did to his family. He has continued to speak publicly about it.
In June 2024 the case settled, with binding rules for Detroit police. They may not arrest anyone based only on a facial recognition result, or on a photo lineup run straight after one. They may not put a facial recognition lead in a lineup without independent, reliable evidence. Officers must be trained on the technology’s risks, including its higher rate of misidentifying people of colour, and the department must audit cases going back to 2017. The court keeps jurisdiction for four years to enforce the agreement. The AP reported that the city would also pay him $300,000.
In Allegheny County, Pennsylvania, calls to the child protection hotline are given a risk score from 1 to 20 by the Allegheny Family Screening Tool. The score is built from county records such as Medicaid, mental health, jail, and probation data. Screeners use it when deciding whether to open an investigation. The tool advises; a person decides.
In April 2022 the Associated Press reported research from Carnegie Mellon University. Between August 2016 and May 2018, the tool’s scores would have sent 32.5 percent of Black children reported for neglect to a mandatory investigation, against 20.8 percent of white children. Screeners disagreed with the score about a third of the time, and the researchers found that on its own the tool would have been more racially disparate than the screeners were. The AP also reported that a technical glitch had shown screeners wrong scores for more than two years.
Families and their advocates. In October 2022 an attorney, Robin Frank, filed a complaint on behalf of a father with an intellectual disability. In 2023 Andrew and Lauren Hackney, parents with developmental disabilities, told the AP that they lost custody of their infant daughter after taking her to hospital because she was not feeding well, and that they believed the tool had flagged them. The county confirmed the tool had used disability-related data. Families cannot see their scores, and the AP could not confirm whether a score affected the Hackneys.
The county called the 2022 research hypothetical, stressed that workers can override the tool, and acknowledged that the tool had not reduced racial disparities. In January 2023 the AP reported, citing anonymous sources, that lawyers in the US Justice Department’s Civil Rights Division were examining whether the tool discriminated against parents with disabilities. No outcome of that examination has been made public.
The Houston Independent School District rated teachers partly with EVAAS, a value-added model sold by the software company SAS. A value-added model estimates how much a teacher changed their students’ test scores. Low scores were used in decisions about teachers’ jobs: the court record lists 12 union members whose contracts ended over low value-added scores between 2012 and 2014.
SAS treated the model’s code as a trade secret. Neither the teachers nor the district could see it. The district admitted it did not verify or audit the scores, and an expert for the teachers could not reproduce them.
Teachers. In 2014 the Houston Federation of Teachers and individual teachers sued the district in federal court. One of them, Daniel Santos, a sixth-grade teacher at a Houston middle school who had been rated ineffective, spoke publicly about his commitment to his students.
In May 2017, US Magistrate Judge Stephen Wm. Smith refused to dismiss the teachers’ claim that their right to due process had been violated. Teachers, he found, had no meaningful way to make sure their scores had been calculated correctly, which left them open to losing their jobs by mistake. He dismissed the teachers’ other claims.
In October 2017 the district settled. It agreed not to use value-added scores to terminate a teacher as long as the teacher cannot independently check or challenge the score, to set up a consultation panel on teacher appraisal, and to pay $237,000 in legal fees. It had already ended its contract with SAS in 2016.
From about 2015 to 2019, the Australian government raised welfare debts automatically using a method called income averaging. It took a person’s yearly income from tax office records, spread it evenly across the year, and compared that average with the income the person had reported while receiving benefits. Anyone whose pay was irregular, such as casual workers, students, or people moving in and out of work, could look as if they had been overpaid when they had not been. Debt notices went out to hundreds of thousands of people.
People who received the debts, and their families, pushed back through a public campaign under the name #notmydebt, through test cases, and through a class action. In 2019 a test case brought by Deanna Amato led the government to concede that debts raised this way were unlawful. At the Royal Commission that followed, people described what the debts had done to them, including a mother whose son died after being told he owed money.
In June 2021 the Federal Court approved a settlement of the class action, covering about 648,000 people: A$112 million in compensation including legal costs, about A$1.76 billion in debts withdrawn, and about A$751 million in refunds. The judge described it as a shameful chapter in the administration of social security.
In July 2023 the Royal Commission into the Robodebt Scheme, led by Catherine Holmes, delivered its report. It called the scheme “a crude and cruel mechanism, neither fair nor legal.” A second settlement, with A$475 million in compensation, was approved by the Federal Court in June 2026.
During the pandemic, Vrije Universiteit Amsterdam (VU) ran online exams with Proctorio, software that watches students through their webcam and checks that a face stays in view. Robin Pocornie, a Black master’s student, said it repeatedly failed to detect her, with messages such as “face not found” and “room too dark.” She said she had to shine a lamp on her face to sit her exams, and that her white classmates did not.
She did, with support from the Racism and Technology Center. She filed a complaint with the Netherlands Institute for Human Rights, arguing that the software discriminated on grounds of race, and described her experience in interviews.
In December 2022 the Institute issued an interim ruling: she had put forward enough facts for a presumption of indirect race discrimination. That shifted the burden to the university, which was given ten weeks to back up its denial with data the Institute could verify.
In October 2023 the final ruling found that VU had not made a prohibited distinction by using Proctorio. The university’s log data showed her problems were no greater than other students’, and the Institute attributed them to her glasses and an unstable internet connection. It also found no prohibited distinction in VU’s failure to have the software checked independently before using it. But it ruled that VU had handled her complaint too slowly, and so not carefully enough, and that this was a prohibited distinction on grounds of race. VU apologised and said it would review its complaints procedure and re-evaluate remote testing.
In May 2023 an instructor at Texas A&M University–Commerce pasted his students’ final essays into ChatGPT and asked whether it had written them. When it said yes, students in the class were given a temporary incomplete grade. A chatbot has no reliable way to recognise its own writing, so its answer was not evidence of anything.
Purpose-built detectors have a different problem. Researchers at Stanford ran seven of them on 91 essays written by non-native English speakers for the TOEFL test and 88 essays by US eighth graders. The eighth graders’ essays were classified almost perfectly. The TOEFL essays were wrongly labelled AI-generated 61.22 percent of the time on average. All seven detectors flagged 18 of them, and at least one detector flagged 89 of the 91. When the researchers had ChatGPT make the TOEFL essays’ word choice more elaborate, the false-positive rate fell to 11.77 percent: plainer vocabulary was being read as a sign of a machine.
Students in the Texas class disputed the accusation publicly, and the story spread from a post on Reddit before the university responded. The Stanford team published its findings in the journal Patterns and cautioned against using such detectors to judge students’ work, particularly the work of non-native English speakers.
The university said several students were cleared and had their grades issued, one admitted using ChatGPT in the course, and no student failed or was barred from graduating. It said it was developing policies on AI in the classroom and adopting detection tools of its own, which is exactly where the Stanford findings matter.
Sheri Lederman taught fourth grade in Great Neck, New York. Under the state’s teacher evaluation system, part of each teacher’s rating was a growth score calculated from their students’ test results. For 2012–13 her growth score was 14 out of 20, which counted as effective. For 2013–14 it was 1 out of 20, which counted as ineffective.
She did. She challenged the 2013–14 score in court, represented by her husband, Bruce Lederman, a lawyer.
In May 2016, Acting Justice Roger McDonough of the New York Supreme Court in Albany County found that her 2013–14 growth score was arbitrary and capricious, and set it aside.
Since 2012, Wisconsin’s Department of Public Instruction (DPI) has run the Dropout Early Warning System, known as DEWS. It scores students in grades 6 to 9 on how likely they are to graduate on time, and its inputs included race, gender, disability, and family income. In 2020–21 it labelled more than 32,000 students, about 15 percent, as high risk.
In April 2023 The Markup and Chalkbeat reported what DPI’s own 2021 analyses had found. When DEWS predicted that a student would not graduate on time, it was wrong nearly three quarters of the time. Its false alarms were 42 percentage points more frequent for Black students and 18 points more frequent for Hispanic students than for white students. In Kenosha in 2018, 43 percent of Black ninth graders were labelled high risk, against 11 percent of white ninth graders.
Students. Reporters interviewed students who had carried the label, and in December 2023 two Black students at Cudahy High School, Mia Townsend and Maurice Newton, wrote a first-person piece about it. They wrote that they were surprised a state tool used race, that labels like this can discourage students, and that their own lives did not match what the system predicted. DPI at first defended the system; a spokesperson told reporters, “It’s math that reflects our systems.”
In September 2023 DPI said it was studying its early warning systems and considering changes. In October 2023 it removed DEWS data from the dashboards schools use.
To decide which childcare-benefit claims to check for fraud, the Dutch tax authority used a risk-classification model. According to Amnesty International, one of its inputs was whether the applicant held Dutch citizenship, so non-Dutch nationals received higher risk scores. Tens of thousands of parents were wrongly treated as fraudsters and told to repay benefits, with severe consequences for their finances and their families. By February 2025 the Dutch government had recognised more than 40,300 parents as affected.
Parents. Those who had been wrongly accused pushed for recognition and compensation. On the day the government resigned, one mother told reporters outside parliament that the government had to take responsibility.
In December 2020 a parliamentary inquiry published its report, titled Ongekend onrecht (“Unprecedented injustice”), finding that fundamental principles of the rule of law had been violated. On 15 January 2021 the cabinet of Prime Minister Mark Rutte resigned.
In December 2021 the Dutch Data Protection Authority fined the tax administration €2.75 million for processing applicants’ dual nationality in ways it found unlawful, discriminatory, and improper. That data should have been deleted in 2014; in May 2018, 1.4 million people were still registered. A compensation and recovery scheme for affected parents followed.
In spring 2020 the pandemic closed schools, and England cancelled its exams. Teachers estimated a grade for each student and ranked the students in each subject. The exams regulator, Ofqual, then ran those estimates through a statistical model. For each school and subject, the model predicted a spread of grades, based mainly on the school’s results in earlier years and adjusted for this year’s students’ prior attainment. It then fitted the ranked students into that spread.
When results came out in August, 39.1 percent of the 718,276 teacher-assessed A-level grades had been lowered: 35.6 percent by one grade, 3.3 percent by two, and 0.2 percent by three or more.
Statistics are unreliable for small groups, so the model trusted teachers more where a subject had few entries: fully below an entry size of 5, and partly between 5 and 15. Small entries are more common in independent schools. In Ofqual’s own figures, the share of A grades and above rose by 4.7 percentage points at independent schools compared with 2019, against 2.0 at comprehensives and 0.3 at sixth-form and further education colleges.
Students. Some lost university places they had been offered. On 16 August around 300 students marched from Parliament Square to the Department for Education and Downing Street, saying the model penalised students from disadvantaged backgrounds.
Scotland moved first. On 11 August its government directed the exams body to reissue downgraded awards based on teachers’ judgements, after about 76,000 candidates had at least one grade lowered. On 17 August Ofqual’s chair announced that students in England would receive their teacher-assessed grade, or the calculated grade if it was higher, and apologised for the distress the process had caused. Wales and Northern Ireland made the same switch. GCSE results, due three days later, were never issued under the model.