Published on: November 12, 2026 at 9:41 am
The stakes riding on whether or not AI gets it right are often high. In fact, when AI is trained on bad data or makes an error, sometimes people suffer.
Academy of Management Scholar Christine Moser of the Vrije Universiteit Amsterdam cited the example of the Dutch childcare benefits scandal known as “toeslagenaffaire,” which involved the Netherlands’s tax authority wrongly accusing about 26,000 parents of fraud between 2005 and 2019, using discriminatory algorithms and ethnic profiling. The agency forced innocent families to repay large sums of money, causing severe financial hardship and the separation of more than 1,000 children from their families.
“There’s a more philosophical question here that goes back to the example that the tax authority in the Netherlands incorrectly identified thousands of individuals saying that they were fraudulently using childcare benefits, even though they were actually eligible,” Moser said. “And so it had a really negative impact on a lot of families who experienced severe financial consequences due to this fact that the error made by this AI algorithm wasn’t double-checked and was just trusted by the tax authority that it was correct.
“It went beyond financial consequences—people had to sell their houses; the worst cases are suicides, but also children placed in foster care. And once they’re in this foster-care system, it’s super hard to get them out, because the family has been tainted by that which they haven’t even caused themselves,” she said. “That’s another example of an algorithm that was designed to make life for the tax authorities more efficient and fairer as well, because they said, ‘Oh, let’s just figure out those who commit fraud to protect the bigger group of people who do not commit fraud,’ but then, in the way that the algorithm was designed, and also in the way that the data were selected and then curated for the algorithm to work on this issue, the algorithm spit out that output because that’s what it’s designed to do.
“It will give you output if you ask it to.”
It was alarming that no one at the organization bothered to check the accuracy of the AI outputs.
“Indeed, there was no human oversight, no judgment to actually look into what’s going on here—‘Is this actually correct?’” Moser said. “Then it went into this bureaucratic machine preventing people from correcting these errors until it was way too late.
“The Dutch government actually fell from power due to this scandal, which doesn’t say much, because now we have another government that fell due to something else,” she said.
Moser noted similar cases in other countries, including a cautionary tale in Australia. The Australian Robodebt scheme used a faulty algorithm to automatically accuse more than 400,000 welfare recipients of debt between 2016 and 2019, causing immense distress and three known suicides.
“This was a similar scheme where the government was supposed to give social benefits to people but used another algorithm to detect fraud, and this also went wrong,” Moser said.
In an incident called the Horizon IT scandal, the British Post Office pursued thousands of innocent sub-postmasters for financial shortfalls that were actually caused by faults in a Fujitsu accounting software system.
“Another one people might be familiar with is the UK Post Office scandal, where franchise-holders of post office subsidiaries were wrongly accused of fraud, of defrauding the system, leading to court cases there as well,” Moser said. “It took years for the authorities to, first of all, see their mistake, and now they’re still working to correct it.
“Here in the Netherlands, the mistakes of childcare benefits scandal still have not been corrected,” she said. “It’s been more than a decade now that these families have been suffering, and it’s tens of thousands of people who have been affected here.
“It’s pretty insane.”
