Since 1999, the broken algorithms in Fujitsu's Horizon have caused at least 700 UK Post Office employees to be prosecuted for fraud under highly dubious reasons.
This is the most widespread, known, miscarriage of justice in the judicial history of the UK.
It has taken twenty years for a full investigation, and to win the first full vindications of innocence. Meanwhile, lives have been destroyed, marriages broken down, reputations ruined, people rotting in prison for years – all because of unaccountable algorithmic systems.
We live in a world today where it's possible to be economically disenfranchised at will. Someone can put a name on a blacklist that matches ours, or a face that some algorithm thinks looks like us, and our lives can be destroyed, without any way to understand what has gone wrong, or why.
Insidious techniques such as shadow banning may even hide from us that anything is wrong at all.
We may have all of our Google, Microsoft, Apple or Twitter accounts removed without any accountability or transparency, cutting us off from even cellphone ecosystems. Our bank accounts may be shut down for political reasons, or our resume may be secretly excluded for arbitrary uncommunicated reasons.
Every one of us is potentially another weeping postmaster.
This is why the transparency of all algorithmic systems is so crucial (not just advanced AI).
Similarly, we must mandate the transparency of the organizations behind those systems, as such issues do not occur in an isolated vacuum. This is why aircraft carry both flight data recorders, AND cockpit voice recorders.