Crash: how computers are setting us up for disaster

“Automation will routinely tidy up ordinary messes, but occasionally create an extraordinary mess.”

[…]

“We are now on more lists than ever before, and computers have turned filing cabinets full of paper into instantly searchable, instantly actionable banks of data. Increasingly, computers are managing these databases, with no need for humans to get involved or even to understand what is happening. And the computers are often unaccountable: an algorithm that rates teachers and schools, Uber drivers or businesses on Google’s search, will typically be commercially confidential. Whatever errors or preconceptions have been programmed into the algorithm from the start, it is safe from scrutiny: those errors and preconceptions will be hard to challenge.”

[…]

“By forcing drivers to confront the possibility of small errors, the chance of them making larger ones was greatly reduced.”

(source accessed 25.10.2016)