“Models are opinions embedded in mathematics.”
“Many forms of WMD were deployed as an answer to institutional bias — in criminal sentencing, in school grading, in university admissions, in hiring and lending. The models are supposed to be race- and gender-blind, blind to privilege and connections.
But all too often, the models are trained with the biased data. The picture of a future successful Ivy League student or loan repayer is painted using data-points from the admittedly biased history of the institutions. All the Harvard grads or dutiful mortgage payers are fed to the algorithm, which dutifully predicts that tomorrow’s Harvard alums and prime loan recipients will look just like yesterday’s — but now the bias gets the credibility of seeming objectivity.”
“WMDs turn the whole of human life into a game of Search Engine Optimization. With SEO, merchants hire companies who claim to have reverse-engineered Google’s opaque model and whose advice will move your URL further up in its ranking.
When you pay someone thousands of dollars to prep your kid for the SATs, or to improve your ranking with the “e-score” providers that determine your creditworthiness, jobworthiness, or mortgageworthiness, you’re recreating SEO, but for everything. It’s a grim picture of the future: WMD makers and SEO experts locked in an endless arms-race to tweak their models to game one another, and all the rest of us being subjected to automated caprice or paying ransom to escape it (for now). In that future, we’re all the product, not the customer (much less the citizen).”