Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Big data has the potential to erase past prejudices and provide an equal playing field for everybody. Rather than rely on biased feelings, computers can look just at the numbers to make objective decisions. This could allow previously redlined groups better access to loans and money. It could help with hiring and termination of employees. It could also better target resources to improve education and reduce crime. Big data could also help companies make a lot of money.

Alas, in the real world things have not quite turned out that way. The one area where big data has had the most success is in earning companies a lot of money. However, this has been fraught with significant negativity.  At its worst, big data may make totally erroneous predictions based on limited or erroneous data. It can be very difficult to appeal these decisions. At other times, the models serve to ingrain past prejudices in seemingly objective ways.

A good big data model needs a continuous feedback loop to improve predictions and improve the results. It relies on good data. It also must have data that is meaningful and well collected. A grading of teachers by improved student test scores may seem objective, but it is easily manipulated. If there was cheating the previous year, the improvement would be lower. The single data point could also be subject to isolated factors (such as sick students on testing day.)

Bucketing people into groups may make analysis more simple, but it could also produce misleading results for individuals. Using proxies can also have problems. Some data may be easier to collect than what is attempted to be measured, but does not provide the same result.

Data also has the challenge that it can lead to unintended changes in behavior. This is especially problematic when proxies are used. Instead of showing an improvement in the desired behavior, the model encourages changes in the proxy. Data may also be fudged or the model tweaked to get a desired result. The "objective" analysis is merely an insulating layer between subjectiveness.

Big data is a tool. It can help humans uncover hidden gems. However, it can also lead to bad subjectivity in the name of objectivity. It is important to understand the details before relying on data to make decisions.