The Efficiency Paradox: What Big Data Can't Do

As a society, we have become more and more efficient. But has that made life better? In many cases no. In some cases it may make us worse - especially as money is drained on minute efficiencies instead of used to make real improvements. 

The book abounds in examples of "efficiency gone wrong". The financial industry takes the crown for the worst offender. Trading has become ultra-efficient, with dedicated lines helping to produce millisecond improvements in executions. The result is more and more rapid trading among intermediaries. There is also more derivative trading. While some of the improved efficiency helps everyday investors, most of the effort does not.

There are some interesting quotes from James Evans regarding the impacts of "efficiency" in science. A "popularity" contest has evolved in science as well as other industries. People can easily find the most cited articles, and then cite them in their work. Instead of a long-tail enabled by easy access, we get more concentration to those that are already dominant.

Efficiency also brings about more challenges. Travel used to be very opaque with complex, hard to find pricing. As online travel agencies came about, more of the information was available to consumers. As they found and exported  inefficiencies in the  pricing structures, the travel companies made updates to make things more "efficient". Now it required more work to find the best deals. Meta travel search sites came out to help find better deals. Even travel agents themselves have been having a comeback. The old inefficiencies have simply been replaced with new ones. 

Are we better off with changes? We can access the internet at any time on our phones. This helps improve some actions that were difficult earlier. But, it also has resulted in a bunch of time wasting. Does it provide a net benefit or not? A computer can more efficiently fly a plane than a human, but a human still needs to take over during catastrophes.  If humans rely too much on autopilot, they may not have the skills when they are needed the most.

The book is aptly titled. Efficiency is a paradox. Making parts more efficient can sometimes make the sum of the parts less efficient. Finding the best way forward for society is a huge challenge.