Meltdown: Why Our Systems Fail and What We Can Do About It

We have a lot of very complex system in the world today. We also want things to be as efficient as possible. This can make them more susceptible to failure. In one simple case, older airplanes have joysticks for the pilot and copilot. If one is moved, the other is moved in the same direction. This redundancy allows erroneous maneuvers to be quickly caught and corrected. Modern fly-by-wire aircraft do not have this. This has resulted in some crashes where one pilot did something improper and the other did not notice in time to correct it.

Sometimes, we try to engineer ourselves out of possible errors. However, this can produce errors that may be harder to catch. As an example, a hospital set up an automated computer system to dispense medication. The doctors accidentally proscribed a huge dose. (They entered the total desired amount in the "amount per pound" section.) Most people just assumed the system was doing what it should and blindly followed it. 

If there are too many safeguard, then people get in the habit of blindly going through them, making it harder to catch errors when they do occur. We do need to have the proper amount of safeguards structured in a way that we can really follow them. It is also important to account for unexpected results. A system that automatically adjusts based on sensors will have issues if the sensor returns erroneous data. More automatication can be useful, but we still need people to make sure things don't get too crazy.