The Signal and the Noise: Why So Many Predictions Fail--but Some Don't

We spend a hug amount of money and computational power on predicting weather, yet we still complain about inaccurate weather forecasts. In spite of this, weather forecasting is one of the "success stories". We have much more accurate forecasts than we did prior to the computational advances. However, beyond a week, weather forecasts fare no better than guesswork. Small changes in variables can significantly alter the long term prognosis. People can also cause kinks in the operations. Commercial weather forecasts have a tendency to overestimate precipitation. (They would rather have someone pleasantly surprised by a sunny day than have an activity ruined by rain.) There was also a case of flooding in North Dakota where an accurate river level prediction was made. However, only the average number was shared rather than the range. The river crested within the range, which happened to be just above the flooding level (and above the average level predicted.)
Baseball provides a rich source of data about many players. It also provides many opportunities for inaccurate predictions about players. Successful teams use a mixture of scouting and statistical analysis to find the best players for the money.
There are a number of biases in the data analysis and predictions we see. Bold predictions are most likely to get press coverage, but are least likely to be right. It is almost always possible to find a significant pattern in the "noise", but that doesn't do much good for predicting future signals. People also tend to be really bad at understanding what data means. Furthermore, news coverage tends to focus on the outliers, even though they tend to be the most inaccurate.
Coverage of global warming provides a cautionary example. There is scientific consensus on the negative impact of human activities on the earth's climate. However, consensus does not necessarily mean good science. Rather than being a balanced average of different opinions, a consensus tends to be dominated by the loudest or most forceful voice. For climate change, the initial view was simply that a greenhouse effect existed and that human activity contributed to increasing in gases. After this point, things got wonky. Discussion switched to global warming, with precise numbers given for warming predictions. When these tended to overstate the warming, the predictions were revised down and models were calibrated. However, the more extreme predictions were the ones that received more press coverage. This would distort the public's view of the situation and give greater credence to the opponents. The response to global warming involves politics, and politics is concerned with the short term impacts, not the long term results. Thus, the noise ends up being twisted towards short term purpose, while the signal is left in the scientific circles.
Predicting terrorism is a lot like predicting earthquakes. We know it is likely to happen, with the minor activities being more frequent than the high-body-count ones. However, we are not good at knowing the specifics. The September 11, 2001 attacks were "unknown unknowns". They were just not something we expected or thought to expect. This made prediction difficult. Terrorism does tend to follow a power-law distribution giving us an idea that a terror attack might be due, but no more than that. (Ironically, Israel seems to buck the power law trend. They permit small-scale attacks to happen, but focus efforts on limiting the more damaging large scale attacks.)
The real problem with predictions is people. The sensational tends to be more appealing than well-thought out. A single pronouncement is given more weight than one couched in uncertainty - even though the uncertain one is much more truthful. People also tend to value "loyalty", giving more credit to those that stick by their guns, even though a willingness to change predictions in face of data makes for more valuable predictions. What are we to do?