When Should You Believe Forecast?

I never believe story tellers (mostly macro economists and strategists) as they try forecasting the future. Their stories are often interesting and can sometimes offer valid perspectives on how current events can unfold and influence the markets and the economy. But the probability that one given economist is right about the future is always minimal even if he or she is only forecasting the short-term future.

That nobody can predict the future should be obvious. In the real world, there are too many variables to account for and too many unpredictable events that can arise at any given time to influence the course of events to realistically rely on anybody or any organization for that matter to be making accurate forecast. Moreover, it is most often impossible to determine with any precision which of these variables should have the most weight in any forecast even using the most sophisticated techniques. Finally, assuming honesty (a big assumption), who are we to believe? When randomness is so high, it is very difficult to attribute any credibility to even the economists with the best track records!

However, there are situations where it is worth trying to make sense of upcoming events, to try to make a forecast or, at least, reduce the uncertainty of what is lying ahead. For instance, I can predict where and when a ball will fall given the speed and angle at which it was thrown. The wind is likely to influence this forecast but I could attempt to measure the speed and direction of the wind to take it into account in my forecast. That forecast should be better than a random number picked out of a hat. Other factors may also influence the trajectory of the ball rendering the forecast wrong (e.g., a bird may pass by and collide with the ball). Some of these factors may only have a minor significance and may safely be ignored. Others may have a significant influence on the ball trajectory but only a low probability of occurrence (such as the bird colliding with the ball). Ignoring these factors will lead to frequent accurate forecasts and to completely inaccurate ones some of the time.

Betting the house on the frequently accurate forecast could lead to ruin however. Yet, it would even more foolish to make even a weak bet against this forecast as it's likely to be correct most of the time as the probability that a bird will collide with a thrown ball is obviously very low. The totally safe bet, however, is to bet on the "no collision" forecast and take insurance against the collision event; if such insurance exists and is "reasonably" priced. Managing the risk by checking for birds before throwing the ball could reduce the risk of being wrong and might reduce the cost of insurance but it won't eliminate the risk completely. The only way to completely eliminate the collision risk is to take insurance against the event if such insurance exists; and then again, the provider of insurance has to be credit worthy when the event occurs …

Moreover, I am assuming here that there are no black swans (i.e. there are no unpredictable events not covered by the insurance that could occur and deflect the ball). As described by Nicolas Taleb, the ignorance of black swans most often fools people into denying existing but unknown risks. The insight form Taleb is that unforeseen events, such as black swans, can make you lose lots of money on bets that appear to be fairly priced and completely safe. History has shown that the incidence and consequences of such events are most often ignored, underestimated or mispriced by market participants. Thus, they often lead to disastrous outcomes for which market participants were unprepared when they unfold.

Obviously, the assumption of that no Black Swan exist is a strong assumption in many economic situations. In such situations, events are generally not completely riskless even when one is insured against all known risks. But I make the assumption of no Black Swan in this essay as it is not even necessary to show that forecasting exercises can often lead humans down the fool's path.

Let's come back to our ball example and the non-zero probability that a known possible event; such as a thrown ball colliding with a bird randomly passing by. In real life (i.e. in economic situations), such events will often have a "price". Depending on the price of the risk, some people will decide that it is worthwhile taking the insurance against the probability that the ball will collide with the bird while others will prefer to provide the insurance. This preference will depend on the relative risk aversion of the parties and the relative costs of getting the forecast wrong as well as other factors. Obviously, the more risk averse individuals and those most negatively affected by the event will tend to express an interest in buying the insurance. If there are also people able to pool the risk and charge insurance buyers more than the costs of that risk, there will also be an interest in selling insurance. We will thus have a "market" for the risk. The strategy that you would yourself adopt would then depend on your own situation and characteristics as well as the market price of the risk.

This is when things get interesting. We are now coming to the good part.

Two features of the event "throwing a ball" described above will help the forecaster. The first one is the fact that the number of variables influencing the trajectory of the ball is limited and the laws governing the ball's motion are known with great precision; which is rarely the case in economics. Things are more complex in real life and the laws governing history are only the results of the imagination of our historians at best. The "bird collision" sub-event is one such complexity: a known event but with very little predictability. The bird collision event introduces an element of randomness which, although we might know about the potential occurrence of the event and even its probability, still carries much unknown elements (timing, speed, angle, size of bird) and thus unknown consequences on what we are trying to predict (the timing and location of the fallen ball). In other words, although we may know about the possibility of a bird collision, its effects on the trajectory of the ball remains unknown. This implies that, even if it is possible to insure oneself against the occurrence of the event, it's impossible to predict where the ball is going to fall when hit by the bird. Yet, many forecasters will ignore this complex reality and unreliably predict an outcome. You know that they are not to be believed but may have a hard time resisting the temptation to consider the forecast a valid one. But this is not even the most interesting feature of the event.

The second and most relevant factor here is the fact that the accuracy of the "regular" forecast (the one conditional on no collision) depends largely on the premise that the ball has already been thrown (i.e. speed, time and angle are known) and the universe in which this happened (winds and birds) can be assessed at least probabilistically. In real life, however, this is most often totally unrealistic. That is, even if we knew in details the laws of physics governing the thrown ball, assuming that we know the circumstances and timing of the event is a mistake. It’s an event bigger mistake to try to model this behavior as the intent of the ball thrower are not governed by the laws of physics but rather by a complex human chemistry which escapes mathematical modelisation; at least with the current state of knowledge. Too often economics focuses on what happens after the ball is thrown forgetting that the most complex part of the event is what happened before (motivations, incentives and intents). These being more difficult to control, monitor, measure and model, they tend to be ignored. Yet, what is the good sense of predicting when the ball is going to fall when you don’t know when it is going to be thrown.

Thus, based on the potential of predicting where a thrown ball could fall in a controlled experiment, we often end up with a black box where the mechanics of the ball are modeled perfectly but where everything else is basically ignored or brushed aside; the winds and the bird dynamics as well as, more subtly, the intent and the timing of the ball thrower. As intents often constitute the heart of economic decisions, predicting the future without taking them into account renders the forecast meaningless. But the magic of the black box of a forecasting model often obscures the fact that forecasts are being made using unreasonable assumption. In other words, even if we know everything perfectly about the process of throwing a ball, if in reality we have absolutely no idea about how to model, and especially how to quantify, motives and intent, a forecast is totally useless. But forecasters, like magicians, make money by selling illusions. If they can make you forget about the wrong assumptions of their model, you may just believe them.

John Maynard Keynes had already warned R.F. Harrod in a letter written in 1938 of the dangers of relying on models and making analogy with the physical science to explain economic phenomenon:

“The point needs emphasising because the art of thinking in terms of models is a difficult--largely because it is an unaccustomed--practice. The pseudo-analogy with the physical sciences leads directly counter to the habit of mind which is most important for an economist proper to acquire.”

“I also want to emphasise strongly the point about economics being a moral science. I mentioned before that it deals with introspection and with values. I might have added that it deals with motives, expectations, psychological uncertainties. One has to be constantly on guard against treating the material as constant and homogeneous in the same way that the material of the other sciences, in spite of its complexity, is constant and homogeneous. It is as though the fall of the apple to the ground depended on the apple's motives, on whether it is worth while falling to the ground, and whether the ground wanted the apple to fall, and on mistaken calculations on the part of the apple as to how far it was from the centre of the earth.”

So what is one to do? Stop attempting to forecast the future? There is not a chance in the world that this warning could be heeded as it is in the human genes since the early ages to attempt to read the tea leaves. The answer is to try distinguishing the consequences of events that have already been set in motion from the rest of events which heavily depend on randomness, human motives and have yet to occur.

For instance, the recent rapid growth of emerging economies has set in motion a growing demand for resources which, given the short term inertia of economic growth, has some predictability for commodity prices. It is impossible to know for sure what will happen to the prices of resources but it is correct, from the moment that we understand the dynamics of emerging country growth, to simply predict that the increased demand for resources will put an upward pressure on their prices. This forecast cannot be as precise as in the case of the thrown ball experiment as timing and magnitude cannot be modeled but the direction can be sufficient to make a sound (yet still risky) investment decisions if prices have not increased yet.

It is also correct to predict that higher prices will provide incentives to grow the supply of these resources and to look for alternatives as well as predicting that these efforts could eventually reduce the pressure on the price of such resources. An attempt at quantifying the price increase in the short run, while the supply is still held constant and alternatives are not yet available, could be made using historical data and yields some credible results (very likely better than a random walk forecast) but the results from such historical models should not be confused with the certainty stemming out of the laws of physics governing the motion of a ball as to when and where the ball (or the apple) is going to fall once it has already been thrown; and then again, Newtonian mechanic was shown to be only an approximation of reality by quantum mechanics. Moreover, to then turn around and try to predict the timing of the reversal of commodity prices following an eventual supply response to their short-term price increase (when the response has not yet occurred) could be interesting but certainly not believable. Any attempt at modeling this process would be pure speculation; and most certainly, the forecasts from such a model would not be sufficient to make a sound investment decision.

The only quasi-certainty which we have here is that, by virtue of the universal law of demand, prices of resources should increase as demand for resources increases; everything else held constant. However, no one has come up with any theory or concept to quantify the price increases, nor their timing, as in the case of the ball. Thus, a forecast about a price increase will be credible; one about the magnitude of the price increase and its timing should be wrapped in warnings and handled with care. Any attempt to extent the horizon of the forecast, based on a supply response from the market which has yet to occur, which would involve modeling complex human motives and interactions and the timing of scientific discoveries that would offer alternatives, is not credible. In short, all longer term forecasts about events that have not been set in motion are speculative in nature and any suggestion that a model can capture more than the eye can see is alchemy and fraud. They should thus be automatically treated with the highest suspicion and essentially be considered as pure speculation from a good story teller.

Also, remember that in the ball example, other factors could affect the ball trajectory (winds, birds, etc.). The same would also apply here if we attempted to forecast prices. But it is worth stressing that these “other factors” are not the only risk involved in the price forecasting exercise; yet, that is a reality which is often neglected by forecasters. This is because the inferred historical relationship that allows quantifying the dynamics between quantities and prices is not a law of physics and most often does not hold. Unlike in the case of the ball, where the laws of physics are pretty much set in stone, in economics, a historical average does not carry any scientific weight and may, in fact, never have occurred. Like with any average (that of two individuals of 20 and 60 years old respectively for instance), the statistics alone does not give us any idea of the dispersion of the variables in the sample or population. Although, it is possible to get an estimate of this dispersion in many cases, it remains an estimate and an estimate made under several assumptions (such as the normality of the distribution of the random process governing the variable). Tell me who is to verify these assumptions? And under which assumptions will these assumptions be verified? Good story tellers usually don’t bother with such details.

As you can see, both the quality and the scope of the forecasts get degraded as we go from the thrown ball (for which we can predict both the position and time of the ball with some precision) to the short term forecast of the prices after an increase in demand (we only get the direction of the move), to the yet un-thrown ball (randomness that may be confined to a limited set of possibilities) and, finally, to a long term forecast of the price of resources after the supply and the technology have had time to adjust to the incentives provided by higher prices (total randomness).

What matters here is the stability of relationships: Newtonian dynamics are very stable (as long as we stay on earth) while the law of demand is very imprecise even if predictions about the direction of prices following an increase in demand is also very robust in the short term (as long as we are dealing with humans). However, this stability tends to erode as time passes because other significant human reactions emerge, interact with the initial reaction and, as a result, modify behavior. We may know a lot about the direction in which each individual variable will influence others but because we often know nothing about the magnitude of these individual reactions, it is most often impossible to combine them to get even a sense of the timing and the direction of the “aggregate reaction”.

This being said, advances in predictive power are still possible for events that have already been set in motion if one can discover the laws that are governing their dynamics. A lot can be gained by developing such approaches and having the discipline and honesty to stick to these dynamics. It does not mean that there are no risks to a forecast made with such tools. We saw that, even in the case of a thrown ball, a collision with a bird could cause a deviation of the ball that would invalidate the “no collision” forecast. Still, the laws of physics governing the motion of a ball, once it is launched, are powerful forecasting tools. This is because making a forecast based of these laws to determine where a thrown ball should fall is likely to yield credible results; credible in the sense that such forecast would systematically yield more accurate results than a random forecast.

The same applies to aggregate demand; the sum of very large number of individual demand functions themselves derived from individual utility maximization under a budget constraint. As long as we attribute some weak property to the utility function, the aggregation of individual demand function (the result of the complex interaction of many consumers) still yields in a downward sloping aggregate demand curve and the powerful forecast that an increase in demand will put upward pressures on prices.

Watching the following fascinating video on using the stable structure characteristics of social networks as a forecasting tool, should convince you of the power of certain relationships for predicting behavior. Stick to such principles to know who to believe next time you hear someone talk about the future and you won’t get fooled by snake oil forecasters anymore.