3Q2016 - Forecasting


In September 2008, the investment bank, Lehman Brothers, filed for bankruptcy. This was an event that virtually no forecaster had predicted, let alone the chain reaction it triggered: all over the world, stock exchanges fell sharply, such as here in South Korea.

Inside ING

Confessions of an economic forecaster

ING Chief Economist Mark Cliffe reveals eighteen secrets and myths about the economic forecasters’ art. Despite its obvious fallibility in an irretrievably uncertain world, there are useful lessons to be learnt.

This article falls within the theme Forecasting

Read also these other articles:

Worse than fallible

It’s not just that economic forecasts are prone to be wrong. It’s that they tend to let you down when you most need them. Paul Samuelson once said the stock market has “predicted nine out of the last five recessions”. By comparison, economic forecasters rarely predict any! They are typically caught out by shocks.

About Mark Cliffe?

But decisions need forecasts

Economists are well aware of the fallibility of forecasts. But they have no option but to keep making them. People can't live without them – all decisions are based on a view of the future. People also keep asking for them because they can provide a (spuriously) precise way of justifying decisions made for other reasons. And if things go wrong, they can blame the ‘expert’!

A deceptive air of certainty

In an uncertain world, forecasts, and the story behind them, provide us with comfort. But if we place too much faith in them they may lead us astray. Such overconfidence gives us an illusion of control. Forecasters often ignore the sage advice that “it’s better to be vaguely right rather than exactly wrong”. Having constructed models to predict the future, forecasters tend to forget that they are only approximations.

Charlatans rule

Confident dogmatic forecasters are provocative and often grab the headlines. Unfortunately, their strong but simple narratives are often way off the mark. The devil is often in the detail, so we should be wary of forecasts based on a single factor. It’s also a good discipline to try to attach a probability to your forecasts. Admitting uncertainty and being open about your degree of confidence in your prediction is not a sign of weakness, but helpful realism.

Try to be objective

We’re all biased, and our forecasts reflect our pre-conceptions. Forecasters need to acknowledge this, and aim for objective assessments. It is easy to pay too much attention to the most recent and most shocking news. Yet, forecasters have a natural human tendency to indulge in elegant, after-the-fact rationalisations. Hindsight is indeed a wonderful thing. But it’s all too tempting to succumb to selective memory, cherishing your successes while forgetting your mistakes.

The more the merrier

By making more forecasts, you have more chances to ‘win’ by getting something right! But more seriously, it also gives you more chances to learn. Rather than sticking with a forecast that is off track, it is important to revise your views in the light of new information. As John Maynard Keynes put it “when the facts change, I change my mind. What do you do, sir?”

Unlike traders, forecasters do get marks for style

Financial market traders and investors are judged by the profits that they make. As a result, they are happy to get it right for the wrong reasons. But relying on luck is dangerous.

By contrast, and contrary to popular opinion, forecasting is not just, or maybe even mainly, about accuracy. Aside from providing provocative or comforting stories, forecasts are about survival rather than getting it right. And accuracy is often relative, just as when faced with a charging bull, your survival depends not on outrunning the bull, but your neighbour.

It’s more about ‘why’ than ‘what’

Meaningful forecasts come with useful explanations. They should provide the users with a framework of thought and wisdom, rather than just knowledge and information. A common error is to assume that because two things are moving together that the two are related, let alone that one is causing the other: as scientists say, correlation is not causation. Wisdom comes from a deeper understanding of why things happen.

Past isn’t always prologue

The models used by forecasters are based on past causal relationships. Former British Prime Minister Harold MacMillan once said that his biggest fear was "Events dear boy, events". So it is for forecasts. Unprecedented shocks have a habit of wrecking them. Economists make a crucial distinction between risk and uncertainty. Risk is where you know the odds, like playing roulette. Uncertainty is where you don’t. Worse still, you may have no idea that the game is about to change entirely; Former US Defense Secretary Donald Rumsfeld referred to this as the ‘unknown unknowns’. This is why forecasting is more of an art than a science.

Some things are more certain than others

The further out you look, the more uncertain things are, making forecasts even less reliable. This is why economic pundits on TV prefer to look smart by focusing on the short term. In fact, some forecasts of monthly indicators, which are released with a month or so delay, are more like ‘backcasts’, because anecdotal information helps to ‘predict’ last month’s outcome.

Apart from the time dimension, some variables are more volatile, or prone to extreme outcomes than others. For example, commodity prices and share prices are far more volatile than consumer prices. The importance of this is reinforced by the fact that the starting point for the forecast is often critical to the outcome. Economies, like the weather or biological systems, may be acutely sensitive to initial conditions. The phenomenon of one thing leading to another, or ‘path dependency’, makes forecasting especially difficult. In the metaphor of chaos theory, a butterfly flapping its wings may ultimately lead to a tornado far away.

Some things matter more than others

In wrestling with the complexity of the economy, the forecaster’s job is helped by the fact that some things matter more than others, and therefore deserve more attention.

However, sometimes we are prone to miss crucial risks to a forecast. We may fail to recognise our implicit assumptions, often because we mistakenly take things for granted. The global financial crisis that broke out in 2008 at least partly stemmed from a widespread complacency that US house prices don’t fall.

Stories can help, but also hinder

One way of dealing with the myriad of possible futures is to construct scenarios. Rather than ‘betting the ranch’ on a single forecast, scenarios are stories constructed around the key drivers of the outlook in terms of their probability and impact. The stories give the users benchmarks for answering the question ”what kind of a world are we in?”. This can be a helpful way of thinking through potentially disruptive events beforehand, because people are liable to make bad decisions under stress. Contingency planning can allow ‘grace under fire’.

Don’t just forecast what you can measure

Economists often tend to ignore non-economic factors that are hard to put into their models. In so doing they often miss the 'elephants in the room'. In particular, politics and institutions matter. In the long run they can be critical in determining whether countries succeed or fail. And even in the short term, the outcome of a divisive election, as we have been observing recently in the US, may be far more important to the outlook than a forensic analysis of the latest economic data.

More data isn’t always better

Data doesn’t speak for itself. Economic forecasters, embarrassed again by the global financial crisis, are in danger of being overshadowed by the new breed of ‘data scientists’. The emergence of Big Data and machine learning is opening up great new forecasting opportunities. In effect, massive computational capacity is being used to spot patterns through automated trial and error.

However, powerful new weapons are dangerous in the wrong hands. ‘Predictive analytics’ is in reality a tool for micro-economic forecasting. It’s not a ‘silver bullet’. More data and more variables gives more chances of correlations. The problem is that the proportion that are dangerously misleading rises even faster. As Nassim Taleb puts it, ‘Big Data’ brings ‘cherry-picking to an industrial level’. It is crucial to put the data into context, check its quality, understand the questions we are trying to answer, and to separate the signal from the noise. Identifying ‘why’ things are happening is vital for developing better forecasts.

Magnitudes aren’t always necessary

For some purposes, we don’t need precise, quantified forecasts. Sometimes the direction is enough. That said, it also helps to get the timing right too! This is particularly true of questions that have binary outcomes. Wars and elections are won or lost. We are far more concerned with who wins an election than their margin of victory.

Such events are not easy to plug into economic models, and require different methods, with an emphasis on probabilities. Scenarios based on these contingent forecasts can then be produced.

Not all relationships are linear

Non-linear relationships create major problems for forecasters. They are difficult to model, and harder still to forecast. This is particularly troubling now because many of the emerging new digital technologies exhibit, or have the potential for, exponential, explosive growth. Minor initial errors in your forecasting model quickly lead to huge ones.

In this environment, Nassim Taleb, in his book ‘Antifragile’, advocates that we give up on forecasting. Instead, we should focus on non-predictive forms of decision-making, looking for choices that are not merely robust in the face of unexpected extreme changes, but actively benefit from them. Sadly, this is easier said than done, and many businesses appear to be paralysed by uncertainty rather than energised by techno-phoria.

Beware of market madness

Financial markets are not like other markets, because their prices are themselves forecasts. Financial asset prices reflect expected future returns and risk. According to the ‘efficient market hypothesis’, you cannot beat the wisdom of the crowd, because its collective intelligence already incorporates all available information. While it is certainly true that markets quickly incorporate incoming information, and it is hard, if not impossible to outperform them, this hypothesis does not stand up to scrutiny. Investor beliefs about the future are loosely held, and prone to shifts from optimism to pessimism. This leads to booms and busts in market prices that leave consensus-minded forecasters trailing in their wake.

The ‘unusually uncertain’ cliché

Forecasters are prone to downplay the potential accuracy of their forecasts by saying ‘times are unusually uncertain’. It’s a cliché that is surprisingly rarely challenged. All eras have their own uncertainties. The current era is surely beset by potential volatility from digital technologies, political and social turmoil, unconventional monetary policies, financial fragilities, and regulatory change. But hindsight makes it easy to forget previous eras had their own uncertainties at the time – just think of the existential threats of the nuclear arms race during the Cold War. By deterring risk taking, the ‘unusual uncertainty’ cliché often equates to a pessimistic outlook. Yet we should not forget that the darkest hour comes before the dawn. Positive surprises do happen.

Related articles


Read more

Next quarter

Read more


Read more

Interview “The more famous an expert is, the less accurate their forecasts are”
Read further