Summary of the impact

Garratt's research on methods for quantifying the uncertainty surrounding macroeconomic forecasts, uncertainty which arises from not knowing the true model of the economy and from having to use inaccurate data, has been applied by Central Banks and national statistical agencies in their forecasting exercises and their analysis of policy interventions. Notably, Norges Bank (the central bank of Norway) has developed a system called the System for Averaging Models, which they use when they make macroeconomic forecasts and when they predict the effects of possible monetary policy actions, which incorporates Garratt's results.

Garratt's research provides new methods to allow for uncertainty about the 'true' model by using combinations of different possible models, when making forecasts. His research provides new procedures to take `data uncertainty' into account, when forecasts have to be based on real-time data (that is, inaccurate data which is available to the policymaker when a forecast is produced but which is revised later on). Garratt's research quantifies the effect of this uncertainty on forecasts by constructing probability density functions. Central banks and statistical agencies have applied his findings when making forecasts and undertaking policy analysis. Garratt's research has been disseminated through refereed journal articles, conference presentations, consultancy work with policy makers, and presentations to policy makers, including an invited presentation to HM Treasury.

Underpinning research

The first piece of research underpinning this case study is a book written by Garratt and several co-authors (3.1) which developed methods of probability forecasting in the context of a small macroeconomic model of the UK economy. In this work a pragmatic implementation of Bayesian Model Averaging was adopted to allow for model uncertainties. Amongst the many possible macroeconomic events of interest, they considered the probability of a recession and of inflation falling in the range 1.5%-3.5%, the target range considered at the time by the Monetary Policy Committee (MPC) of the Bank of England.

Following his initial work on model forecast uncertainty, Garratt's subsequent research then analysed the inaccuracy of initial measurements of important UK macro variables such as output growth, inflation, retail sales, unemployment, and its effects on the degree of uncertainty surrounding `now-casts' of the current state, and forecasts of the future state, of the economy (3.2). Knowing the size and direction of any bias, and the timing of any change in bias, forecasters can incorporate this information into their forecasts and therefore reduce or at least have a greater understanding of the uncertainty surrounding them. The results of this research show that first measurements, and indeed subsequent measurements, should be treated with caution. People expect data to be revised, but the research shows that the biases change over time and the revised data does not always change smoothly and monotonically towards its final `true' value. The patterns of the revision process that Garratt's research documents enable an assessment of the usefulness of various releases of data. For example, there are large biases in the national accounts data but very little in money supply numbers. These results argue for greater resources to be put into data collection. Reference (3.4) expands on Garratt's contribution to the Pickford report on ONS preparation of national income statistics.

Garratt's research then develops and applies methods of model combination which show how to quantify uncertainty using density forecasts, robust to a constantly changing macroeconomic environment (3.4) This extends previous work which has focused on point forecasts. The expert combination method that Garratt develops assumes that each "expert" uses a distinct model to produce a density forecast which is then aggregated by a "decision maker". Since models differ in their sensitivity to structural instabilities, he shows that the aggregated density is robust to structural breaks.

Garratt shows that his procedures provide a useful means of computing density forecasts for a range of economic and financial variables, including GDP, the output gap (the difference between actual and potential output) and inflation. These forecasts, in general, are robust to individual model (expert) misspecification. In other words, his results indicate that combining density forecasts delivers gains, and mitigates the dangers of using mis-specified individual models. He shows that the combination approach can be used to improve the forecasts from policymaking (Dynamic Stochastic General Equilibrium) models, which, despite their competitive point-forecast performance, can fail density-forecast evaluation-tests, unless combined. He shows that it is important to evaluate probabilistic forecasts using economic as well as statistical loss functions. The uncertainty associated with the output gap, a concept commonly discussed by policymakers, can be quantified and is extremely wide (3.4, 3.5). It is imprudent to present, as is common, only a central estimate of the output gap, since the density (the distribution of forecast errors around it) is often complex. Probabilistic estimates should be presented instead.

The broad context of Garratt's research, and motivation for it, is that macroeconomic forecasts are imperfect. If a forecaster provides information about inflation next month, it contains considerable inaccuracy. The inaccuracy stems (in part) from imprecise real-time measurements, latent variables, model uncertainty, parameter uncertainty, and the inherently unpredictable nature of the macro-economy. Since forecasting is a key input into the work of statistical agencies, finance departments and independent fiscal watchdogs, it is important to understand more about the sources of uncertainty. Nevertheless, most policymaking institutions and statistical agencies provide little information on the imprecision of their forecasts. Further, the probabilities of outcomes which are economically substantive, although not the most likely, receive little attention. Put differently, conventional macroeconomic forecasting neglects the assessment of risk, and the probability of extreme events.

References to the research

Details of the impact

The long-run small macro modelling approach developed by Garratt and co-workers has been adopted by a number of central banks and government authorities around the world, including the European Central Bank, and the central banks of Switzerland, Norway, New Zealand, Indonesia and Malaysia. (5.1, 5.8).

By demonstrating the gains to density forecast combination and their ability to produce well- calibrated density forecasts, in the face of macroeconomic instabilities, Garratt and his co-authors have facilitated the adoption of these methods at central banks. For example, their methods have been directly taken up by policy makers at the Norges Bank (5.3) who have developed their System for Averaging Models. (Norges Bank gives an account of this on its web site (5.6).) They now routinely produce density forecasts for inflation and output growth, based on a weighted average of a range of forecasting models. Unlike point forecasts, density forecasts capture uncertainty fully and facilitate more informed policymaking.

The findings and output from this body of work have impacted researchers at policy institutions. Garratt and Simon Price (a senior official in the Bank of England's Monetary Analysis Division) organised a one-day conference in October 2011 on density forecasts and the use of fan charts at the Bank of England at which Garratt's work was presented (5.7). Academics, policy makers and Monetary Policy Committee members at the Bank of England and officials of the Federal Reserve Bank of New York took part.

Since 2008, the academic papers produced have been circulated widely, both as discussion papers/journal papers and at international conferences. Over the course of 2009 and 2010, Garratt and his co-authors visited central banks (de Nederlandsche Bank, the Bank of England, the European Central Bank, Norges Bank and the Reserve Bank of New Zealand) and Eurostat to disseminate their findings. They have written papers jointly with staff at Eurostat and Norges Bank, and shared their software with them, to further encourage data producers and policy makers to use their density-forecast combination-methods.

Garratt and co-workers have worked directly with Eurostat staff to produce early statistical estimates of Euro-area Gross Domestic Product.

Garratt participated in David Stockton's review of forecasting at the Bank of England in July 2012. Since the review the Bank of England has taken up the suggestion of producing cumulative density forecasts as developed in the work of Garratt et al (5.4). The Bank of England's Inflation Report, May 2013, pages 46-47, gives an account of this. Garratt presented his work at HM Treasury (5.5) in a series of Academic Policy Perspective seminars on "Probabilistic Forecasting and Economic Value" in March 2013.

As a direct consequence of his work, Garratt and his co-authors have established a network on the theme of probabilistic forecasting. Partner organizations, who have committed modest funding for the first year, include the Norges Bank, Bank of England and the European Central Bank. Partner investigators are researchers, including senior and junior academic econometricians, applied statisticians and practitioners. The major aim of this network is to provide a forum for exchanging ideas for operationalizing methodologies and to stimulate and coordinate research into new methods for probabilistic forecasting, forecast evaluation and communication. It seeks to develop methods for use by both practitioners and academic economists. Since forecasting is a key input into the work of statistical agencies, finance departments and independent fiscal watchdogs, many public sector institutions will benefit from the outputs of the network. The first meeting of this network was held on 17^th January 2013 at Birkbeck. The topic was `Now-casting in Central Banks' and it was attended by and had presentations from the Norges Bank, Sveriges Riksbank, Central Bank of Ireland, Bank of England, European Central Bank and the Central Banks of Austria and Spain (5.2, 5.9).

Sources to corroborate the impact

5.1 Deputy Head of Division, DG Research, European Central Bank.

5.2 Senior Statistician at European Commission, Eurostat (Head of Section "Principal European Economic Indicators").

5.3 Economist, Monetary Policy Division, Norges Bank.

5.4 Senior Advisor, Monetary Analysis, Bank of England.

5.5 Deputy Director for Macroeconomic Analysis, H.M Treasury.

5.6 Norges Bank have developed a system for Averaging Models based on Garratt's research

http://www.norges-bank.no/en/price-stability/models-for-monetary-policy-analysis-and-forecasting/sam/.

5.7 Bank of England, Inflation Report, May 2013, pages 46-47. http://www.bankofengland.co.uk/publications/Pages/inflationreport/2013/ir1302.aspx.

5.8 Swiss National Bank, Economic Study 2009-06 "A VECX Model of the Swiss Economy". http://www.snb.ch/en/iabout/pub/oecpub/id/research_studies. (pdf file of the study listed as 6, 2009)

5.9 European Central Bank Working paper no. 1087, Sept 2009, "Modelling Global Trade Flows: Results From A GVAR Model".
http://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp1087.pdf.