### Allowing for Model Uncertainty and Data Revisions in Central Banksâ€™ Forecasting and Policy Analysis

**Submitting Institution**

Birkbeck College**Unit of Assessment**

Economics and Econometrics**Summary Impact Type**

Economic**Research Subject Area(s)**

Mathematical Sciences: Statistics

Economics: Applied Economics, Econometrics

**Download original**

PDF**Summary of the impact**

Garratt's research on methods for quantifying the uncertainty surrounding macroeconomic forecasts, uncertainty which arises from not knowing the true model of the economy and from having to use inaccurate data, has been applied by Central Banks and national statistical agencies in their forecasting exercises and their analysis of policy interventions. Notably, Norges Bank (the central bank of Norway) has developed a system called the System for Averaging Models, which they use when they make macroeconomic forecasts and when they predict the effects of possible monetary policy actions, which incorporates Garratt's results.

Garratt's research provides new methods to allow for uncertainty about the 'true' model by using combinations of different possible models, when making forecasts. His research provides new procedures to take `data uncertainty' into account, when forecasts have to be based on real-time data (that is, inaccurate data which is available to the policymaker when a forecast is produced but which is revised later on). Garratt's research quantifies the effect of this uncertainty on forecasts by constructing probability density functions. Central banks and statistical agencies have applied his findings when making forecasts and undertaking policy analysis. Garratt's research has been disseminated through refereed journal articles, conference presentations, consultancy work with policy makers, and presentations to policy makers, including an invited presentation to HM Treasury.

**Underpinning research**

The first piece of research underpinning this case study is a book
written by Garratt and several
co-authors (**3.1**) which developed methods of probability forecasting
in the context of a small
macroeconomic model of the UK economy. In this work a pragmatic
implementation of Bayesian
Model Averaging was adopted to allow for model uncertainties. Amongst the
many possible
macroeconomic events of interest, they considered the probability of a
recession and of inflation
falling in the range 1.5%-3.5%, the target range considered at the time by
the Monetary Policy
Committee (MPC) of the Bank of England.

Following his initial work on model forecast uncertainty, Garratt's
subsequent research then
analysed the inaccuracy of initial measurements of important UK macro
variables such as output
growth, inflation, retail sales, unemployment, and its effects on the
degree of uncertainty
surrounding `now-casts' of the current state, and forecasts of the future
state, of the economy (**3.2**).
Knowing the size and direction of any bias, and the timing of any change
in bias, forecasters can
incorporate this information into their forecasts and therefore reduce or
at least have a greater
understanding of the uncertainty surrounding them. The results of this
research show that first
measurements, and indeed subsequent measurements, should be treated with
caution. People
expect data to be revised, but the research shows that the biases change
over time and the
revised data does not always change smoothly and monotonically towards its
final `true' value. The
patterns of the revision process that Garratt's research documents enable
an assessment of the
usefulness of various releases of data. For example, there are large
biases in the national
accounts data but very little in money supply numbers. These results argue
for greater resources
to be put into data collection. Reference (**3.4**) expands on
Garratt's contribution to the Pickford
report on ONS preparation of national income statistics.

Garratt's research then develops and applies methods of model combination
which show how to
quantify uncertainty using density forecasts, robust to a constantly
changing macroeconomic
environment (**3.4**) This extends previous work which has focused on
point forecasts. The expert
combination method that Garratt develops assumes that each "expert" uses a
distinct model to
produce a density forecast which is then aggregated by a "decision maker".
Since models differ in
their sensitivity to structural instabilities, he shows that the
aggregated density is robust to
structural breaks.

Garratt shows that his procedures provide a useful means of computing
density forecasts for a
range of economic and financial variables, including GDP, the output gap
(the difference between
actual and potential output) and inflation. These forecasts, in general,
are robust to individual
model (expert) misspecification. In other words, his results indicate that
combining density
forecasts delivers gains, and mitigates the dangers of using mis-specified
individual models. He
shows that the combination approach can be used to improve the forecasts
from policymaking
(Dynamic Stochastic General Equilibrium) models, which, despite their
competitive point-forecast
performance, can fail density-forecast evaluation-tests, unless combined.
He shows that it is
important to evaluate probabilistic forecasts using economic as well as
statistical loss functions.
The uncertainty associated with the output gap, a concept commonly
discussed by policymakers,
can be quantified and is extremely wide (**3.4, 3.5**). It is imprudent
to present, as is common, only a
central estimate of the output gap, since the density (the distribution of
forecast errors around it) is
often complex. Probabilistic estimates should be presented instead.

The broad context of Garratt's research, and motivation for it, is that macroeconomic forecasts are imperfect. If a forecaster provides information about inflation next month, it contains considerable inaccuracy. The inaccuracy stems (in part) from imprecise real-time measurements, latent variables, model uncertainty, parameter uncertainty, and the inherently unpredictable nature of the macro-economy. Since forecasting is a key input into the work of statistical agencies, finance departments and independent fiscal watchdogs, it is important to understand more about the sources of uncertainty. Nevertheless, most policymaking institutions and statistical agencies provide little information on the imprecision of their forecasts. Further, the probabilities of outcomes which are economically substantive, although not the most likely, receive little attention. Put differently, conventional macroeconomic forecasting neglects the assessment of risk, and the probability of extreme events.

**References to the research**

3.1 Garratt, A., K. Lee, M. H. Pesaran and Y. Shin, 2006, *Global and
National Macroeconometric
Modelling: A Long-Run Structural Approach*, Oxford University Press:
Oxford.

3.2 Garratt, A., and S. Vahey, 2006, "UK Real-Time Macro Data
Characteristics," *Economic
Journal*, Volume 116, Issue 509, February, Pages F119-F135.

3.3 Garratt, A., K. Lee, E. Mise and K. Shields, 2008, "Real-Time
Representations of the Output
Gap", *Review of Economics and Statistics,* 90, No.4, 792-791.

3.4 Garratt, A., J. Mitchell and S. P. Vahey, "Measuring Output Gap
Now-cast Uncertainty",
*International Journal of Forecasting,* forthcoming. An earlier
version circulates as "Measuring
Output Gap Uncertainty", CEPR Discussion Paper no. 7742, 2010.

3.5 Garratt, A., Lee, K. and K. Shields. "Measuring the Natural Output
Gap Using Actual and
Expected Output Data*", Discussion Paper*,*University of
Nottingham, March 2011*.

**Details of the impact**

The long-run small macro modelling approach developed by Garratt and
co-workers has been
adopted by a number of central banks and government authorities around the
world, including the
European Central Bank, and the central banks of Switzerland, Norway, New
Zealand, Indonesia
and Malaysia. (**5.1, 5.8**).

By demonstrating the gains to density forecast combination and their
ability to produce well-
calibrated density forecasts, in the face of macroeconomic instabilities,
Garratt and his co-authors
have facilitated the adoption of these methods at central banks. For
example, their methods have
been directly taken up by policy makers at the Norges Bank (**5.3**)
who have developed their
System for Averaging Models. (Norges Bank gives an account of this on its
web site (**5.6**).) They
now routinely produce density forecasts for inflation and output growth,
based on a weighted
average of a range of forecasting models. Unlike point forecasts, density
forecasts capture
uncertainty fully and facilitate more informed policymaking.

The findings and output from this body of work have impacted researchers
at policy institutions.
Garratt and Simon Price (a senior official in the Bank of England's
Monetary Analysis Division)
organised a one-day conference in October 2011 on density forecasts and
the use of fan charts at
the Bank of England at which Garratt's work was presented (**5.7**).
Academics, policy makers and
Monetary Policy Committee members at the Bank of England and officials of
the Federal Reserve
Bank of New York took part.

Since 2008, the academic papers produced have been circulated widely, both as discussion papers/journal papers and at international conferences. Over the course of 2009 and 2010, Garratt and his co-authors visited central banks (de Nederlandsche Bank, the Bank of England, the European Central Bank, Norges Bank and the Reserve Bank of New Zealand) and Eurostat to disseminate their findings. They have written papers jointly with staff at Eurostat and Norges Bank, and shared their software with them, to further encourage data producers and policy makers to use their density-forecast combination-methods.

Garratt and co-workers have worked directly with Eurostat staff to produce early statistical estimates of Euro-area Gross Domestic Product.

Garratt participated in David Stockton's review of forecasting at the
Bank of England in July 2012.
Since the review the Bank of England has taken up the suggestion of
producing cumulative density
forecasts as developed in the work of Garratt et al (**5.4**). The Bank
of England's *Inflation Report*,
May 2013, pages 46-47, gives an account of this. Garratt presented his
work at HM Treasury (**5.5**)
in a series of Academic Policy Perspective seminars on "*Probabilistic
Forecasting and Economic
Value"* in March 2013.

As a direct consequence of his work, Garratt and his co-authors have
established a network on
the theme of probabilistic forecasting. Partner organizations, who have
committed modest funding
for the first year, include the Norges Bank, Bank of England and the
European Central Bank.
Partner investigators are researchers, including senior and junior
academic econometricians,
applied statisticians and practitioners. The major aim of this network is
to provide a forum for
exchanging ideas for operationalizing methodologies and to stimulate and
coordinate research into
new methods for probabilistic forecasting, forecast evaluation and
communication. It seeks to
develop methods for use by both practitioners and academic economists.
Since forecasting is a
key input into the work of statistical agencies, finance departments and
independent fiscal
watchdogs, many public sector institutions will benefit from the outputs
of the network. The first
meeting of this network was held on 17^{th} January 2013 at
Birkbeck. The topic was *`Now-casting in
Central Banks'* and it was attended by and had presentations from the
Norges Bank, Sveriges
Riksbank, Central Bank of Ireland, Bank of England, European Central Bank
and the Central
Banks of Austria and Spain (**5.2, 5.9**).

**Sources to corroborate the impact **

5.1 Deputy Head of Division, DG Research, European Central Bank.

5.2 Senior Statistician at European Commission, Eurostat (Head of Section "Principal European Economic Indicators").

5.3 Economist, Monetary Policy Division, Norges Bank.

5.4 Senior Advisor, Monetary Analysis, Bank of England.

5.5 Deputy Director for Macroeconomic Analysis, H.M Treasury.

5.6 Norges Bank have developed a system for Averaging Models based on Garratt's research

5.7 Bank of England, *Inflation Report*, May 2013, pages 46-47.
http://www.bankofengland.co.uk/publications/Pages/inflationreport/2013/ir1302.aspx.

5.8 Swiss National Bank, Economic Study 2009-06 "A VECX Model of the Swiss Economy". http://www.snb.ch/en/iabout/pub/oecpub/id/research_studies. (pdf file of the study listed as 6, 2009)

5.9 European Central Bank Working paper no. 1087, Sept 2009, "Modelling
Global Trade Flows:
Results From A GVAR Model".

http://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp1087.pdf.