Allowing for Model Uncertainty and Data Revisions in Central Banks’ Forecasting and Policy Analysis
Submitting InstitutionBirkbeck College
Unit of AssessmentEconomics and Econometrics
Summary Impact TypeEconomic
Research Subject Area(s)
Mathematical Sciences: Statistics
Economics: Applied Economics, Econometrics
Summary of the impact
Garratt's research on methods for quantifying the uncertainty surrounding
forecasts, uncertainty which arises from not knowing the true model of the
economy and from
having to use inaccurate data, has been applied by Central Banks and
national statistical agencies
in their forecasting exercises and their analysis of policy interventions.
Notably, Norges Bank (the
central bank of Norway) has developed a system called the System for
Averaging Models, which
they use when they make macroeconomic forecasts and when they predict the
effects of possible
monetary policy actions, which incorporates Garratt's results.
Garratt's research provides new methods to allow for uncertainty about
the 'true' model by using
combinations of different possible models, when making forecasts. His
research provides new
procedures to take `data uncertainty' into account, when forecasts have to
be based on real-time
data (that is, inaccurate data which is available to the policymaker when
a forecast is produced but
which is revised later on). Garratt's research quantifies the effect of
this uncertainty on forecasts
by constructing probability density functions. Central banks and
statistical agencies have applied
his findings when making forecasts and undertaking policy analysis.
Garratt's research has been
disseminated through refereed journal articles, conference presentations,
consultancy work with
policy makers, and presentations to policy makers, including an invited
presentation to HM
The first piece of research underpinning this case study is a book
written by Garratt and several
co-authors (3.1) which developed methods of probability forecasting
in the context of a small
macroeconomic model of the UK economy. In this work a pragmatic
implementation of Bayesian
Model Averaging was adopted to allow for model uncertainties. Amongst the
macroeconomic events of interest, they considered the probability of a
recession and of inflation
falling in the range 1.5%-3.5%, the target range considered at the time by
the Monetary Policy
Committee (MPC) of the Bank of England.
Following his initial work on model forecast uncertainty, Garratt's
subsequent research then
analysed the inaccuracy of initial measurements of important UK macro
variables such as output
growth, inflation, retail sales, unemployment, and its effects on the
degree of uncertainty
surrounding `now-casts' of the current state, and forecasts of the future
state, of the economy (3.2).
Knowing the size and direction of any bias, and the timing of any change
in bias, forecasters can
incorporate this information into their forecasts and therefore reduce or
at least have a greater
understanding of the uncertainty surrounding them. The results of this
research show that first
measurements, and indeed subsequent measurements, should be treated with
expect data to be revised, but the research shows that the biases change
over time and the
revised data does not always change smoothly and monotonically towards its
final `true' value. The
patterns of the revision process that Garratt's research documents enable
an assessment of the
usefulness of various releases of data. For example, there are large
biases in the national
accounts data but very little in money supply numbers. These results argue
for greater resources
to be put into data collection. Reference (3.4) expands on
Garratt's contribution to the Pickford
report on ONS preparation of national income statistics.
Garratt's research then develops and applies methods of model combination
which show how to
quantify uncertainty using density forecasts, robust to a constantly
environment (3.4) This extends previous work which has focused on
point forecasts. The expert
combination method that Garratt develops assumes that each "expert" uses a
distinct model to
produce a density forecast which is then aggregated by a "decision maker".
Since models differ in
their sensitivity to structural instabilities, he shows that the
aggregated density is robust to
Garratt shows that his procedures provide a useful means of computing
density forecasts for a
range of economic and financial variables, including GDP, the output gap
(the difference between
actual and potential output) and inflation. These forecasts, in general,
are robust to individual
model (expert) misspecification. In other words, his results indicate that
forecasts delivers gains, and mitigates the dangers of using mis-specified
individual models. He
shows that the combination approach can be used to improve the forecasts
(Dynamic Stochastic General Equilibrium) models, which, despite their
performance, can fail density-forecast evaluation-tests, unless combined.
He shows that it is
important to evaluate probabilistic forecasts using economic as well as
statistical loss functions.
The uncertainty associated with the output gap, a concept commonly
discussed by policymakers,
can be quantified and is extremely wide (3.4, 3.5). It is imprudent
to present, as is common, only a
central estimate of the output gap, since the density (the distribution of
forecast errors around it) is
often complex. Probabilistic estimates should be presented instead.
The broad context of Garratt's research, and motivation for it, is that
macroeconomic forecasts are
imperfect. If a forecaster provides information about inflation next
month, it contains considerable
inaccuracy. The inaccuracy stems (in part) from imprecise real-time
variables, model uncertainty, parameter uncertainty, and the inherently
unpredictable nature of the
macro-economy. Since forecasting is a key input into the work of
statistical agencies, finance
departments and independent fiscal watchdogs, it is important to
understand more about the
sources of uncertainty. Nevertheless, most policymaking institutions and
provide little information on the imprecision of their forecasts. Further,
the probabilities of
outcomes which are economically substantive, although not the most likely,
receive little attention.
Put differently, conventional macroeconomic forecasting neglects the
assessment of risk, and the
probability of extreme events.
References to the research
3.1 Garratt, A., K. Lee, M. H. Pesaran and Y. Shin, 2006, Global and
Modelling: A Long-Run Structural Approach, Oxford University Press:
3.2 Garratt, A., and S. Vahey, 2006, "UK Real-Time Macro Data
Journal, Volume 116, Issue 509, February, Pages F119-F135.
3.3 Garratt, A., K. Lee, E. Mise and K. Shields, 2008, "Real-Time
Representations of the Output
Gap", Review of Economics and Statistics, 90, No.4, 792-791.
3.4 Garratt, A., J. Mitchell and S. P. Vahey, "Measuring Output Gap
International Journal of Forecasting, forthcoming. An earlier
version circulates as "Measuring
Output Gap Uncertainty", CEPR Discussion Paper no. 7742, 2010.
3.5 Garratt, A., Lee, K. and K. Shields. "Measuring the Natural Output
Gap Using Actual and
Expected Output Data", Discussion Paper,University of
Nottingham, March 2011.
Details of the impact
The long-run small macro modelling approach developed by Garratt and
co-workers has been
adopted by a number of central banks and government authorities around the
world, including the
European Central Bank, and the central banks of Switzerland, Norway, New
and Malaysia. (5.1, 5.8).
By demonstrating the gains to density forecast combination and their
ability to produce well-
calibrated density forecasts, in the face of macroeconomic instabilities,
Garratt and his co-authors
have facilitated the adoption of these methods at central banks. For
example, their methods have
been directly taken up by policy makers at the Norges Bank (5.3)
who have developed their
System for Averaging Models. (Norges Bank gives an account of this on its
web site (5.6).) They
now routinely produce density forecasts for inflation and output growth,
based on a weighted
average of a range of forecasting models. Unlike point forecasts, density
uncertainty fully and facilitate more informed policymaking.
The findings and output from this body of work have impacted researchers
at policy institutions.
Garratt and Simon Price (a senior official in the Bank of England's
Monetary Analysis Division)
organised a one-day conference in October 2011 on density forecasts and
the use of fan charts at
the Bank of England at which Garratt's work was presented (5.7).
Academics, policy makers and
Monetary Policy Committee members at the Bank of England and officials of
the Federal Reserve
Bank of New York took part.
Since 2008, the academic papers produced have been circulated widely,
both as discussion
papers/journal papers and at international conferences. Over the course of
2009 and 2010, Garratt
and his co-authors visited central banks (de Nederlandsche Bank, the Bank
of England, the
European Central Bank, Norges Bank and the Reserve Bank of New Zealand)
and Eurostat to
disseminate their findings. They have written papers jointly with staff at
Eurostat and Norges Bank,
and shared their software with them, to further encourage data producers
and policy makers to use
their density-forecast combination-methods.
Garratt and co-workers have worked directly with Eurostat staff to
produce early statistical
estimates of Euro-area Gross Domestic Product.
Garratt participated in David Stockton's review of forecasting at the
Bank of England in July 2012.
Since the review the Bank of England has taken up the suggestion of
producing cumulative density
forecasts as developed in the work of Garratt et al (5.4). The Bank
of England's Inflation Report,
May 2013, pages 46-47, gives an account of this. Garratt presented his
work at HM Treasury (5.5)
in a series of Academic Policy Perspective seminars on "Probabilistic
Forecasting and Economic
Value" in March 2013.
As a direct consequence of his work, Garratt and his co-authors have
established a network on
the theme of probabilistic forecasting. Partner organizations, who have
committed modest funding
for the first year, include the Norges Bank, Bank of England and the
European Central Bank.
Partner investigators are researchers, including senior and junior
applied statisticians and practitioners. The major aim of this network is
to provide a forum for
exchanging ideas for operationalizing methodologies and to stimulate and
coordinate research into
new methods for probabilistic forecasting, forecast evaluation and
communication. It seeks to
develop methods for use by both practitioners and academic economists.
Since forecasting is a
key input into the work of statistical agencies, finance departments and
watchdogs, many public sector institutions will benefit from the outputs
of the network. The first
meeting of this network was held on 17th January 2013 at
Birkbeck. The topic was `Now-casting in
Central Banks' and it was attended by and had presentations from the
Norges Bank, Sveriges
Riksbank, Central Bank of Ireland, Bank of England, European Central Bank
and the Central
Banks of Austria and Spain (5.2, 5.9).
Sources to corroborate the impact
5.1 Deputy Head of Division, DG Research, European Central Bank.
5.2 Senior Statistician at European Commission, Eurostat (Head of Section
5.3 Economist, Monetary Policy Division, Norges Bank.
5.4 Senior Advisor, Monetary Analysis, Bank of England.
5.5 Deputy Director for Macroeconomic Analysis, H.M Treasury.
5.6 Norges Bank have developed a system for Averaging Models based on
5.7 Bank of England, Inflation Report, May 2013, pages 46-47.
5.8 Swiss National Bank, Economic Study 2009-06 "A VECX Model of the
(pdf file of the study listed as 6, 2009)
5.9 European Central Bank Working paper no. 1087, Sept 2009, "Modelling
Global Trade Flows:
Results From A GVAR Model".