Better risk management through improved weather forecasting
Submitting Institution
London School of Economics & Political ScienceUnit of Assessment
Mathematical SciencesSummary Impact Type
EnvironmentalResearch Subject Area(s)
Mathematical Sciences: Statistics
Earth Sciences: Atmospheric Sciences
Economics: Econometrics
Summary of the impact
Research by Professor Leonard Smith and the LSE Centre for the Analysis
of Time Series (CATS) on forecasting in non-linear and often chaotic
systems, with particular attention to weather, has led to advances in
three areas: 1) national and international weather industry products and
services that are built upon state-of-the-art research and knowledge, 2)
dissemination of state-of-the-art practice in forecast production and
verification to national, regional and local weather centres around the
world, and 3) the introduction of, and new applications in,
state-of-the-art forecasting methods in industries facing high uncertainty
and risk, e.g. insurance and energy.
Underpinning research
Research Insights and Outputs: From its beginnings in 2002, CATS
has advanced the generation and interpretation of state-of-the-art weather
forecasts in support of decision-making. Professor Leonard Smith's
research over the past 25 years has focused on forecasting nonlinear,
often chaotic, systems, including weather forecasting. In 2003, Professor
Smith was awarded two of the ten Department of Trade and Industry Faraday
Maths grants (REMIND and DIME). One focused on improving the
interpretation and evaluation of ensembles of simulations, the other on
their application in a variety of industrial settings. Modern operational
weather forecasts take an ensemble (Monte Carlo) approach to account for
uncertainty (in the initial condition); scientific, statistical and
philosophical questions still remain as to how to interpret an ensemble of
model-trajectories as a probability forecast for the future of the system.
The research involved translating a sample of about 51 points (the
ensemble forecast) in a 10,000,000 dimensional space into information
regarding the reliability of a single, high resolution forecast or into a
probability forecast for a similar target variable (which was sometimes a
nonlinear combination of several meteorological variables). A key aspect
is that, since the weather model is nontrivially imperfect, on some days
the ensemble is seen, in hindsight, to capture the evolution of the
weather, while on other days the model is unable to shadow the events due
to model error. The research [5] combined (a) kernel dressing the ensemble
members (turning simulations into probabilities), (b) blending with the
climatological distribution (a prior based upon historical observations
for the relevant phase of the seasonal cycle), (c) an empirically-driven
recognition of fundamental limitations for the method itself and (d) input
from energy traders in terms of presentation and tolerances, in order to
achieve a robust, actionable probabilistic tool.
In 2005 the applied aspects of this research were extended under the NERC
grant NAPSTER and both theory and application under the EU FP6 ENSEMBLES
project. Following IJ Good's work on quantifying skill in the 1950s, CATS
[1,4] enabled the improved tuning of models and the communication of
probabilistic skill [6], alternatively in "bits" for those familiar with
information theory and as an "effective interest rate" for those more
familiar with financial work (traders in the energy sector; managers in
the weather sector). This work was performed jointly with Hagedorn of the
European Centre for Medium-range Weather Forecasts, which hired Smith as a
consultant.
The key research insight here was that a sample of simulations from one
(or more) imperfect physics-based simulation models can provide more
information if they are not treated as a sample drawn from some target
probability density function. CATS led research development of kernel
dressing of individual ensemble members and blending with the background
climatological distribution. While this method is superficially similar to
Bayesian Model Averaging techniques in weather forecasting, which appeared
shortly after (and cite) the CATS work, the epistemological framework is
rather different. The novel use of a prior distribution (the
"climatology") to lessen the impact of model error was important. Breadth
of applicability was shown in additional research papers on off-shore wave
height with Royal Dutch Shell [3] and on wind energy production [2].
Key researchers: Professor Leonard Smith, Reader/Professor of
Statistics, LSE, March 2000-
Dr Jochen Broecker, Postdoctoral Research Officer, LSE March 2003-April
2007
Dr Liam Clarke, Postdoctoral Research Officer, LSE March 2003-May 2008
Dr Renate Hagedorn, Scientific Researcher, ECMWF, ~2000-2011
Dr Hailiang Du, Graduate Student/Postdoctoral Research Officer, LSE,
2004-present
References to the research
1. MS Roulston & LA Smith (2002) Evaluating probabilistic forecasts
using information theory, Monthly Weather Review 130 6: 1653. DOI:
10.1175/1520
2. MS Roulston, DT Kaplan, J Hardenberg & LA Smith (2003) Using
Medium Range Weather Forecasts to Improve the Value of Wind Energy
Production, Renewable Energy 29 (4) April 585.DOI:
10.1016/S0960-1481(02)00054-X
3. MS Roulston, J Ellepola & LA Smith (2005) Forecasting Wave Height
Probabilities with Numerical Weather Prediction Models, Ocean
Engineering 32 (14-15), 1841. DOI:10.1016/j.oceaneng.2004.11.012
4. J Bröcker & LA Smith (2007) Scoring Probabilistic Forecasts: The
Importance of Being Proper Weather and Forecasting, 22 (2), 382.
DOI: 10.1175/WAF966.1
5. J Bröcker & LA Smith (2008) From Ensemble Forecasts to Predictive
Distribution Functions Tellus A 60(4): 663. DOI:
10.1111/j.1600-0870.2008.00333.x
6. R Hagedorn & LA Smith (2009) Communicating the value of
probabilistic forecasts with weather roulette. Meteorological
Applications 16 (2): 143. DOI: 10.1002/met.92
Evidence of quality: Publications appeared in peer-reviewed
journals; Professor Smith received the FitzRoy Prize of the Royal
Meteorological Society for distinguished work in applied meteorology in
2002; and the research received grant funding that included:
• Direct & Inverse Modelling in End-to-End Environmental Estimation
(DIME) EPSRC, 2003-2005 (£94,360), GR/R92363/01.
• Real-time Modelling of Nonlinear Datastreams (REMIND). EPSRC, 2003-2005. (£85,827, plus industrial in-kind support from National Grid Company
and Intertec), GR/R92271/01
• Nonlinear Analysis & Prediction Statistics from Timeseries &
Ensemble-forecast Realizations (NAPSTER) NERC, 2005 - 2008. (£152,481),
NE/D00120X/1.
• ENSEMBLE-based Predictions of Climate Changes and their Impacts
(ENSEMBLES), EU FP6, 2004-2009. (£112,926), GOCE-CT-2003-505539.
• Towards Identifying and Increasing the Socio-Economic Value of
High-Impact Weather Forecasts, US NOAA, 2003-2004 (£94,538).
• BIOS RPI Grant for evaluation of seasonal forecasts for the insurance
industry.($50k) 2012.
Details of the impact
Weather has an impact on people, businesses and economies every single
day. Even fairly common weather, with no extreme meteorological elements
whatsoever, can produce costly and stressful disruptions — to power and
water supplies, food production and distribution networks, travel systems,
communication networks (e.g. GPS) and other infrastructure necessary to
the smooth functioning of families, communities and societies. And changes
in forecasts themselves, not the weather at all, can have huge impacts on
prices and the provision of (cancellation of) services. There is therefore
an imperative for tools which enhance the use of modern (probabilistic)
weather forecast in addition to the need for weather predictions that are
timely, accurate and enable users with varying needs and levels of
understanding to plan for and manage their responses to weather conditions
and events. There are three ways in which Professor Smith and his
colleagues at CATS have responded to this need: 1) advancing the state of
the art in probabilistic forecasting in the weather industry; 2) advancing
the state of practice in weather forecasting; and 3) advancing
understanding and application amongst various types of users in business
and industry.
Advancing the state of the art in the weather industry
Professor Smith and CATS have significant and often longstanding
relationships focused on transferring knowledge and advancing the state of
the art with a number of key institutions involved in weather forecasting.
The relationship with the European Centre for Medium Range Weather
Forecasts (ECMWF), "considered the worldwide leader in global,
medium-range, monthly and seasonal predictions", dates back to the 1994
Royal Society of Meteorologists meeting on predictability [A].
Longstanding partners also include: the UK Met Office [B]; the US Naval
Research Laboratory (NRL), where one of Professor Smith's former students
now leads a group forecasting hurricanes in the Pacific; and the US
National Centre for Atmospheric Research (NCAR), where Dr. Du is sharing
methods and providing a test bed for data assimilations. More recent
partnerships include: the International Research Institute for Climate and
Society (IRI) [C], which focuses on climate service development, primarily
within developing countries; the Risk Prediction Initiative (RPI) of the
Bermuda Institute of Ocean Sciences (BIOS) [D]; and the Industrial
Mathematics Knowledge Transfer Network of the Smith Institute. These
relationships focus on two critical areas of weather forecasting:
predictability and verification.
According to the head of ECMWF's Predictability Division, Roberto Buizza,
"During the past 15 years, Prof. Lenny Smith has contributed to a range of
key subjects in the area of predictability, which had an impact on the
design of the operational systems of ECMWF, and in the use of
ensemble-based, probabilistic forecasts...The work we did on the
estimation of the impact of non-linearity ...provided estimates on the
time when non-linearity impacted on forecast accuracy, thus helping us
refining our techniques and methodologies. Prof. Smith and his group's
contribution in this area continued throughout the years, and the systems
operational today are still benefitting from his research done years
ago...and supported their gradual extension of the forecast range to the
monthly, seasonal and decadal time scale" [A]. IRI's Chief Climate
Scientist, Dr. Simon Mason, likewise has been using Professor Smith's work
on second-order uncertainty and intractability in "communicating forecast
uncertainty honestly at all times scales" and relying on Smith's advice in
contributing to the World Climate Research Programme Working Group on
Regional Climate, "where issues of estimating forecast uncertainty are
paramount" [C].
ECMWF, IRI and BIOS have all been influenced by the work of Professor
Smith and CATS on verification, i.e. metrics to evaluate the accuracy of
weather forecasts. For ECMWF, "his work on the assessment of probabilistic
forecasts using information theory provided a new and extremely valuable
measure of the quality of a probabilistic forecasting scheme. Based on the
amount of a data compression it allows, called ignorance (Roulston &
Smith 2002), this measure has been used routinely to assess the quality of
ECMWF operational products" [A]. IRI's Dr. Mason was also influenced by
the underpinning research [e.g. 6] and by extensive discussions with
Professor Smith on the properties of verification scores, which is
reflected in guidance and training that Dr. Mason has produced and
delivered for the World Meteorology Organisation (WMO) [C] (see next
section). Professor Smith has worked with BIOS on evaluating the skill of
seasonal forecasts of sea surface temperatures in the tropical Atlantic
and the equatorial Pacific Ocean where such temperatures play a role in
the formation of hurricanes. This knowledge has been transferred through
papers and through two workshops with members of BIOS' Risk Prediction
Initiative [D].
Advancing practice in probabilistic weather forecasting
The advance of practice is where the reach of the impact has perhaps been
greatest, through tools, guidance, training, software and products based
on the underpinning research that have influenced weather forecasters
around the world. What has proved to be highly influential is a conceptual
framework called Weather Roulette [6] that allows weather predictors to
more easily communicate the skill and value of forecasts to customers and
users by translating the probabilities into effective daily interest
rates, which is particularly useful in situations where small
probabilities can lead to large costs or benefits. Weather Roulette has
been incorporated into ECMWF's Ensemble Verification Training Course since
at least 2010 [E]. It is also an essential component of a verification
training course run by the WMO's Commission on Climatology over the past
five years to train trainers from regional climate centres, who then
assist in training seasonal forecasters in their regions. To date
trainings have been held in China (2x), Trinidad, South Africa, Colombia,
Argentina, Kenya (2x) and the US, involving over 150 trainers [C].
Weather Roulette has also been incorporated into the Climate
Predictability Tool, a software package being used by all meteorological
services in South America and extensively in Central America, the
Caribbean, Africa, South and East Asia, and elsewhere for the production
of seasonal forecasts [C]. Equally significant, effective interest rates
(i.e. weather roulette) [6] and ignorance scores [1] have been included as
two of the seven recommended verification scores and procedures in the Guidance
on Verification of Seasonal Climate Forecasts being officially
disseminated to the 191 member countries and territories of the WMO to
guide seasonal forecasting by their National Climate Centres and regional
and local forecasters [C,F]. This guidance document is also expected to
influence the approaches utilised by both governmental and commercial
seasonal forecasters around the globe.
In addition, the UK Met Office has used the concepts of kernel dressing
and blending climatology from the underpinning research to redesign its
3-month Outlook product, which is constantly available on behalf of the
Cabinet Office to assist contingency planners across the public and
private sectors to prepare for extreme weather events and potential
emergencies [B,G].
Advancing the application of probabilistic weather forecasting
(industry)
Professor Smith and CATS have a long history of partnership with industry
in embedding the research findings in contexts where real-time forecasting
is helpful in managing uncertainty and risk. Particular attention has been
given to the challenges of constructing actionable information from raw
ensemble weather forecasts and framing this information in a format useful
for specific users. The wide range of applications has included: road
gritting, food sales, horse race-track conditions, crop forecasting,
insurance brokerage, energy trading, wind power production, hydrocarbon
exploration and production (oil reservoirs and flows), and electricity
generation. About one-third of Professor Smith's students and postdocs now
continue this applied work in organisations such as the Bank of England,
Royal Dutch Shell, the Met Office and Risk Management Solutions. Current
CATS partnerships are under way with the UK Department of Energy and
Climate Change (DECC) and the Royal National Lifeboat Institute.
The application impact has been most obvious in the energy sector. Altalo
and Smith (2002) "promoted the use of probabilistic weather forecasts in
business, thus increasing the return on investments in weather forecasting
and reducing energy production costs. The benefit of this work is still
felt today, which sees the energy sector (producers and energy traders) as
one of the main users of weather forecasts" [A]. More specifically, work
was done with EDF Energy to develop and implement improved methods for
electricity demand forecasting, which helped to reduce risk, avoid u-turn
trades, manage supply and improve performance. Metra, the global
commercial arm of the New Zealand meteorological service, also partnered
with CATS on the design and marketing of a product called Vantage to help
energy managers and traders in managing weather-related opportunity and
risk and in making operational decisions [H], which has led to a new
generation of Metra products and services in use worldwide [I].
Sources to corroborate the impact
All Sources listed below can also be seen at: https://apps.lse.ac.uk/impact/case-study/view/4
A. Letter from Head of Predictability Division, ECMWF. This source is
confidential.
B. Letter from UK Met Office. This source is confidential.
C. Letter from Chief Climate Scientist, IRI. This source is confidential.
D. Letter from former manager of BIOS Risk Prediction Initiative. This
source is confidential.
E. ECMWF training: https://apps.lse.ac.uk/impact/download/file/1573
F. IRI Guidance on Verification of Seasonal Climate Forecasts
https://apps.lse.ac.uk/impact/download/file/1575
G. UK Met Office 3-month outlook (as example of impact):
https://apps.lse.ac.uk/impact/download/file/1576
H. Metra Vantage marketing brochure https://apps.lse.ac.uk/impact/download/file/1577
I. Letter from Business Services Manager, Metra. This source is
confidential.