Summary of the impact

An innovative method enabling firms to improve the accuracy of their demand forecasting has resulted from research analysing data from 70,000 company sales forecasts. It was concluded that, although judgmental adjustments to statistical forecasts were common, they often wasted management time, reduced accuracy and introduced bias. University of Bath led research, which determined how computer-based systems could support more effective forecasting adjustments, has informed the design of a new commercial product, ForecastQT^™. This product is now being marketed globally. Early applications of the product suggest estimated savings of 2% of total revenue for one multinational company and $200m for another. The research has also influenced the development of software and services for clients at SAS, the world's largest privately owned software company.

Underpinning research

The underpinning research for this case began in 2001 with experiments to determine how forecasters might be provided with tools to improve the accuracy of forecasts based on judgment. An experiment in stock price forecasting, which compared the effectiveness of outcome and performance feedback to forecasters (reference 1), suggested that outcome feedback is less effective than other forms of feedback in promoting learning by users of decision support systems. However, if circumstances can be identified where the effectiveness of outcome feedback can be improved, this offers considerable advantages.

A further study to develop this area, funded through the EPSRC and conducted jointly with Lancaster University between 2004 and 2007, investigated how companies in supply-chains made short-term forecasts of the demand for their products (reference 2). The team of researchers (at Bath: Professor Goodwin, SL since 2000, Professor since 2005) analysed over 70,000 forecasts and outcomes in a range of companies, observed forecasting meetings and interviewed forecasters. A survey of 120 mainly US-based forecasters was also conducted (reference 3). Surprisingly, this found that 25% of organizations did not measure forecast accuracy and hence had no feedback on the performance of their forecasts.

The results revealed that most companies use statistical software to produce forecasts. However, considerable management time is then spent judgmentally adjusting these forecasts to try to improve their accuracy. For example, in a pharmaceutical company, an estimated 80 person-hours of management time each month was spent in forecast review meetings (reference 4). In a major food company, over 90% of forecasts were adjusted. In all the companies studied, most adjustments were relatively small and yet these reduced accuracy. Adjustments were made because managers incorrectly perceived systematic patterns in the random movements of their sales graphs and overreacted to the latest figures, adding volatility to the forecasts. Larger adjustments led to the overestimation of future sales and excessive stockholding costs (reference 2).

Following this field-based research, Bath researchers conducted a series of international experiments, in collaboration with researchers from Bilkent University, Ankara (2003), Lancaster University (2004-2010) and the University of New South Wales, Sydney (2004-2010) to investigate how computerised forecasting systems might be enhanced to support managers. Potential enhancements that were evaluated included: providing an algorithm and a database enabling forecasters to identify the effect of past promotion campaigns which were most similar to a forthcoming campaign; restricting forecasters to large adjustments; and providing on-line advice and guidance to forecasters (references 4 & 5).

The findings of the research that underpinned the impact were that:

(1) Bias is a major problem associated with judgmental forecasts;

(2) Excessive volatility in forecasts caused by inconsistency in judgment and making judgmental interventions too frequently also reduces accuracy;

(3) No major commercial forecasting system included a facility for aiding and improving judgments, despite the widespread use of management judgment in company forecasting;

(4) A number of possible facilities suggested in the literature (e.g. restricting adjustment) would be unlikely to be effective;

(5) Providing feedback on the accuracy and biases of judgment would be beneficial if its presentation and time frame were designed appropriately.

The knowledge generated from the research findings has been used by Professor Goodwin to advise a new UK-based company, Catchbull, on the design of a forecasting support system called ForecastQT^™ (sources 1 and 2). This innovative system provides feedback and guidance to companies on the extent to which avoidable errors caused by bias and excessive variation in forecasts reduces accuracy. Significantly, the system enables the monetary costs of the resulting inaccuracy to be estimated and it also allows the nature of any problem (bias or excessive variation or both) to be diagnosed. These estimates can be made at the level of the individual product or aggregated over groups of products. An automatic detection system enables managers to pinpoint products or product groups where attention needs to be focused because forecast errors are proving to be costly, thereby allowing for more effective use of expensive management time. This is particularly important for companies that have thousands of products, for which forecasts are required on a regular basis. ForecastQT^™ is being marketed globally.

References to the research

Grant
EPSRC (GR/60198/01): The effective design and use of forecasting support systems for supply chain management, January 2004 to December 2006, £79,205. (PI Paul Goodwin). Rated as "Tending to Outstanding" in final review. (Note: This research was conducted in collaboration with an EPSRC-funded project at Lancaster for which Paul Goodwin was CI (GR/60181/01, £108,577).

Details of the impact

Research undertaken at Bath, as part of an international research team working on demand forecasting, has enabled the development of a new commercial product, called ForecastQT^™, which is of benefit (particularly, but not exclusively) to multi-national companies (Sources 1 and 2). The product, which became commercially available in 2011, significantly reduces forecasting inaccuracies and thereby creates business savings on stock costs, decreases the amount of obsolete stock and improves customer service. The financial benefits to companies are significant. For example, a company using this product has reported that: "by identifying the size and trend of forecast inaccuracies and putting a dollar value on them, we now have the visibility and a common language to fix them ... It could save us up to 2% of total revenue, amounting to £50m for the UK part of the business. Two simple strategies made possible by the tool should enable the company to save at least half of this cost" (Source 2).

The ForecastQT^™ system is a relatively large-scale installation that is designed to the requirements of individual companies (e.g. different companies will incur different costs as a result of forecast errors), rather than being an off-the-shelf product. For example, the system has been used to evaluate forecasts for a US-based consumer durables multinational with turnover of approximately $5 billion per annum. It demonstrated that current forecasts displayed a significant degree of bias and unnecessary variation and estimated that savings of at least $16 million annually (0.03% of revenue) were feasible in an enterprise-wide implementation of the product (Source 2). More generally, based on research findings and diagnostics using the system, it is estimated that 10% of total stock costs can be saved through eliminating the overstocking, obsolete stock, lost sales and poor customer service caused by inaccurate forecasts. In the case of one firm's $15 billion business, company executives claimed: "we have no bigger priority than improving forecast processes. We estimate that we can drive up to $200m of avoidable costs out of the business" (Source 2).

The Catchbull Company has identified important benefits from the implementation of the ForecastQT^™ product in the companies where it has already been used. The product has improved the accuracy of demand forecasts through pinpointing the source, nature and estimated cost of systematic forecast errors. Improved accuracy has delivered savings in inventory costs, such as capital, warehousing, depreciation, insurance, taxation, obsolescence, emergency delivery and shrinkage costs. The systematic over-forecasting of demand and resource waste arising from obsolescence of inventory has been reduced.

ForecastQT^™ is a significant, innovative product arising from research. Other commercial forecasting software products use `standard' measures of forecast performance that can, at best, only act as a proxy to the financial costs arising from these errors. ForecastQT^™ estimates and reports these costs.

ForecastQT^™ has led to changes in practice. The product has provided companies with a dynamic diagnostic tool for analysing and reporting systematic forecasting errors in an accessible form. The product can adapt and respond to changing market conditions.

ForecastQT^™ has encouraged changes in behaviour. Currently managers in many companies overreact to the most recent demand forecasting error, or use inappropriate accuracy measures or do not monitor accuracy at all. Few managers measure bias in their forecasts (reference 3). This prevents them from learning about preventable problems associated with their forecasts.

The benefits of this research have also had a wider reach in informing practice in companies. SAS, the world's largest privately held software company (Source 3) with revenue of $2.7b (2012) and 400 offices in 50 countries, has reported that it has "proved very helpful in developing and publicizing a range of novel methods and concepts in forecasting which have provided valuable to the SAS forecasting team and influenced the software and services we are able to offer our clients" (Source 4). SAS has also incorporated the key findings in one of its White Papers: "What Management must know about forecasting" (Source 5).

Sources to corroborate the impact

1. For evidence of Paul Goodwin's on-going role, see http://catchbull.com/about.
2. Testimonial letter from the Director of Product Development, Catchbull
3. www.sas.com
4. Testimonial letter from the Product Marketing Manager, SAS
5. What management must know about forecasting, SAS White Paper.
   http://www.sas.com/reg/wp/corp/17407