Summary of the impact

Short Term Conflict Alert systems are used by NATS to alert air traffic controllers to the risk of aircraft becoming dangerously close. This research has provided the means to enhance international air traffic safety by automatically optimising STCA systems so as to simultaneously maximise the number of alerts raised in response to truly dangerous situations, while, at the same time, minimising the number of false alerts. This has been achieved by developing multi-objective evolutionary algorithms to automatically locate the Pareto front describing the optimal trade-off between the numbers of true and false positives. The optimiser is described by NATS as "an outstanding improvement to our safety" [KTP-1395, final report].

Underpinning research

The research was undertaken under the auspices of an EPSRC grant "Critical Systems and Data Driven Technology" (GR/R24357/01), which examined how safety critical systems that deal with data can make decisions and derive measures for the confidence that can be placed in those decisions. One strand of this work deals with Bayesian treatments, which average over many possible decision makers and is particularly appropriate where the safety critical system is being learned ab initio from data, e.g. [1]. The research identified another class of safety critical systems, exemplified by Short Term Conflict Alert (STCA) systems, which have pre-existing models whose parameters can be adjusted to affect the true and false positive rates.

Work at Exeter developed a multi-objective evolutionary algorithm for the optimisation of the STCA parameters to locate an approximation to the Pareto front, the optimal trade-off between the true and false positive rates [2]. At heart the automated optimisation process is a directed form of `trial and error': small perturbations are made to STCA parameters whose true and false positive rates are no worse than the rates for any other parameters found so far (so called non-dominated parameters); the performance of the perturbed parameters is then evaluated on historical data, and parameters which are non-dominated are retained. Over many generations an approximation to the Pareto set emerges.

This idea was also extended so as to maximise the warning time given to controllers, along with optimisation of the true and false positive rates [2]. The STCA system allocates pairs of aircraft trajectories to one of two classes: dangerously close and well separated. Further work showed how these methods can be extended to classification into more than two classes [3].

The optimisation is difficult because (a) air-space is segregated into several interacting sectors and (b) each evaluation is computationally expensive, taking 6 or 7 minutes. Special methods were developed to tackle these problems, enabling the impact described and contributing to research in the field of computationally expensive multi-objective optimisation [4]. As the data on which performance is evaluated is not unlimited, there is some uncertainty in the evaluation of each objective; methods were therefore developed to quantify this uncertainty [5] and to account for it during optimisation to prevent over-fitting [4].

Optimisation extracts the best possible true and false positive rates from the particular STCA architecture employed by NATS. Recent MPhil work, using data supplied by NATS, is on improving the classification architecture itself, particularly for aircraft outside controlled airspace. Everson and Fieldsend continue to collaborate with NATS, particularly on avoiding over-fitting during evolutionary optimisation, and NATS were partners in a proposal to EPSRC on surrogate modelling for evolutionary optimisation.

References to the research

References [2], [3] and [4] best indicate the quality of the research.

Details of the impact

NATS, the UK's main air traffic provider responsible for providing safe air traffic control for millions of flights each year, recognised the importance of this research by the uptake in 2008-2009 of a KTP with Everson and Fieldsend. With many thousands of aircraft in UK airspace per day and true positive alert rates of only about 70%, it is crucial to extract the optimum performance from STCA systems, which are a component of the NATS "safety net" ensuring safe air traffic management. As STCA is one of several barriers in the safety net, it is impossible to directly assess the impact of its optimisation in terms of lives saved or crashes averted, but it is worth noting that the Überlingen mid-air crash in 2002 would have been averted if the STCA system had been operational — it was switched off for software maintenance [A].

The effectiveness of the STCA is achieved by striking the right balance of true alerts to false alarms. Too many false alarms make the tool unusable, distracting and unsafe; too few true positives mean that controllers do not get the warning they need of possible losses of separation between aircraft — this is a delicate balance. Historically the optimisation of the STCA has been based on best engineering judgement about the 1500 or so parameters, which define its performance. The Exeter research provides an algorithm for automatically locating the optimal trade-off.

The KTP with NATS produced a properly software-engineered suite of tools to perform the optimisation on any STCA system, including the new ESTCA system being introduced in the UK (which has approximately 3000 parameters and thus presents an even greater challenge to manual tuning) and other STCA systems in Europe. The tools were used by NATS for operational planning and tuning STCA across the UK airspace; this is a continual process in response to changing air-traffic patterns and airspace regulations. Importantly, in addition to finding optimal parameters, the Pareto front shows NATS personnel the range of available trade-offs, information that was hitherto unknown, allowing them to make a more reasoned choice of operating point. The optimiser was used during the period 2007-2010, but due to re-organisation and changes in practise at NATS the optimiser is no longer in current use.

Work jointly written with NATS staff was presented at an international Eurocontrol conference, which is primarily a meeting of representatives of air-traffic control organisations (Eurocontrol, www.eurocontrol.int, is responsible for overseeing European air traffic). The joint work generated interest from Europe and the USA both for the operational advantages described but also for the possibility of assessing airspace reconfigurations and comparing airspaces [B]. Internationally, the optimisation-adaptation capabilities of the optimiser are being explored in Europe through the Single European Sky Air Traffic Management Research (SESAR) programme (www.sesarju.eu), where it has "enhanced NATS' innovation profile" [C, D]. The NATS representative writes in the KTP final report: "This KTP has delivered to NATS an automatic optimiser which is quicker, more rigorous and has shown to provide more optimum configurations (up to 10% better) than judgement alone. It is a tremendous contribution to NATS and to the safety of air traffic services in UK." [C]

Sources to corroborate the impact

A. German Federal Bureau of Aircrafts Accidents Investigation report AX001-1-2/02. Available from http://www.bfu-web.de

B. Reckhouse, W.J., Everson, R.M., Fieldsend, J.E., Bush, D., Arnold, T., Hayward, R. and Slater, K, 2008, "Assessment & optimisation of STCA performance: Using the Pareto-optimal receiver operating characteristic", EUROCONTROL Safety R&D seminar. Southampton, October 2008.

C. KTP 1395 with NATS En Route plc. Final Report. 2009. Available from www.ktponline.org.uk

D. Corroborating letter from Head of Operational Analysis at NATS, dated 18/11/2013.