Summary of the impact

This case study concerns the development and subsequent uptake of the Feature Selective Validation (FSV) method for data comparisons. The method has been adopted as the core of IEEE Standard 1597.1: a `first of its kind' standard on validation of computational electromagnetics and is seeing increasingly wide adoption in industry practice where comparison of data is needed, indicating the reach and significance of this work. The technique was developed by, and under the guidance of, Dr Alistair Duffy, who has remained the world-leading researcher in the field. The first paper on the subject was published in 1997 with key papers being published in 2006.

Underpinning research

One of the factors that has transformed Electromagnetic Compatibility from `art' to `science' is the availability of increasingly advanced numerical electromagnetics solvers and increasingly powerful platforms on which to perform the simulations. A high level of confidence in these simulations is vital, and the results can be dependent on factors such as how the model has been formulated and implemented, what simplifying assumptions have been made in the model implementation and how long the simulation has been given for convergence. Moreover, the way in which people look at the data itself can lead to a range of opinions based on the backgrounds and experiences of those undertaking the tests. So, when performing the comparison, simple binary `pass/fail' decisions are typically difficult to make. The more challenging factor is that this range of opinion is not divided amongst correct and incorrect views, but all experienced opinions are valid. Hence, any attempt to quantify comparisons should fit into a framework for a variety of opinions. A statistical approach is also unsatisfactory for many reasons.

The Feature Selective Validation algorithm was the first method to provide a quantitative comparison of general data resulting from computational electromagnetics and its comparison with measured data. The majority of the developments in this technique have been undertaken by Duffy and researchers under his supervision or as part of international networks of researchers with Duffy as the main intellectual driver for the work. The Feature Selective Validation algorithm overcomes many of the limitations of statistical approaches by relating the results of the comparison to the original data on a point-by-point basis (in addition to relating the results to natural language descriptions). It was based on a similar heuristic philosophy to the development of R-factors (Reliability Factors). Feature Selective Validation allows a single value goodness-of-fit summary, a point-by-point analysis or a direct prediction of the opinions of a population of experienced users. For example, from the original data in the figure below (measured (red) and simulated (blue)), the comparison between a visual response and FSV is shown (the x-axis in the diagram on the left represents a dimensionless normalised frequency and the y-axis is a transmission parameter in arbitrary units). While not a direct one-to-one agreement, this provides a good estimate of the population response, giving a reasonable agreement between the means and the spreads. The diagram on the right shows the overall FSV value (the Global Difference measure (GDM)) `binned' into six categories corresponding with specific FSV values compared with the visual response grouped by category (the y axis is the proportion of the total in that category).

Feature Selective Validation was designed as an heuristic to provide an analogue of the decision making of a group of experts when comparing data to validate computational electromagnetics, particularly when applied to electromagnetic compatibility (EMC) problems.

In addition to the work undertaken in Duffy's group, there are at least five other PhD students at universities in the UK and internationally associated with Feature Selective Validation developments that have sought Duffy's advice.

Feature Selective Validation results/application are becoming increasingly common in research papers. For example one paper ([3] in Section 5) concludes that `the Feature Selective Validation method adequately simulates the 'expert' opinion for all the complex data sets. More agreement between the theory and the Feature Selective Validation results would be preferred but the Feature Selective Validation method quickly compares large data sets which would traditionally take an 'expert' considerably more time.' Another paper ([4] in Section 5) applied Feature Selective Validation to validate the models used and show an improvement in the designs.

References to the research

Note: references 3 and 4 are papers of primary reference for FSV

Details of the impact

The Feature Selective Validation method represented a revolutionary new approach to solving the existing problem of quantifying data comparisons and has significant impacts upon industrial practice.

The papers published by Duffy in 2006 provided a solution to what was then believed to be an intractable problem at a time when the IEEE had established a Standards working group to look at numerical modelling validation. Duffy's Feature Selective Validation was thus adopted as the core technique to be used in IEEE Standard 1597.1 [1] - the IEEE Standard for validation of computational electromagnetics computer modelling and simulation, and its associated Good Practice Guide 1597.2.

IEEE Standard 1597.1 was published in 2008 and was the first International Standard dealing with the validation of computational electromagnetics (CEM). The standard has been taken up internationally; for example, the EU-funded project HIRF-SE (High Intensity Radiated Fields — Synthetic Environment) is a four-year, EUR 26.5m project (started 1^st December 2008), coordinated by Alenia Aeronautica SpA (Italy), involving over 40 partners from both industry and academia around Europe [2], It includes a mandate that IEEE 1597.1 should be used to demonstrate compliance within the data being used by the consortium, and hence Feature Selective Validation must be used as the validation tool.

Duffy actively pursued a policy of free dissemination of the details of the technique. This policy has resulted in two additional Feature Selective Validation software tools being made generally available from the University of L'Aquila and UPC, Spain. Duffy's group acted as an advisor to both of these additional software projects. At least 40 companies and universities have taken copies of the software from both these sources. Whilst there was no systematic tracking of users, the tools have been downloaded worldwide. The list below indicates some of these users and represents a mixture of cases where the usage is related to the standard and other cases where the FSV is being used as a non-parametric statistic. In all these cases, the information has been put into the public domain by the user through publication. Whilst details of where to access the evidence of these applications are given in section 5, some specific examples which demonstrate the reach and significance of the FSV method and the IEEE standard with users of international standing are:

• Boeing (USA) have evaluated FSV for comparing reverberation chamber measurements [3] - the Boeing Company analysed FSV as a means to compare results from different measurements and compared these results with typical non-parametric statistics used for this particular application and noted the advantage of using FSV compared to relying on expert opinion.
• Cisco Systems (USA) using Feature Selective Validation software tools to assess their simulation strategy [4] - Cisco Systems Inc used FSV to investigate the quality of their simulations and measurements for radiation from heatsinks. FSV showed that the simulations were in broad agreement, but there were notable differences with the measurements.
• The Canadian Department of National Defence are adopting Feature Selective Validation for waveform comparison [5] - the Department of National Defence, Canada, concluded that FSV can be used as indicators of convergence for iterative determination of computation parameters, with a particular focus on antenna radiation patterns.
• IBM (USA) use Feature Selective Validation for assessing modelling capability for circuit board design [6] - IBM were interested in the problem of comparing data from several simulation techniques in circumstances where reliable measurement data is difficult to obtain and to use FSV to quantify the cross-comparison of the data.
• The Southwest Research Institute (San Antonio, Texas) have used Feature Selective Validation for test site assessment [7] - the Authors needed to compare measurements with simulations as part of test site analysis. FSV provided the measure of acceptability.
• ANDRO Computational Solutions (USA) has used Feature Selective Validation for antenna co-site performance [8] - Andro Computational Solutions needed to perform sensitivity analyses for antenna placement. FSV allowed them to show the high level of sensitivity in antenna placements and identified further experimentation required.

Sources to corroborate the impact

[1] IEEE Std 1597.1, "Standard for validation of computational electromagnetics computer modeling and simulation", IEEE, Piscataway, NJ, 2008

• this is the first-of-its-kind standard to provide a means of quantifying data associated with the valiation of computational electromagnetics. Sponsored by the IEEE EMC Society, this standard relies on the FSV method to provide that quantification in a robust and objective way. For more information see http://ieeexplore.ieee.org/Xplore/home.jsp and search for "IEEE Std 1597.1". (accessed 28/08/13)

[2] the High Intensity Radiated Field Synthetic Environment (HIRF SE) research project http://www.hirf-se.eu/hirf/ (accessed 28/08/13)

• this "large scale integrating project" mandated that [1] should be used for all results processing — and hence that FSV should be used. This allowed all partners (industrial and academic) to discuss approaches, procedures and results with an objective frame of reference. The fact that this was mandated independent of the original research team indicates that FSV is becoming embedded in engineering practice.

[3] Hankins GJ & Lewis DM (2010), Validating the FSV method using reverberation chamber measurements, IEEE International Symposium on Electromagnetic Compatibility, pp 737-742, DOI 10.1109/ISEMC.2010.5711370.

• Boeing is the world's largest aerospace company operating in the civil and military realms; see http://www.boeing.com/boeing/ (accessed 28/08/13).

[4] Bhobe A & Sochoux P (2010), Comparison of measured and computed near and far fields of a Heatsink using the Feature Selective Validation (FSV) method, IEEE Int Symposium on Electromagnetic Compatibility, pp 732-736, 25-30 July, DOI 10.1109/ISEMC.2010.5711369.

• Cisco Systems are one of the world's leading IT network solutions providers; see http://www.cisco.com/ (accessed 28/08/13).

[5] Hiltz LG (2009), Characterization study of the Feature Selective Validation (FSV) technique on simple and complex waveforms, IEEE Int Symposium on Electromagnetic Compatibility, pp 268-273, 17-21 Aug, DOI 10.1109/ISEMC.2009.5284672.

[6] Archambeault B & Diepenbrock J (2010), Quantifying the quality of agreement between simulation and validation data for multiple data sets, IEEE Int Symposium on Electromagnetic Compatibility, pp 722-725, 25-30 July, DOI 10.1109/ISEMC.2010.5711367.

• IBM is a "globally integrated" business which operates in more than 170 countries; seehttp://www.ibm.com/us/en/ (accessed 28/08/13).

[7] Brench CE & Brench BL (2009), Application of the Feature Selective Validation method to test site evaluation, IEEE Int Symposium on Electromagnetic Compatibility, pp 254-258, 17-21 Aug, DOI 10.1109/ISEMC.2009.5284663.

• the lead author is with the Southwest Research Institute, Texas, According to their website, "SwRI provides contract research and development services to industrial and government clients in the United States and abroad. The Institute is governed by a board of directors, which is advised by approximately 100 trustees (...) The Ford Motor Company has designated the Institute a Tier 1 product development engineering services supplier and has awarded the Institute its Q1-2000 award." Information taken from this link: http://www.swri.org/6swsa/work/facts.htm (accessed 28/08/13).

[8] Kasperovich I, Drozd AL, Carroll CE & Croneiser AA (2010), Antenna co-site performance analysis for complex systems using Feature Selective Validation, IEEE Int Symposium on Electromagnetic Compatibility, pp 712-717, 25-30 July, DOI 10.1109/ISEMC.2010.5711365.

• AndroCS is an independent electromagnetics software developer based in Rome, NY, USA. According to their website "ANDRO Computational Solutions is a small, independently owned company. We research and develop expert system solutions in the areas of electromagnetic environmental effects (E³), spectrum management, radar systems, target recognition, data fusion, image registration, and more. ANDRO also provides E³ consulting, engineering, and technical services to defense and commercial industries." — see http://www.androcs.com/index.html (accessed 28/08/13).