Summary of the impact

Effective industrial design and simulation require efficient and versatile computing systems. As a result of research performed by our team experienced in High Performance Computing (HPC), novel software structures and aligned hardware architectures have led to significant benefits to the energy supply industry and to microprocessor manufacturers.

As a result of our research with supercomputing, simulation times for electric field patterns in power components have reduced more than 30-fold, with accurate complex 3-D outputs for an increased range of configurations, thereby enabling our partner company to achieve results not possible with commercial software and to reduce product development costs by $0.5M - $5M p.a.

Our research has been incorporated by Intel into their numerical libraries and now made available to the general public supported by their latest processor architectures. Intel now has a 82% share of processors, according to the November 2013 Top500 list.

Underpinning research

Prof. Carsten Trinitis, Professor of Distributed Computing (since 2010)

Dr Mehmet Aydin, Lecturer in Computer Science

In the research described, Prof. Trinitis also collaborated with staff at the Technical University of Munich, Germany, in particular Dr Alexander Heinecke, Dipl-Math Thomas Müller, Dr Josef Weidendorfer and Dr Benoit Chaigne.

Background

Since the early 1990s, researchers led by Prof Carsten Trinitis have been engaged in pioneering fundamental research to develop and apply high performance computing techniques to both the global energy supply and medical fields, while leading microprocessor vendors have recently incorporated the results of the research group's algorithms and designs into numerical libraries on their latest processors.

In the design process for high voltage electrical components, Asea Brown Boveri (ABB) needed to simulate the field patterns for transformers and switchgear from the early 1990s, but the tools available to them at the start of their association with Prof. Trinitis' High Performance Computing group could only provide simulations for simple two-dimensional or axi-symmetric geometries. There was a strong demand for accurate, fast, three- dimensional component simulation and optimisation to reduce the overall design time and to enable earlier, lower-risk commencement of manufacture. According to ABB's corporate research centre brochure, "Experimental trials, theoretical modelling, and numerical simulations are the tools for success".

In 2010 Prof. Trinitis became Professor of Distributed Computing at the University of Bedfordshire (UoB). The work summarised below is built on work which began at the Technical University of Munich (TUM), Germany, and carried out at both TUM and UoB since 2010. Optimisation algorithms developed by Dr Mehmet Aydin [3.1] of UoB were successfully applied to and integrated into ABB's simulation system Toolbox and are now in production use.

REF period activity

The research addressed and continues to address the following questions.

1. Can the code be parallelised, to reduce simulation times significantly?
2. Can the code utilise the underlying hardware to reduce run time and provide a good match to the desired output format?
3. Can the underlying hardware architecture be made upgradable to take advantage of future developments in supercomputers?
4. Can the user interface be made user-friendly and suitable for use by non-computer scientists?
5. Can the computing process be readily upgraded as improved algorithms become available?

All these questions have been answered positively, through the following innovations.

a) As optimisation requires multiple simulation runs, novel parallel optimisation methods based on genetic algorithms were developed and applied [3.1], [3.3].

b) The code was specially redesigned to take advantage of novel hardware features at all levels: vector registers, shared memory thread parallelism and distributed memory parallelism [3.3], [3.4].

c) Ideas from upcoming hardware architectures such as 512-bit vector registers and many-core thread parallelism were integrated into the research and design process [3.2], [3.5].

d) The new features were integrated into Toolbox, which is a standard interface used by ABB engineers for carrying out simulations and optimisation runs. These features have been successfully deployed at various ABB research centres across Europe.

Throughout this work, close contact was maintained with engineers and researchers at the ABB research group in Switzerland and Sweden who trialled and deployed the research results to accelerate their simulations of electrical power grids using a multicore architecture [3.4].

References to the research

Details of the impact

Our research has provided economic and technical impact in both the energy supply industry and processor manufacturers, helping the former to carry out simulations their competitors are not capable of, saving millions of dollars per year through this, and the latter to maintain their position as worldwide market leader in x86 architectures [5.1, 5.2, 5.3, 5.4].

A Energy Supply industry

The contributions from our research group made it possible for ABB engineers to simulate and optimise entire devices for the first time in their company's history, saving a significant amount of money for development and components, and putting ABB at the leading edge of energy supply technology worldwide. Our cooperation with the group of the Senior Scientist at ABB, Switzerland, has been very successful over many years. It has significantly contributed to establishing high performance computing technology as one of the basic elements in the ABB research and development infrastructure and has had a considerable impact on many development projects involving simulation tools.

In the 1990s our cooperation efforts were focused on speeding up the ABB in-house codes for electromagnetic simulations as well as on creating an efficient computing environment. In 1999, the first Linux cluster was implemented at ABB together with Carsten Trinitis' research group. This was a long time before cluster technology became an industry standard. Today, several clusters are operated in ABB and the engineers at worldwide distributed ABB sites can run the parallel codes which are continually being optimised and adapted to latest high performance computing technology by Carsten Trinitis' team, in their on-going relationship with ABB.

During almost 2 decades of collaboration between the modelling and simulation team at ABB's corporate research centre led by their senior scientist Dr Andreas Blaszczyk and Carsten Trinitis' team, ABB has performed several projects that have been successfully been implemented as production tools and put into production use from 2010 on. The most important examples, beyond the parallel cluster environment mentioned above, are as follows.

• Creation of a parametric optimisation framework used by ABB engineers to run complex simulations in an automatic loop involving direct coupling between CAD systems and electromagnetic simulations.
• Performance optimisation of a SPICE-based solver for fast computation of transformer cooling; the new solver is used as a component of an interactive transformer design system in ABB.
• Acceleration of contingency analysis of electric power grids; increasing the efficiency of the in-house code developed by ABB Sweden running on the new multi/many core processor architectures.

According to the Senior Scientist of ABB, Switzerland [5.1], the group of Carsten Trinitis has shown not only a high level of professional skills in the area of high performance computing, but they have been adjusting and extending their own scientific goals according to the needs of industrial cooperation. This is something which ABB considers to be very positive (although not very common in the academic community), and which enables new research topics to be found that are challenging to the academics and which have practical interest and application for the industry partner. New features which were not available before have been integrated into their simulation system Toolbox, enabling them for the first time to optimise day-to-day problems by reducing runtimes from 20 seconds to 200-300 milliseconds per iteration at around 1000 iterations per optimisation run. These features are not available in commercial software, making the flexibility ABB gained through collaboration with Carsten Trinitis' team unique in the energy supply industry. According to their Senior Scientist, production costs were reduced by between $0.5M to $5M per year through this [5.1], and he plans for the collaboration to continue over the next few years at least.

B Processor manufacturers

The work done by Carsten Trinitis' team on many-core architectures has focussed on power efficiency, efficient algorithm implementation, and sparse and dense matrix solvers, and has significantly helped Intel prove and improve their many-core products. In 2009 and 2010, Intel had to make decisions on whether a many-core architecture was actually necessary. There were real questions if such an architecture would provide substantial benefits to Intel's user community. Intel selected very few sites to conduct studies on the performance and utility of their many-core prototypes. Work done by Carsten Trinitis' team on TiffaMMy was one of the first independent studies that validated what became Intel's MIC Architecture. The findings of performance and programmability provided a valuable early proof-of-concept that supported their ideas, and it helped Intel make the choice to press forward with further investments. The result was the Intel Xeon Phi coprocessor. Other feedback given through Intel's software development phase and during the first product development influenced the decision which Intel made in future Knights Landing — the first highly parallel host CPU. The work with Carsten Trinitis' team has been highly influential in Intel's product design, resulting in a 82% market share in the November 2013 Top500 list [5.2], [5.3], [5.4]. Through this share, Intel have significantly strengthened their position in high performance computing, making them the number one manufacturer of processors for supercomputing systems worldwide not only in terms of compute performance, but also in terms of energy efficiency [3.5].

The experience gained through the above mentioned research allowed Carsten Trinitis' research group to participate in the development of numerical libraries for the latest processor architectures at Intel, who have now incorporated parts of our code into their Math Kernel Library (MKL) for the Xeon Phi architecture [3.2], [3.5].

Sources to corroborate the impact

5.1 Senior Scientist, ABB Corporate Research Centre, Switzerland

5.2 Director of Marketing for Technical Computing, Intel Corporation, USA

5.3 Director Parallel Computing Lab, Intel Corporation, USA

5.4 Director of Leibniz Computing Centre, Garching bei München, Germany