Summary of the impact

Researchers in the Department of Mathematics at Swansea University have developed novel geometric methods for image processing, feature extraction and shape interrogation. The research has delivered commercial and clinical impact in a variety of settings, ranging from new water marking techniques to improve piracy detection in the film industry, to medical research investigating the replacement of traditional CT scans with safer MR scans. The research has also delivered an automatic feature and gap detection tool that has been successfully applied to aircraft data files provided by BAE Systems. A consultancy company is exploiting the methods and a licence for the commercialisation of the technology is in process.

Underpinning research

The research underpinning this impact case is based on compensated convexity theory, a powerful mathematical tool discovered by Kewei Zhang (Professor of Mathematics at Swansea University until December 2012), which introduces a new class of geometric convexity-based transforms. The initial paper on this work, that provides the foundation for all later developments, was published in a highly-regarded pure mathematics journal in 2008 [R1].

Compensated convexity theory provides a geometric convexity-based tight-approximation method for general functions [R1, R2]. This can be used, in particular, as a new way to identify singularities in functions, and thus can be exploited, via a numerical implementation of the transforms, to detect features in images or data or remove noise from images.

Based on this insight, innovative geometric methods for image processing, feature extraction and geometric interrogation have subsequently been developed [R3-R6]. Key advantages of this new geometric approach over previous image and data processing techniques include its use of blind global methods which are stable under perturbation and different sampling techniques, and also provide scales for features that allow users to select which size of feature they wish to detect.

Applications for the numerical implementation of the new convexity-based tight-approximation methods to perform specific image and data processing and feature extraction tasks have been developed in collaboration with organisations such as BAE Systems, the John Radcliffe Hospital, Oxford, and Fortium Technologies since late 2008 onwards. The research has been conducted in conjunction with parallel work in the theory of geometric singularity extraction.

Applications include image processing, reconstruction based on scattered data such as impulse noise removal, image repair, affine ridge, valley and edge detections, corner detection, sharp turning point and intersection detection for lower dimensional objects, end point and boundary detections for manifolds and curves, multiscale medial axis, boundary detection of objects based on scattered samples, high oscillation area detection, outlier enhancement and suppression, and geometric watermarking.

Following the initial work of Zhang, the further development of the theory, its applications to image and data processing, and the associated numerical methods and software, has been carried out by a research group consisting of Professor Kewei Zhang, Dr Elaine Crooks (Associate Professor in Mathematics, Swansea University) and Dr Antonio Orlando (Lecturer in Engineering at Swansea University until September 2010), between late 2008 and the present. Two short-term research assistants (Dr Yasmin Friedmann, March-June 2012, July-August 2013, and Ms Natalia Ubilla, May-June 2012) were also employed by Swansea University to assist with this project.

Two substantial research papers on the theoretical aspects of geometric singularity extraction have been submitted for publication [R3, R4] and further research papers are in preparation; these papers collectively develop the theoretical framework that underpins all of our applications to image and data processing and feature extraction tasks. A UK patent application on these new methods of image and data processing (GB 0921863.7) was filed by Swansea University, with inventors Zhang, Crooks and Orlando, in December 2009, and a PCT (Patent Corporation Treaty) application (WO 55010) was filed in December 2010 [R5]. The inventors entered the UK national phase of the patent application process in June 2012.

References to the research

Publications and submitted articles (R1, R2 and R5 best show the quality of the research):

R5) Patent application: Image Processing and Feature Extraction — UK patent application ((GB 0921863.7) December 2009, PCT (Patent Corporation Treaty) application (WO 55010) December 2010. Application for the UK National Phase, June 2012.

Grant awarded: EPSRC `Pathways to Impact' grant (£25,070 — Development of tools in image processing, feature extraction, approximation and interpolation, and shape interrogations in computer aided geometric design), awarded to Kewei Zhang via the College of Science Research Committee, Swansea University, which funded the employment of two research assistants in the period March-June 2012 to develop a user interface and webpage to aid demonstration of the image processing software to industry, etc. This work was important in underpinning impact d).

Following the initial identification of the potential impact of this research in late 2008/early 2009, the Swansea University Department of Research and Innovation funded the patent and PCT applications (2009 and 2010, respectively), the Wales Institute of Mathematical and Computational Sciences (WIMCS) and the Department of Mathematics, Swansea University, allowed Kewei Zhang and Elaine Crooks to spend time developing the theory and numerical methods for applications, and the Department of Mathematics also provided computing equipment and software.

Details of the impact

The case described here centres on the impact of our new technology on the business sector.

a) Improving piracy detection for the film industry [C1]

The geometric techniques developed at Swansea are being used as a tool for inserting robust watermarks in individual frames of a film in order to improve piracy detection in the film industry. Fortium Technologies Ltd were impressed by a test example of using the technique to embed a text-based watermark in an X-ray. Piracy is a major commercial problem for film companies, costing the US economy alone over $20 billion per year^#, and there are currently only a couple of companies worldwide involved with film watermarking. There is thus a strong interest within the film industry to find alternative approaches; Fortium is seeking to develop a method of embedding a watermark in each frame of a film so that, if the film is pirated, information about when or where the piracy took place can be obtained.

Zhang, Crooks and Orlando have subsequently entered into a formal contract with Fortium to develop their geometric method to meet the specific needs of film watermarking. Trials of embedding a watermark into a test image provided by Fortium have shown that our approach allows embedding of a watermark in a localised way that gives a high PSNR (peak signal to noise ratio) that is well above the threshold that the industry accepts as being enough to ensure the watermark will be imperceptible to the viewer: the PSNR for our locally embedded watermark is 66dB and the industry threshold is around 45dB. Our technique allows watermarks that are either image or text based, and can be incorporated anywhere within the image frame, both of which are seen as key advantages in comparison with existing watermarking techniques. The inclusion of text-based watermarks allows the possibility of human-readable watermarks. Current development is focussing on trying to improve the robustness of our watermarking approach.

"...the project has already enough promise to get some of the biggest content owners and producers in the movie and television industry eager to monitor its progress" (Fortium CEO)

b) Feature and gap detection for computer aided design [C2, C5]

Working with BAE Systems, the Department has provided a confidential report concerning the extraction of intersection and high curvature parts and gaps for geometrical objects based only on given loosely sampled point clouds defining the surfaces of the object. Following a visit of Zhang and Crooks to BAE Systems Advanced Technology Centre in Bristol in June 2011, including a well-received presentation on some of our new methods and subsequent discussions, we were provided with some data files of surface meshes for the surfaces of aircraft, to which our intersection/high curvature and gap detection methods were successfully applied.

One application of our gap-detection method is as an automatic tool to find gaps between parts of the underlying geometric design of an aircraft, for instance in data files provided by manufacturers to engineers for the purpose of performing fluid-dynamics simulations. These gaps are deliberately left for soldering purposes, whereas for fluid dynamics simulations, different parts must be connected. Currently such gaps are detected by a time-consuming manual process, while our methods provide a fast, automatic gap-detection tool. Work is ongoing in the field of application.

c) Quantitative comparison for medical images [C3, C6]

A further confidential report was provided in June 2012 to clinicians from the John Radcliffe Hospital, Oxford, who are investigating the feasibility of replacing traditional, benchmark computerised tomography (CT) scans of children's skulls, known to be accurate but also to impart high levels of radiation, by safer magnetic resonance (MR) scans. The report was concerned with quantitative Hausdorff-distance measurement between image sets based on magnetic resonance (MR) and computerised tomography (CT) scans of a phantom box, with the aim of providing a quantitative method of comparing CR and MR scans.

d) Commercial potential identified by consulting company [C4]

After a demonstration in January 2013 of a selection of our image processing and feature extraction methods using the interface initially developed with the EPSRC `Pathways to impact' grant described above, the consulting company Cadarn Technik, which has extensive experience of dealing with video algorithms and of interacting with electronics, chemical, materials, and life science companies, expressed an interest in commercialising our technology. A draft of a licence for the purpose of commercialisation of this technology is in process.

^# http://www.theguardian.com/film/2012/may/11/release-date-piracy-time-warner

Sources to corroborate the impact

External contacts:

[C1] CEO, Fortium Technologies Ltd

[C2] Communications, Networks & Image Analysis, BAE Systems

[C3] Nuffield Department of Surgical Sciences, University of Oxford, and Department of Maxillofacial Surgery, John Radcliffe Hospital, Oxford

[C4] Director, Cadarn Technik Ltd, Dylan Thomas Centre, Swansea, SA1 1RR Confidential reports to external bodies:

[C5] `Shape Interrogation of Some BAE Systems Geometric Models', provided to BAE Systems Advanced Technology Centre (October 2011)

[C6] `A Comparison of 3D Reconstruction from MR and CT Scans of a phantom Box', provided to Nuffield Department of Surgical Sciences, University of Oxford, and Department of Maxillofacial Surgery, John Radcliffe Hospital, Oxford (June 2012)