Summary of the impact

Companies in the manufacturing, construction, courier, freight and airline industries have made use of Photometric Stereo based imaging technology, developed by the Centre for Machine Vision (CMV) at UWE Bristol, to capture 2D and 3D data simultaneously. This has enabled them to develop new products for large volumetric measurement, characterisation of aggregates, tile quality control and automotive wheel alignment. These examples have addressed hitherto challenging tasks or have extended functionality to new or poorly structured environments, for instance: (i) the robust capture of accurate 3D data from postal packages exhibiting complex shape and coloration; and (ii) a new capability for distinction between printing and moulding defects during fast moving tile quality control — previously impossible. In all cases ultra-low-cost equipment is used — underlining the attractiveness of the techniques developed by the CMV.

Underpinning research

Research at UWE's Centre for Machine Vision (CMV) has developed photometric stereo (PS) imaging techniques that can be used in unstructured, real-world environments, away from the ordered and controlled environment of the laboratory. Three key advances of the underpinning research are as follows:

1. CMV has developed techniques to extract a "bump map" from objects — a 3D texture that can be "peeled off" a surface, allowing the colour, 3D texture and underlying surface to be isolated and inspected from any angle;
2. PS techniques from elsewhere can only image in a static way, whereas CMV's approach was the first to succeed in doing this for a moving object or one with non-ridged surfaces; and
3. others have worked in a lab setting with idealised illumination, i.e. a small specialist lighting source at a distance, whereas CMV's method has worked with cheap off-the-shelf lights placed close by.

The following people were involved: Prof Melvyn Smith (Director of the CMV); Dr Lyndon Smith (Co-director of the CMV); Dr Gary Atkinson (Senior Lecturer, CMV); Dr Abdul Farooq (Senior Lecturer, CMV); and Dr Jiuai Sun (Lecturer, CMV).

Photometric stereo (PS) for practical surface metrology, inspection and analysis was pioneered at UWE's CMV (formerly named the Machine Vision Laboratory) in the 1990s (R1). Before this, no practical means existed to capture detailed 3D topography at sub-pixel resolution to identify objects, textures or defects while camouflaged by complex surface reflectance/colour patterns, which obscure topography.

Prior work had tended to concentrate on the `reverse engineering' of 3D objects, for which the application of conventional PS per se was problematic. Early research insight at CMV by M. Smith in 1999 identified PS as advantageous for the capture of fine 3D surface features at sub-pixel resolution from predominantly planar surfaces. A key development by M. Smith, L. Smith and Farooq in 2005, Dynamic PS, served to extend the technique from static to moving scenes. M. Smith and Farooq established techniques from 2000 to 2006 that could isolate concomitant 3D and 2D surface features for inspection of fast moving decorative ceramic tiles (R2) and other natural materials, such as polished stone (R3).

In 2003 M. Smith and L. Smith also identified PS as offering a useful synergy with other established 3D imaging techniques, such as laser and stereo triangulation, where novel hybrids offered new functionality by combining the excellent high-resolution but poor low-resolution performance of PS with the complementary attributes of conventional 3D imaging. In addition, because PS allowed the appearance of surface colour to be depressed, while isolating and amplifying 3D shape attributes, the technique was identified as potentially useful in simplifying a wide range of conventionally difficult unstructured imaging tasks.

CMV was able to overcome a major obstacle to the practical application of PS: the step from a static to moving imaging technology (M. Smith in 2005), together with a number of other limiting practical issues. The solution was realised by allowing PS to operate in unconstrained moving, cluttered, changing or confined environments and to integrate the technology with existing techniques. This was first achieved by M. Smith, L. Smith and Farooq between 2000 and 2005 by introducing a patented multiplexing approach — either space, frequency or temporally based — to realise a new dynamic form of PS.

By 2005, the investigation of novel hybrid PS methods and advanced modelling by M. Smith and L. Smith culminated in a practical dynamic or moving PS demonstrator (R5). In 2007, this research established the minimum number of illuminates (six) to allow any convex object to be fully recovered, and later (by Sun and M. Smith in 2007) to new models allowing use of practical low-cost hardware, such as non-specialist illumination (R4). These findings were particularly important in widening application and in realising practical commercial applications of PS able to address conventionally challenging imaging tasks, such as the segmentation, characterisation and metrology of embedded 3D objects in complex scenes and in using PS at close range (1mm) or at a large stand-off (50m and more).

More recently, the outcomes of this earlier research in using PS to robustly capture 3D data in industrial application for metrology and complex shape characterisation have opened up new avenues at CMV exploring further advanced forms of PS and analysis methods in unstructured environments. These have been investigated in applications such as: (i) 3D facial recognition by M. Smith and Atkinson (2007 to date), where PS-derived surface normals have been shown to offer the most effective face biometric data; (ii) skin analysis and respiratory assessment by L. Smith (2008 to date); and (iii) facial expression and emotion analysis (M. Smith in 2012-13), where high- resolution 3D micro-expressions have been captured for the first time.

References to the research

Publications

Grants (reverse chronological order and showing total grant values)

Shape and outline amplification using 2.5D and 3D imaging. A new tool for covert surveillance aimed at real threat identification such as left object detection, M. Smith, DSTL, 2012, £60k

Intelligent Advertising Proof of Concept, M. Smith, TSB Smart award, 2012-13, £100k

Research and develop new 3D metrology equipment within the automotive sector, commercial, M Smith, 2011-12, £100k

Intelligent Video Surveillance KTP, A Farooq, 2010-13, £215k

Skin Reflectance and Face Shape Estimation Using Photometric Stereo (PhotoSkin), G Atkinson, EPSRC, 2010-13, £127k

Novel non-invasive assessment of respiratory function (NORM), L Smith, NIHR i4i Stream 1, 2008-10, £125k

Use of 3D Facial Asymmetry in Better Diagnosis and Treatment of Plagiocephaly, M Smith, MRC, 2008-9, £120k

Innovative Technology for the Quality Control of Specular Ceramic Materials using a Specular Photometric Stereo Sensor, M Smith, Great Western Research and Surface Inspection Ltd, 2007-10, £75k

Face Recognition using photometric Stereo (PhotoFace), M Smith, EPSRC, 2007-10, £672k

Application of Photometric Stereo in Dermatology (PhotoDerm), M Smith, 2007-10, DTI, £647k

Stealthy Object Detection and Recognition, M Smith, 2007-8, MOD, £154k

Visual inspection of polished stone, M Smith, EU 5th Framework, 2003-5, €1.3 million

Petroscope: visual 3D characterisation of aggregate materials, M Smith, DTI Eureka Initiative, 2002-4, €1.43m

Photometric stereo for tile inspection, M Smith, EPSRC and Surface Inspection Ltd, 2000-4, £169k

Irregular object metrology, M Smith, Commercial, multiple projects 2000-13, £119k

Details of the impact

The research developments that took place at CMV attracted much interest from companies in the manufacturing and construction sector. This led to a range of commercial applications in: (i) ceramic tile manufacture inspection; (ii) polished stone defect inspection; and (iii) 3D characterisation of aggregate materials. All of these followed from CMV's work on evolving static PS to dynamic PS. Further application has been in: (iv) 3D automotive wheel alignment; and (v) the segmentation and characterisation of large objects. Both these followed from CMV work in isolating 2D and 3D data, as detailed below.

One of the companies that approached the CMV was Quantronix Inc, an American company operating globally, specialising in object metrology for freight forwarding. In 2000 the company funded the CMV to develop an Overhead Dimensioning System (ODS — see right) able to rapidly acquire 3D data from any object(s) of up to 3m x 3m in size. CMV researchers developed the system using the PS method (hybrid) and in 2001 the innovation was protected and a patent application filed by the company. In 2004, a full prototype ODS was realised with a commercial product, known as the CubiScan 1000-VS, becoming available in 2009.

First sales were made to an international freight forwarder and, since then, over 40 systems have been sold in the US and in other countries, including ten to FedEx and one to our own MOD at $50k each, with a total value of over $1m. A range of benefits were delivered to their customers, through new capability and more efficient processes. For instance, courier, freight handlers and airlines customers, including FedEx, have used it to measure the sizes of packages automatically, enabling them to achieve efficiency gains by reducing both time for measurement and previous losses due to inaccurate manual measurement, offering an average 3-week payback (S1) and other benefits such as reduced fuel consumption, pollution and packaging materials.

Government facilities (MOD) and retail customers have been able to optimise the use of storage space, handling assets and processes, with one customer reporting that "one of the CubiScan 1000-VS machines they purchased... generated over $1 million additional revenue for the company in one year" (S1). The impact of the ODS has been continuous since its availability and has resulted in broadening Quantronix's in-house knowledge base to include vision-based technology. The company has now produced a business case for developing a countertop device based on CMV PS vision technology (S1). For this, they returned to CMV in 2012/13 to undertake the necessary R&D incorporating some of the latest hybrid PS approaches and a new system developed by CMV is now scheduled for release this year.

CMV's leading research in 3D metrology attracted a new collaboration from a company in India in 2011 — Precision Testing Machines Pvt Ltd (PTM) — which operates in the automotive sector and is the lead supplier of sophisticated automotive diagnostic and body shop equipment manufactured in India. A non-contact optical measuring device for automatic automotive wheel alignment was developed by CMV, minimising subjectivity, operator error and allowing faster throughput. In 2012, the demonstrator was handed over to PTM engineers to develop the system to commercial requirements. [text removed for publication]

Another application has been that of the Dynamic PS prototype developed by CMV in 2005 for Surface Inspection Ltd, a global company which specialises in machine vision systems and automation equipment for the ceramic tile industry. The system developed was used to detect, isolate and characterise concomitant 2D printing (including colour and glossiness) and 3D moulding defects on complex surfaces with deeply relieved topographic features and is currently in use for quality control in ceramic tile manufacturing worldwide (S3). [text removed for publication]

The final application has been the Petroscope system developed by CMV for Petromodel, a knowledge-based company based in Iceland, providing software and high-technology testing instruments for quality and process control in the aggregates industry. The Petroscope replaces labour intensive techniques and is shaping the inspection methodology used by the construction mineral industry (S4). A patent for this work was filed in 2004 and granted in USA and Eurasia in 2012 and in China in 2013. This system has been demonstrated in a commercial environment for petrographic analysis, including the characterisation of the complex 3D morphology of aggregate particles using techniques developed as part of CMV's earlier work in 3D face recognition, with use in the construction sector and benefiting companies producing gravel and crushed rock for asphalt, concrete, and railways. The system is currently being marketed and a new version including composition (PETROSCOPE 4D) has been under test by the Austrian railways (Jan — Aug 2013) (S4).

Sources to corroborate the impact

Testimonials listed below are available from UWE, Bristol, and numbered on REF system in same order as below.

S1 Testimonial from CEO of Quantronix Inc, USA. Large object metrology — corroborates customers' benefits and device sales to date, including use by courier, freight handlers and airlines

S2 Testimonial from Managing Director of Precision Testing Machines Pvt Ltd, India. Corroborates benefits and sales of device developed by CMV, UWE, that automatically measures the alignment of car wheels.

S3 Testimonial from Chief Technical Officer of Surface Inspection Ltd, UK. Corroborates benefits in surface analysis for tile inspection.

S4 Testimonial from Founder and Managing Director of Petromodel, Iceland. Morphology of particles in petrographic analysis — corroborates benefit in new technology for petrographic analysis.