Summary of the impact

The Geometric Modelling and Pattern Recognition (GMPR) Group at Sheffield Hallam University (SHU) has developed and patented internationally-known line projection technologies for fast 3D scan, reconstruction and recognition. Three types of impact can be identified: (i) through our patents, we have licensed to companies in Europe and the USA; (ii) these technologies are being transferred to Small and Medium-sized Enterprises (SMEs) across Europe, through the European funded MARWIN and ADMOS projects; and (iii) social and cultural impacts are evidenced by the 3D scanning of representative items from the Museums Sheffield Metalwork Collection which have been made publicly available on the web, and through the `Man of Steel' community project where a landmark sculpture will form a gateway to South Yorkshire and the Sheffield City Region.

Underpinning research

Optical techniques for 3D surface reconstruction include active methods requiring controlled light to be projected onto the scene or passive methods such as stereoscopy, monocular Shape-from-X (SfX) and Simultaneous Localization and Mapping (SLAM). There are three themes to the GMPR research using active methods: (i) the development of unique structured light techniques for 3D scanning and reconstruction, (ii) pattern recognition using the acquired 3D data, and (iii) developing mathematical models for 3D data compression. The technologies were entirely developed within Sheffield Hallam University (SHU) by Professor Marcos A Rodrigues (SHU 2001- 2003 Reader; 2003-present Professor) and Dr Alan Robinson (SHU 2007-2009 Senior Lecturer; 2009-present Principal Lecturer), mainly employing internal funding, but also benefitting from financial support from JISC [Grant 3], marketing and financial support from the previous Regional Development Agency, Yorkshire Forward (2005-2008), commercial licensing [Licence 1], and European funding [Grant 1, 2]. At the start of this research, in 2001, the state of the art in 3D scanning using structured light were single stripe systems or coded light patterns. The research gained momentum after 2004 with our patented uncoded multiple line projection technique [Ref 1]. The distinguishing feature of this new technology is speed and accuracy as it can provide accurate real time 3D reconstruction from a single 2D image using uncoded structured light and it can operate in real time regime both in the visible, and near-infrared, spectra.

The latest (2011) GMPR scanner has a resolution of 0.25mm in the horizontal (4 vertices per millimetre) and 0.5mm in the vertical direction (2 vertices per millimetre). This has been possible because we were able to solve and refine (2004-2011) a fundamental problem: reliably to detect and index stripes in the scene. The counterpart to the indexing problem in passive methods is the stereo vision image correspondence problem. However, the GMPR technology is superior to stereo vision in relation to speed and accuracy, as it can perform 3D reconstruction in 40ms and it is much more reliable especially for smooth or featureless objects - a well-known issue where stereo vision fails. There is a strong continuing interest in active structured light technology as evidenced by the Kinect box from Microsoft using coded structured light. Our technology has greater accuracy and measurement density than Kinect, making it appropriate to applications in the medical, industrial inspection, quality control, and security domains.

Our research was partially guided by the National Science Foundation Face Recognition Grand Challenge set in 2005 to improve 2D face recognition success rates by using 3D face measurements. Our accurate scanned facial data allowed us to pursue original solutions and 2D- 3D eye detection algorithms such that a 3D mesh of the face could be recognised. This involved developing unique methods and algorithms for 2D image and 3D mesh processing including automatic 3D pose normalisation, automatic feature extraction and cropping of the face region in 3D space [Ref 2]. Fast eigenvector decomposition methods were developed for 3D face recognition with high degree of accuracy [Ref 3, 4]. We demonstrated (in 2010-2011) real time processing by showing that in just over one second the following tasks could be accomplished: face tracking with superimposed eye detection, the 2D image being automatically taken according to some predefined constraints of face size and pose, 2D image processing with median and weighted mean filter, uncoded line detection and indexing, point cloud reconstruction in 3D, mesh triangulation, automatic 3D pose normalisation and feature detection, and recognition from a database.

3D data files are normally large as it is necessary to represent the geometry and the connectivity of the mesh. There is a strong requirement for 3D data compression using mathematical modelling for improved database performance, network transmission, remote processing and visualisation. The emphasis of current research is on 3D mesh compression methods [Ref 5], and on developing solutions for robotics [Ref 6], and medical engineering [Ref 7]. Standard approaches to 3D data compression are focused on encoding the connectivity of the mesh with geometry as a dependent property. We took the inverse approach of encoding the geometry having connectivity as a derived property. Novel compression methods based on partial differential equations (PDEs) were demonstrated by iteratively solving Laplace's equation over the 3D mesh domain expressed by an elliptic PDE. We showed (in 2012) that PDE surfaces are appropriate to represent and unpack large data files yielding compression rates of over 97% (typically from 17MB to 0.45MB).

References to the research

Patents on the 3D technologies:

Selected publications (available on SHURA from Sheffield Hallam University):

Research Grants:

[Grant 1] MARWIN A Cognitive Computer Vision Based Welding Robot, EU Grant Agreement FP7- SME-2011-286284, Research for the Benefit of SMEs, from Nov 2011 to Oct 2013, with 7 partners across Europe. Funded value 1,108,800 Euros. http://www.marwin-welding.eu/

[Grant 2] ADMOS Advertising Monitoring System Development for Outdoor Media Analytics, EU Grant Agreement FP7-SME-2012-315525, Research for the Benefit of SMEs, from Sep 2013 to Aug 2015, with 6 partners across Europe. Funded value 967,636 Euros. http://admos.eu/

[Grant 3] JISC e-Content Programme, "Rapid 3D Digitization of Sheffield Metalwork Collection", in collaboration with Museums Sheffield, from March to August 2011. Value of project £118,954. http://www.jisc.ac.uk/whatwedo/programmes/digitisation/rapiddigi/metalwork.aspx

Licensing agreements of GMPR technologies to industry:

[Licence 1] SHU-Adatis Licence Agreement, between Sheffield Hallam University and Adatis GmbH & Co. KG (Nurnberg, Germany), 01 July 2009, giving non-exclusive rights to the 3D technologies to develop products for the security market. Amended by a Supplemental Agreement of 9^th Feb 2011 to include GMPR 2D face recognition technologies. £40,000 access fee plus share of royalties. The signed licensing agreement SHU-Adatis will be provided upon request.

[Licence 2] SHU-Polyskopos Licence Agreement, between Sheffield Hallam University and Polyskopos Inc, 16 Jan 2012, giving exclusive rights to the 3D technologies to the USA market. The signed licensing agreement SHU-Polyskopos will be provided upon request.

Details of the impact

(i) Commercial licensing to companies
The GMPR research-based technologies have been patented and licensed to companies in Europe and in the USA. Non-exclusive licensing agreements with Adatis GmbH of Nurnberg, Germany (July 2009 and Feb 2011) has enabled that company to develop a line of access control products using both 3D and 2D face recognition algorithms developed at SHU (the Face Entry line of products in [Source 1]. In particular, we provided close assistance to Adatis on porting the algorithms to their hardware. Furthermore, a number of 3D algorithms were customised to the characteristics of their processors. The licensing agreement was based on an access fee of 40,000 GBP for the source code, plus a share of the royalties on a sliding scale for all Adatis products and derivative products using algorithms and methods developed by GMPR. The licensing agreement with Polyskopos Inc San Jose, California (Jan 2012) follows on the same lines but this is exclusive to the USA market. The access fee to Polyskopos was stipulated at 50,000 US Dollars plus a share of the royalties from all products and derivatives containing or using original or modified GMPR algorithms. Polyskopos' business plan is ambitious, targeting various sectors including medical and entertainment markets. We also signed a technology cooperation agreement with xCAD Solutions GmbH, Leoben, Austria (Aug 2012) to develop 3D scanning solutions for the furniture industry. Key factors are our fast and accurate acquisition technology, our pattern recognition algorithms, and methods for face recognition in 3D and in 2D.

(ii) Transferring knowledge to SMEs
The GMPR research outcomes are being transferred to SMEs across Europe through the EU funded MARWIN project (FP7 Research for the Benefit of SMEs 2011-2013, 7 partners [Source 2]. The MARWIN project provides a cognitive 3D based vision system for robotic welding tasks, in which welding parameters and robot trajectories are calculated directly from CAD models. This is a revolutionary concept in robot welding tasks designed to increase overall productivity and quality of welding assemblies. The GMPR technologies are a critical component of the MARWIN system and several alternative designs of a 3D vision system have been developed. The SMEs in the project are from Spain, The Netherlands, Bulgaria and Hungary. They own the rights to commercialise the 3D technologies within the MARWIN solution while SHU profits from background IP royalties and also through exposure to such markets. The EU funded ADMOS project (FP7 Research for the Benefit of SMEs 2013-2015, 7 partners) is another vehicle transferring GMPR technologies to SMEs across Europe. The aim of the project is to provide intelligent analytics on outdoor media by analysing and categorising passers-by. ADMOS tracks people and determines their approximate age and gender and whether or not they have noticed the advert. GMPR technologies on real-time detection and tracking, recognition and depth estimation are critical to the project. There are four SMEs on the project from Belgium and The Netherlands, Spain, and Hungary. They own the rights to commercialise ADMOS technologies while SHU will benefit from background IP licensing.

(iii) The social and cultural context of our research
With regard to impacts on society and culture, the JISC e-Content Programme provided funding in 2011 for the 3D scanning of Museums Sheffield Metalwork Collection [Source 3]. This collection has a designated status, meaning it is internationally important. Scanning and modelling the metallic objects was extremely difficult due to their shininess and complex surface shapes [Source 4]. Our research on structured light methods using both multiple and single stripe scanning was crucial in overcoming these problems. This project produced a digital record of the collection, thus helping wider understanding of the city's contemporary and historical contribution to the metalwork industries. The 3D models are universally accessible through standard web browsers located on the Museums' Sheffield website [Source 5]. The `Man of Steel' will be a `made in Sheffield' 30m tall stainless steel landmark and visitor centre for the Yorkshire and Sheffield region celebrating the community's connections with the steel industry [Source 6 - 9]. In 2012 GMPR technologies were used to scan a model of the sculpture to a high resolution of 4 vertices per millimetre. The scanned model has over 2 million faces. It has been a critical tool in allowing architects to place it within virtual models assisting visualisation of the sculpture's final appearance and also in helping to obtain the required planning permissions. Mehdi Sculptures Ltd [Source 10] stated that: `This [3D scanning of Man of Steel sculpture] has been handled with considerable skill, resolving many issues along the way and achieving a result that will have an immediate effect in many areas including design, planning and engineering.'

Sources to corroborate the impact

[Source 1] Adatis security applications (signed agreement SHU-Adatis provided on request) http://www.adatis.com/index.php?language=deutsch&content=produkte&sub=_inoutdoor

[Source 2] MARWIN A Cognitive Computer Vision Based Welding Robot FP7 Research for the Benefit of SMEs http://www.marwin-welding.eu/

[Source 3] JISC Rapid 3D digitisation of Sheffield metalwork collection http://www.jisc.ac.uk/whatwedo/programmes/digitisation/rapiddigi/metalwork.aspx

[Source 4] 3D Scanning of Highly Reflective Surfaces: Issues on Scanning the Museums Sheffield Metalwork Collection, 2012. http://representingreformation.net/marcos-a-rodrigues-and-mariza-kormann/

[Source 5] Objects in 3D at Museums Sheffield can be accessed at http://www.museums-sheffield.org.uk/collections/objects-in-3d/

[Source 6] Man of Steel web site: `Sheffield Hallam University have offered invaluable assistance to our project, providing crucial 3D data that will be used to model a full scale version of the figure. The 3D information will also form the basis of a full planning application.' http://www.yorkshireicon.com/index.php?option=com_content&view=category&layout=blog&id=20&Itemid=180&limitstart=15

[Source 7] Man of Steel website acknowledges Sheffield Hallam University technologies http://yorkshireicon.com/index.php?option=com_content&view=article&id=46:news-item-1&catid=20:latest-news&Itemid=180

[Source 8] Yorkshire's 'Man of Steel' gets bigger thanks to hi-tech manufacturing skills. The Northerner Blog, published 16 Jan 2013. http://www.guardian.co.uk/uk/the-northerner/2013/jan/16/sheffield-yorkshire-boeing-man-of-steel-rotherham-engineering-design-sculpture, also Scanning used in 30m sculpture creation, published 16 Jan 2013. Also http://www.sparpointgroup.com/News/Vol11No2-Scanning-used-in-30m-sculpture-creation/ 3-D SCANNER HELPS CREATE MAN OF STEEL, published 16 Jan 2013. http://www.vision-systems.com/articles/2013/01/3-d-scanner-helps-create-man-of-steel.html

[Source 9] AMRC creates new industrial icon for Sheffield, published 16 Jan 2013. http://www.pandct.com/media/shownews.asp?ID=35456

[Source 10] Mehdi Sculptures Ltd thanks Sheffield Hallam University, Man of Steel Website, http://yorkshireicon.com/index.php?option=com_content&view=article&id=71:sheffield-hallam-university-3d-support&catid=2&Itemid=251