Summary of the impact

A new multi-purpose computer vision system to identify sub-standard food products has been created. The research developed a user-trainable software technology with a range of possible applications, thus overcoming the specificity and other limitations such as the high set-up cost of existing visual inspection systems. This research is achieving impact in several areas within the food industry, including quality analysis of fresh produce, food processing and food packaging. The technology is currently being trialled at the leading post-harvest applied research facility for agricultural storage in the UK, and is also being licensed to a world-leading supplier of food packaging machines and equipment for inclusion in a new product range under development. The longer-term impacts include safer food, reduced food waste, more efficient food production, and better use of natural resources (e.g. reduced use of water, pesticides and other inputs), through early detection of potentially harmful flaws in production and packaging.

Underpinning research

Trainable Anomaly Detection and Diagnosis (TADD) is a system for automatic visual inspection of food products. The underpinning research involves computer vision and machine learning algorithms that automatically learn salient image features (e.g. colour and texture) to differentiate between expected and anomalous states of a given product. To cope with the natural variation in food products and to maximise the range of applications to which TADD can be employed, the system is trainable - meaning that a non-expert operator is able to quickly set-up the system to distinguish visible properties of the objects of interest: for example, blemishes in potatoes, insect damage in beans and faulty seals in food packaging.

While automated inspection systems exist already, they are limited by their specificity, because the image features for recognition have to be selected by the trained engineer to work with a specific configuration of product, imaging system and environmental conditions (e.g. lighting and background conditions). Such systems do not generalise well to other configurations, where the required image features may differ from those used to design the original system. The currently deployed systems also require manual calibration and have limited accuracy.

The original research was carried out as part of a project on detection and identification of blemishes in potatoes (co-funded by EPSRC and the AHDB Potato Council, 2006-10), in collaboration with Sutton Bridge Crop Storage Research. In this work, machine learning classifiers were trained to classify features in images of potatoes according to categories including blemish versus non-blemish (blemish detection), and further into sub-categories corresponding to particular potato blemishes (blemish identification). The contribution of this work included automatic selection of colour and texture features that best discriminate the classes of interest, using a learning algorithm based on boosted classifiers [1,2].

A second application area was explored in a Defra-funded project (2011), with the objective of reducing food waste, using different sensors: detection of faults in heat-sealed food packaging, using polarised stress images and laser scatter images. This work demonstrated that the approach achieved high accuracy in detecting faulty pack seals using both modalities, and that the core algorithms could be generalised to work across diverse applications [3].

A follow-up project developed a real-time prototype system (funded by a HIRF Innovation Fellowship, 2011-12), to demonstrate the technology to a wider industry. This work included developing an intuitive graphical user interface (GUI) and massively accelerating the performance of the basic technology using heterogeneous computing, combining multi-core processing and graphics processing unit (GPU) technology, together with various algorithmic improvements, so that the core algorithms now run at 30 fps [4, 5] - normal video rates.

Laboratory testing of the prototype system was conducted at the AHDB Potato Council's Sutton Bridge Crop Storage Research (2011), demonstrating the ability of the technology to detect major causes of blemishes, defects and diseases affecting the UK potato harvest. Further industrial tests were conducted in the Quality Control department of Branston Ltd. (potato packers and distributors, and suppliers of around 60% of potatoes sold by Tesco) in late 2011/early 2012.

Ongoing research and technical development is now extending the basic approach beyond colour and texture features in 2D images, to include shape features and 3D imaging, as well as further applications in food processing and packaging (from Feb. 2013, TSB-funded CR & D project with industrial partners Ishida Europe Ltd., Branston Ltd. and the AHDB) and detecting damage in field beans (from Oct. 2013, TSB-funded CR & D project with the Processors and Growers Research Organisation (PGRO) and Frontier Agriculture Ltd.).

Key researchers (All based at University of Lincoln unless stated otherwise)

1. Prof. Tom Duckett (Principal Investigator, 2006 - present)
2. Michael Barnes (PhD student, Jun. 2007 - May 2012, also Research Fellow on Defra LINK Project AFM284/FT1579 from Jan.-June 2011)
3. Jamie Hutton (Research student Jun. 2011-- Jun.2012)
4. Dr. Grzegorz Cielniak (Jun. 2007 - May 2012)
5. Dr. Glyn Harper and Dr. Graeme Stroud (Plant pathologist, AHDB Potato Council, Sutton Bridge Crop Storage Research, collaborator on ESPRC Industrial CASE studentship, HIRF Innovation Fellowship and TSB-funded Technology Inspired CR&D - ICT Project 2007 - present).
6. Mike Dudbridge (Principal Lecturer, University of Lincoln Holbeach Campus, collaborator on Defra LINK Project AFM284/FT1579, Jan. - June 2011)
7. Dr. Ran Song (Research Fellow, July 2013 -), TSB-funded Technology Inspired CR&D - ICT Project.
8. Dr. Hossein Malekmohamadi (Research Fellow, Nov. 2013 -), TSB-funded Technology Inspired CR&D - ICT Project.

References to the research

References

Key grants

• EPSRC Industrial CASE Award, "Quantitative estimation of blemishes in potatoes using machine vision", £81k, 2006-10 (PI: T. Duckett, co-funded by the Potato Council).

• DEFRA AFM-LINK Award: "Monitoring of heat sealed packaging using thermal and multi- spectral imaging - a pilot study", £60k, 2011 (Co-PIs: T. Duckett and M. Dudbridge).

• HIRF Innovation Fellowship: "Low-cost automatic identification of potato blemishes", £16k, 2011-12 (PI: T. Duckett, funded by EMDA and European Regional Development Fund).

• Technology Strategy Board funded Technology Inspired CR&D - ICT Project: "Trainable vision-based anomaly detection and diagnosis". Total value £823.3k (Project lead: Ishida Europe Ltd. PI for University of Lincoln: T. Duckett. TSB grant to University of Lincoln: £355k).

• Technology Strategy Board CR&D Competition on Measurement Technologies for Efficient Agrifood Systems: "Novel computer vision techniques for food quality analysis - identification of Bruchus rufimanus (bean seed beetle) damage in field beans (Vicia faba) for export for human consumption". Lead organisation: PGRO (contact: Rebecca Ward). PI for UoL: G. Tzimiropoulos, Co-Is: G. Cielniak and T. Duckett. Value to UoL: £24k. 2013

Patent application

• UK patent application 1120865.9 | P50316GB. Automation of Image Analysis. Inventor: T. Duckett.

Details of the impact

Impact on Quality Control of Fresh Produce
Potatoes (Solanum tuberosum), with an estimated worldwide production of over 300 million tonnes in 2005 (Food and Agriculture Organisation, 2005), account for 70-80% of the carbohydrate consumed in the UK. For the fresh market the main factor affecting consumer preference is physical appearance and, to maximise return, great effort is expended ensuring that the appearance best matches a particular market. There are no current legislation standards for tuber blemishes; however such standards are driven by market forces, principally by the larger supermarkets. Most potatoes are still sorted by hand with the associated problems of variable subjectivity, operator fatigue and high cost of human inspectors, Currently deployed artificial vision systems require frequent manual calibration and have limited accuracy and utility. Hence there is great potential for the TADD technology to achieve worldwide impact over the coming years, and the first steps are already in progress, documented as follows.

A second version of the TADD prototype system was built (1Q 2013), incorporating a larger chamber and technical improvements to meet commercial specifications, and is currently being evaluated in trials at Sutton Bridge Crop Storage Research (SBCSR), which is the leading post-harvest applied research facility for agricultural storage in the UK. These trials have included work on distinguishing various visible properties of potatoes such as blemishes versus non-blemishes, diagnosis of different types of blemishes (including scabs, scurfs, black dot and greening), peeled versus unpeeled skin, etc.

The prototype will continue to be used commercially for analysis of potatoes and other crops in storage at SBCSR, and the technology is being further developed towards commercialisation in a new TSB-funded project, described in the following paragraph. Ongoing dissemination activities are being carried out at farmer, grower, packer and processor levels, in collaboration with the project partners at the AHDB Potato Council and their Knowledge Transfer team. The TADD prototype system was also demonstrated to the industry at the British Potato (BP 2011) conference (Nov 2011), the Crop Protection in Northern Britain (CPNB 2012) conference (Feb 2012), and the World Potato Congress (May 2012).

Impact on Food Processing and Packaging
A new Technology Strategy Board-funded project on "Trainable Vision-based Anomaly Detection and Diagnosis" is in progress (Feb 2013 - Oct 2015). The project is led by Ishida Europe Ltd., who plan to incorporate and extend the technology in their online QC systems for food processing and packaging. Ishida Europe has been designing, manufacturing and delivering weighing and packing solutions to the global food industry for over 25 years. The parent company in Japan has been established since 1893 and has approximately 2,500 employees, which includes 550 R&D engineers. Ishida has many years of experience in the field of quality control with weighing and x-ray technology.

The project is currently developing the technology for inclusion in the Ishida product range, with the first version being targeted for exhibition at Interpack, Düsseldorf, Germany, May 2014, which is the world's leading trade fair for the packaging industry and related process technologies. A beta machine will also be installed at Branston Ltd.'s site near Lincoln. Branston Ltd. is one of the UK's largest potato companies and handles about 7% (400,000 tonnes) of the national crop at its three production sites. The underpinning research documented in the previous sections forms a key part of the Background IPR for this project, which has been licensed by the University to Ishida Europe Ltd.

The project outputs will be exploited directly through licensing of our technology IP. Food equipment manufacturers and supermarkets will be targeted first because of the existing scale and reach of their operations. This route and others will be explored within the project. Ishida Europe Ltd. will further conduct Open Days to showcase the technology, and will utilise both online and offline media, in multiple languages, addressing the company's existing sales territories worldwide. Routes to market will be through the Ishida product range.

Further Pathways to Impact
The research is also being disseminated through the National Centre for Food Manufacturing (NCFM), where the technologies developed in the ongoing research and technical development will be available for trials in the NCFM factory, which includes automated fresh food packaging lines with robotic case packing sponsored by Ishida Europe Ltd. The results have been incorporated into teaching materials for the industry-based short training provision and undergraduate courses at the NCFM, University of Lincoln. The research has also received worldwide attention through its coverage in the press and media, reported below.

Development of a generally applicable and robust anomaly detection and diagnosis system could generate considerable market opportunities. The AgriFood market in the UK is worth £80.5 billion. Food manufacturing represents 6.8% of the manufacturing sector and is the largest UK manufacturing sector with £13.2 billion food exports. The food industry is also the biggest manufacturing sector in Europe; employing 4.4 million people (14% jobs in EU manufacturing) and accounting for €965 billion turnover (13% of turnover of EU manufacturing sector).

Sources to corroborate the impact

Collaborators

• Agriculture and Horticulture Development Board (AHDB) Potato Council, Stoneleigh Park, Kenilworth, Warwickshire, CV8 2TL
• Sutton Bridge Crop Storage Research, Eastbank, Sutton Bridge, Spalding, Lincolnshire PE12 9YD
• Branston Ltd., Mere Road, Branston, Lincoln LN4 1NJ.
• Ishida Europe Ltd., 11 Kettles Wood Drive, Woodgate Business Park, Birmingham B32 3DB.
• The Processors and Growers Research Organisation (PGRO), The Research Station Great North Road, Thornhaugh, Peterborough, PE8 6HJ
• Frontier Agriculture Ltd., Camp Road, Witham St Hughs, Lincoln LN6 9TN

Media Coverage

• The research was highlighted on the BBC News Technology website: "New potato spotting AI built with off-the-shelf tech", 2 March 2012: http://www.bbc.co.uk/news/technology-17218662.

The research has also received wide coverage in the trade press, for example:

• Article in Farmers Weekly: "British Potato 2011: Auto blemish grading system profiled"
• Article in Potato Review (Jan/Feb 2012): "Sorting can be digitally enhanced"
• Article in FreshPlaza: "UK: Intelligent potato processor for improved efficiency"
• Article in The Engineer: "Team creates artificially intelligent potato scanner"
• Article in Vision Systems Design: "Potato industry reaps benefits of computer vision"
• Article in PCR Online: "AI potato-picker made from standard desktop PC"
• Article in Potatoes Australia (Aug/Sep 2013): "A new approach to potato defect analysis"