Summary of the impact

[text removed for publication], a developer of high-precision medical devices, have produced a new data annotation tool ([text removed for publication]) based on research in CSRI on data storage formats and activity recognition for applications within smart home environments. Within [text removed for publication] stereo-based cameras record activities in a specified environment (e.g. kitchen) which are then annotated using user-based pre-configured activity labels (e.g. prepare meal, wash dishes). [text removed for publication] is currently used by [text removed for publication] users and has yielded additional sales worth [text removed for publication]. [text removed for publication] have employed [text removed for publication] additional technical development staff to extend [text removed for publication] functionality, and through an MoU [text removed for publication] now supports automated annotation based on CSRI's research on activity recognition.

Underpinning research

Smart environments enable sensor technologies, information and communication technologies, and adaptive interfaces to be combined to record users' movements and interactions with objects in the environment (e.g. home, workplace). The purpose is often to conduct subsequent analysis of user behaviours that is based on automated recognition of activities being undertaken (e.g. cooking, eating). Within the smart environments research community, the lack of validated and annotated data sets, stored in a common format, is a recognised problem. Without such datasets, training and evaluation of automated activity recognition models are limited, which in turn leads to the development of methods for automatically recognising behaviour change being limited in terms of generalisation, scalability and transferability to other domains. Such methods are an essential element of technology-based approaches to assessing functional deterioration in persons with conditions such as dementia.

For the past 10 years a core theme of CSRI research has been data collection and storage [1, 2], coupled with automated activity recognition within smart environments [3, 4]. A range of approaches to govern data collection have been developed that are based on specifically designed data structures using XML [2]. Presently there are no common formats for storage and exchange of data collected in a smart environment, without which, exchange, re-use and validation of common datasets are limited.

During 2003 CSRI developed and evaluated an approach for storage and exchange of electrocardiogram data through an XML-based approach [1]. The output, referred to as ecgML, was the motivation for a subsequent XML-based approach developed for use within the smart environments research domain (homeML) [2]. HomeML provides a structure that enables all user-related data (activity levels, vital signs, object interactions), both within the home environment and beyond, to be stored in a common format. This format has been used to re-purpose four internationally available datasets, requiring only minor amendments to the format to accommodate all types of data present. In addition, homeML was evaluated for its usability by researchers from 11 different international research centres during 2009-2012. The evaluation results established the need to introduce the homeML concept more widely across the research community. HomeML is now a freely available schema, and a repository where datasets can be uploaded/downloaded is also being promoted (http://www.home-ml.org/Browser).

Through research funding from the NI Department of Employment and Learning for the Centre in Intelligent Point-of-Care Sensors (2008-2011) and Deployment of Sensing Technology in Connected Health Care (2009-2011) the work on collection and exchange of data was extended to research on developing activity recognition algorithms [3, 4]. In particular, this research yielded successful approaches to a significant practical problem when managing sensor data in smart environments: how to handle various sources of error in data recordings (data transmission errors, faulty sensors, battery failures). Information engineering techniques with the ability to reason under uncertainty (Dempster-Shafer Theory of evidence) have been incorporated into the process of activity recognition [5, 6], with improved accuracy of activity classification demonstrated compared with approaches that do not address uncertainty in the data [6].

This work has been conducted by a team of key researchers in CSRI:

Chris Nugent	Professor of Biomedical Engineering (joined as Lecturer, 05/2000)
Dr Luke Chen	Lecturer/Senior Lecturer/Reader (09/2005-present)
Dr Mark Donnelly	PhD student/Research Associate/Lecturer (10/2004-present)
Dr Haiying Wang	PhD student/Research Associate/Lecturer/Senior Lecturer (10/2000-present)
Dr Xin Hong	Research Associate (11/2005-01/2012)

References to the research

* References that best indicate the quality of the underpinning research.

Key Grants

Project: Cross-border Centre for Intelligent Point-of-Care Sensors
Funder: NI Department of Employment and Learning £1,991,283 (to Ulster)
Dates: 11/2008-03/2011
Ulster grant-holders: CD Nugent, D Finlay, P McCullagh, SI McClean, BW Scotney

Project: Deployment of Sensing Technology in Connected Health Care
Funder: NI Department of Employment and Learning £623,900 (to Ulster)
Dates: 02/2009-03/2011
Ulster grant-holders: CD Nugent, D Finlay, P McCullagh, L Chen, SI McClean, BW Scotney

Project: Personal IADL Assistant
Funder: EU Ambient Assisted Living Joint Programme (AAL-2012-5-033) £101,352 (to Ulster)
Dates: 03/2013-02/2015
Ulster grant-holders: L Chen, CD Nugent

Details of the impact

[text removed for publication], a manufacturer of innovative medical products, have produced a new product ([text removed for publication]) to support data annotation (labelling of user activities based on playback of video) within smart environments based on research undertaken by CSRI. The [text removed for publication] product has the ability to record user interactions with objects (e.g. turning on a tap, lifting a cup, opening a door) within a smart environment through a set of stereo-based video cameras and to synchronise recordings with data generated by other sensors (e.g. notifications via contact or motion sensors of: a door opening, a person moving, a household object being lifted). Pre-configured activity labels appropriate to the environmental context are used to manually annotate user activities (e.g. preparing a meal, using the telephone, making a drink).

In 2009 CSRI entered into collaboration with [text removed for publication]. By using stereo-based cameras with millimetre accuracy, the [text removed for publication] Platform marketed by [text removed for publication] at that time had the ability to track the movement of surgical equipment, in relation to a reference point, during surgery. During 2009-2012, research results from CSRI's evaluations of homeML established the need for large validated and annotated datasets sharing a common format to be collected within smart environments to support the development of activity recognition algorithms. This, together with [text removed for publication] expertise in precision measurement and tracking, was the motivation for developing the [text removed for publication] product. [text removed for publication] have recognised the guidance provided by CSRI in the product development of [text removed for publication] [E1], which is designed for use by research organisations to produce validated and annotated datasets that can be shared in a common format.[text removed for publication] is a unique product and the first of its kind. Via stereo-based cameras and wireless sensor networks, the [text removed for publication] product can record and synchronise multi-channel data about user activities within a smart environment (e.g. opening a door, lifting a cup, turning on a tap). Following completion of a set of activities, recorded video may be replayed and the users' actions annotated (labelled), subsequently automatically annotating all other sensor-based data that have been simultaneously recorded. The [text removed for publication] product incorporates CSRI's research results on the development of a common format for smart environment data by storing its data using the homeML format [2].

Through development of the new product, [text removed for publication], that incorporates CSRI's research on data storage and common format for smart environments, [text removed for publication] have experienced a positive economic benefit. Since 2010 the economic impact of [text removed for publication] for the company has been:

• [text removed for publication] additional staff have been employed by [text removed for publication] as research and development engineers to progress and support development of the [text removed for publication] product [E1];
• [text removed for publication] have secured [text removed for publication] contracts to use [text removed for publication], yielding additional new revenue of [text removed for publication] [E1];
• The [text removed for publication] product is currently being used by [text removed for publication] users [E1].

The results from CSRI's research on activity recognition [3, 4, 5, 6] have been exploited further by [text removed for publication] through establishment of a Research Agreement with CSRI to extend the functionality of the [text removed for publication] product by incorporating automated activity recognition modules [E2]. A communication architecture has been defined at a software level to be used by [text removed for publication] and CSRI to support the software integration of activity recognition modules within [text removed for publication]. The purpose of including activity recognition modules is to facilitate the automatic annotation of activities and thus reduce the amount of manual annotation required when using the [text removed for publication] system, significantly reducing the time required to generate a fully annotated dataset.

An additional impact of this research has been in public sector service enhancement. During 2012 Belfast City Council made a successful funding bid to the NI Department of Culture Media and Sport's £100M Urban Broadband Fund to position Belfast as a "super-connected city". As part of the bid CSRI provided supporting rationale to Belfast City Council's application in the form of Connected Health Case Studies that would benefit the community if a super-connected city were to be established. One of our Case Studies demonstrated the significance of high-speed network access for remote monitoring of smart environments and automated recognition of activities taking place in those environments. Belfast City Council was awarded £13.7M, with full recognition of the support provided by CSRI being acknowledged in this process [E3]. The award is enabling Belfast to become a world-class digital city, providing consumers with faster access to wireless broadband services throughout the city and growth potential for local industry.

Whilst the research on developing approaches for storage and exchange of data within smart environments [2] is associated in part with a long-standing research collaboration since 2007 with Josef Hallberg and Kare Synnes, Lulea Technical University, Sweden, it is the contribution of CSRI to that research and CSRI's research results on automated activity recognition that are incorporated into the [text removed for publication] product.

Sources to corroborate the impact

[E1] Factual Statement in the form of a letter from [text removed for publication]. This item provides corroborating evidence that the company [text removed for publication] has secured revenue from the new [text removed for publication] product, has increased its number of staff to further develop the [text removed for publication] product, and that the system is currently being used by a range of end-users.

[E2] MoU between [text removed for publication] and CSRI. This item provides corroborating evidence of the open development platform from [text removed for publication] to support integration of activity recognition modules based on research by CSRI at Ulster.

[E3] Factual Statement in the form of a letter from Belfast City Council. This item provides corroborating evidence of the recognition of the influence of CSRI's research in the establishment of the Super-connected Belfast initiative through provision of a Case Study about remote monitoring in smart environments that was used by Belfast City Council in their successful funding proposal for the initiative.