Summary of the impact

The case described shows that our research reporting on the limitations and challenges associated with traditional approaches to presentation and management of information search results for different types of users has led to design and implementation of non-conventional user interface technology - INVISQUE (Interactive Visual Search and Query Environment [7]). This technology has had a direct influence and usage in a range of domains including information seeking in Citizens Advice Bureaux and in the security domain both nationally (UK Ministry of Defence) and internationally (US Dept. of Homeland Security). The research has created a community of practice around the emerging field of Visual Analytics and has formed the basis of a successful FP7 project (EC grant €13.1M) bringing together a consortium of 18 industrial, internationally leading Visual Analytics researchers (e.g. PNNL, University of Konstanz, City University London), and police end-user partners, from across the UK, Europe and the US.

Underpinning research

The use of lists to present information search results is a common approach (e.g. Google search results). However there is evidence that users often get lost or lose sight of their original goals after drilling down, backtracking and so on [2]. When further considerations such as the domain of search, and the type of user (e.g. the level of literacy) are taken into account then limitations of such presentation formats become further pronounced. Researchers at Middlesex, historically, Prof Ann Blandford, Prof Harold Thimbleby (1999-2003), and more recently Prof William Wong (since 2003) have been researching the design of user interfaces to present complex information in novel visual forms taking into account the complexities associated with types of users. The impacts described in this case study arise from several strands of research activity.

In 2005 two key papers reported on research exploring information seeking behaviours utilised in digital libraries (DL) and websites supporting the services of the National Citizens Advice Bureau (CAB). The papers noted that experts could deploy a repertoire of simple but effective strategies that could be used to generate design ideas for user interfaces that aid `ordinary' users. [3] [1]. Specific cases identified included guidance on how to integrate adaptive help systems into DL systems. Both papers utilised empirical data collected in situ through observation studies, cognitive task analysis and think aloud techniques.

Whilst one study [3] explores strategies based on expert users, other research [1] investigates the effects of low literacy skills on information retrieval. The results indicated the increased likelihood of getting lost; taking longer; the inability to backtrack amongst other issues. The latter issue was also reported in a subsequent report produced for the Joint Information Systems Committee (JISC) following a study examining search behaviours of first year business students in higher education. Additionally and grounded in HCI and cognitive systems engineering, the need for novel interaction and visualisation techniques to support spatial-temporal reasoning in complex environments has led to applications such as air traffic control where research on novel 3D-in-2D visual representations [5] has been conducted. More recent research exploring techniques for displaying multiple dimensions of information across physically distinct layers of information has been reported in [6].

Research in this area is broadly coalescing around the notion of visual analytics - an area of interaction design that has now become firmly established at Middlesex.

Visual analytics is the emerging science of analytical reasoning supported by highly interactive visual interfaces that combine a range of automated analysis techniques with user driven visualisations. The integrated technologies collectively enable real-time analytic interaction with very large complex datasets from multiple sources to support effective understanding, reasoning and decision making.

A research outcome from our activities in visual analytics is INVISQUE, a non-conventional user interface technology. The original prototypes were developed in 2009 following funding from JISC and were refined and applied to the areas of digital resources in digital libraries [4] and in the context of low literacy users [1]. Evaluation of INVISQUE showed significant improvements in performance. The non-conventional INVISQUE UI, in the context of low literacy users, was found to significantly improve task completion and success in finding information, reduce lost-ness and back-tracking. INVISQUE helped them see what the system retrieved, and how it organised, grouped the results, showed links and revealed associations between data. The results indicated the applicability of this technology to the security domain hence the impact on that community.

References to the research

This research was based on competitively funded projects, with robust peer review systems. The outcomes from the research were published in leading peer review journals and conferences in the field.

Related funded research projects associated with research all peer reviewed and competitively won and most supported by expert advisory committees are:

1. JISC Rapid Innovation: INVISQUE, Interactive Visual Query Environment; 40,000 GBP; 2009. PI: Prof. William Wong.

2. EPSRC: Making Sense: 10 UK partners; 245000 GBP; 2010-2013; Co-PI: Prof. William Wong.

3. UKVAC (UK Visual Analytics Consortium) Phase 1, NVAC; 2010-2011; $250,000; PI: Prof. William Wong.

4. TSAM: Team Sense Making Assessment Method. Partners with BAe Systems, ARA Klein Associates, MoD CDE. 50000 GBP; 2011.

5. UKVAC (UK Visual Analytics Consortium) Phase 2, US Dept of Homeland Security, UK Home Office; 2012-2013; $500,000 + 400,000 GBP ; PI: Prof. William Wong.

6. VALCRI: Visual Analytics in Criminal Intelligence Analysis; EU FP7. 2013-2017; €13.1M; PI: Prof. William Wong.

Details of the impact

Visual analytics (VA) continues to mature as a subject area within Interaction Design and Middlesex has increasingly provided international leadership in this area both to researchers and the wider research user community. Impacts arising from our research in VA address deployment of software at government research labs in the security sector; supporting low literacy users such as those using services located in Citizens Advice Bureaux (CABs) and leadership at both a national and international level in VA.

A key aspect of applied aspects of the research detailed earlier is the need to support `theory building' (such as that found in intelligence analysis) using visual mechanisms for reasoning and narrative construction based on data that has been retrieved and organised by the various automated techniques e.g. automated knowledge extraction and thematic clustering. Our ideas have been implemented in the INVISQUE technology leading to impact in three areas.

Firstly, INVISQUE enables visual support for human reasoning actions to support theory building and this has applications in the intelligence domain. Secondly, as this has developed, the research has contributed to and established a community both nationally and internationally in the area of visual analytics. Thirdly, recognising that the software technology originates from research exploring issues facing low literacy users, the software has also applicability in CABs.

The research is being used by the UK MoD to help them define theory-building tools for their intelligence community. In our work, we refer to the concept of a Reasoning Workspace and how such a Reasoning Workspace is formed in design. The constituent elements are underpinned by the research reported from 2005 onwards that identified insights into how expert users bring to bear techniques to support their information seeking behaviours. The research also identified the need for novel visual representations and the need to support backtracking within such visual representations. Some of these findings were most clearly noted when first working with low literacy users but now have their use in a security setting.

While the software arising from the research, INVISQUE, can be provided on request, it has been made available to both the Pacific North Western National Laboratory (PNNL) and the UK Ministry of Defence (DSTL). At PNNL, a variant of INVISQUE is installed and used as a complementary tool to other technologies developed by PNNL [S3]. DSTL has used the software for experiments with security related information. The usage of the technology is reported in an internal manuscript and confirmation of the activity noted by DSTL [S4]. Thus the primary beneficiaries of this research are the security organisations of the UK and US. The impact of the research is largely the design and principles underpinning the technology.

The novelty of the INVISQUE environment and the supporting research led to the first UK workshop/conference on the recently established field of visual analytics. This workshop in 2010, at Middlesex, was attended by delegates from both the UK Government and US Dept of Homeland Security and was a direct consequence of the research presented from Wong and others. The workshop provided the basis for the formulation of the UK Visual Analytics Consortium led by Prof. Wong with funding from Department of Homeland Security and the UK Home Office [S2].

UKVAC is funded by to carry out research in technologies based around INVISQUE and to provide advanced training on related technologies for visual analytics. Other members of this consortium include Oxford, Imperial, UCL, Bangor and Swansea.

The UKVAC has run three International Summer Schools and an accompanying conference (2011, 2012 and 2013) and is the premier UK located event in this area attracting delegates attending the summer school from the intelligence, finance and academic communities amongst others.

The INVISQUE technology has also been applied in Citizens Advice Bureau settings. Further we planned to develop a deployable version for the National CAN to begin in Feb 2012. However due to budget cuts at the CAB, the project was postponed until further funding could be found. A number of commercial organisations have expressed various levels of interest in this technology, including Microsoft, IBM i2, BAE Systems, Detica, Taylor and Francis (which was prepared to adopt the INVISQUE user interface concept - but had to cancel the project as a result of funding cuts) [S1], JISC and the British Library.

Professor Wong has played a pivotal role in ensuring that INVISQUE has also had impact on the wider international community through the offices of the NATO IST110 committee and is presenting a keynote at the NATO Visual Analytics Symposium. This impact is evidenced by the supporting letter from Prof Margaret Varga, Chairman of NATO IST100/RTG055 [S6].

INVISQUE and its underlying concepts have been the basis for a successful grant proposal to the European Commission. The project VALCRI (Visual Analytics in Criminal Intelligence Analysis) has been costed at €16.8 mil (EC grant €13.1 mil) and involves 18 industrial, research and police end- user partners across the UK, Europe and the US [S5].

Sources to corroborate the impact

The nature of the primary beneficiaries means that it is difficult to get confirmation from security related organisations on the use of technology.

S1. Taylor and Francis: Emails to confirm the interest of publishers in the Invisque platform.

S2. Pacific Northwest National Laboratory (for the US Dept of Homeland Security). Statement of Work contract.

S3. Pacific Northwest National Laboratory: Letter confirming use of Invisque at PNNL.

S4. DSTL: Letter describing role of Invisque platform at DSTL.

S5. VALCRI funding letter.

S6. Supporting letter from NATO IST110 / RTG055