Summary of the impact

Augmented reality (AR) and physiological computing (PC) represent computing paradigms for wearable technology. Both forms may be combined to deliver Adaptive AR (A2R) where changes in psychophysiology are used to adapt digital artifacts in real-time. A number of art exhibits were created that represented A2R and were presented to the public as part of the Turning FACT Inside Out show in Liverpool. The impact of this research is evidenced by: (a) engaging the public with emerging technology, (b) influencing the strategy of an arts organisation, and (c) informing the practice of artists.

Underpinning research

Advances in electronics have accelerated the development of wearable computing. The availability of the smartphone and tablet provide a convenient hardware platform for a range of augmented reality (AR) applications. Parallel developments in wearable sensors have enabled the collection of physiological data (e.g. heart rate) for personal health informatics. The development of wearable computer technology prompts a number of research questions in the area of human-computer interaction (HCI): which applications and interactive experiences are viable? What are the issues surrounding wearable systems and user acceptance? It is important that nascent forms of technology are designed around the person and not vice versa; hence psychologists and HCI professionals must drive the debate around wearable computing.

The field of physiological computing encompasses all categories of technology where physiological data is used as a control input for a computing system. Research on physiological computing at LJMU began in 2008 via the REFLECT project (http://reflect.pst.ifi.lmu.de) funded by the European Union. LJMU developed an interactive computer game (Tetris) where changes in motivation were quantified from spontaneous EEG activity and used to adjust game difficulty, e.g. if the player was bored, the game became more difficult and vice versa. The team at LJMU also worked closely with Philips Research Laboratories in Eindhoven on the Mood Music Player; this was an interactive mp3 player designed to promote positive emotions based on psychophysiological responses from the listener. LJMU investigated the potential of positive music to alleviate anger in the context of driving.

In 2011, Professor Fairclough was part of the consortium which successfully obtained funding from the Cultural Heritage division of the EU to fund the ARtSENSE project (http://www.artsense.eu) under Framework 7. One goal of this project was to combine physiological computing with AR to produce a wearable system where AR content would adapt to the level of interest exhibited by the psychophysiological response from the user; this hybrid concept was called Adaptive Augmented Reality (A2R). A real-time prototype of an A2R system was developed to work in the context of art galleries and museums in order to personalise the experience of the visitor.

A major partner for LJMU in the ARtSENSE consortium was the Foundation for Art and Creative Technology (FACT) (http://www.fact.co.uk) who decided to commission a new piece of art based upon the A2R concept as part of the project. LJMU worked in partnership with FACT and a collective of AR artists called ManifestAR (http://www.manifestar.info) to create a number of exhibits. The three groups spent two weeks together in Liverpool in June 2012 working on concepts for a show in 2013.

The idea behind the ManifestAR commission was to enable collaboration between artists, an arts organisation and academics to push the boundaries of interactive digital art using wearable technology. In addition, the show allowed members of the public to experience an emergent category of technology in the context of an experience that was both interactive and aesthetic.

References to the research

^** note that ACM conference papers were reviewed in advance by four experts and acceptance rates for these conferences are generally conservative, e.g. approx. 38% for the 2012 conference.

Key Grants

2008-2011 REFLECT project (http://reflect.pst.ifi.lmu.de) £225,000

2011-2014 ARtSENSE project (http://www.artsense.eu) £395,000

Details of the impact

The ManifestAR exhibit was called `Invisible ARTaffects' and was presented as part of the Turning FACT Inside Out show that opened on the 13th June 2013. The physiological computing group at LJMU contributed to the design of three out of the five pieces that comprised the ManifestAR exhibit. The `Human Conference Sensors' project was concerned with sustaining audience concentration via the introduction of augments based upon heart rate activity. `Things We Have Lost' incorporated an EEG sensor into an AR exhibit whereby the act of mental relaxation elicited the appearance of digital objects. Biomer Skelters represented a piece where vegetation in augmented reality was "planted" in the streets of Liverpool via a biofeedback loop.

The opening of `Invisible ARTaffects' was preceded by a number of public engagement events and workshops at FACT. Between the opening of the show on 13th June and 31st July 2013, 1,488 people had visited the `Invisible ARTaffects' part of the show. Biomer Skelters was an interactive experience based in the physical environment of the city. The idea behind this piece was to `plant' a series of `virtual' trees and vegetation as one walked a route through Liverpool. The rate of planting was controlled by the heart rate of the person; specifically, the system was calibrated to increase the rate of planting when heart rate was low indicating a state of relaxation. LJMU, FACT and ManifestAR designed a Biomer Skelters tour where two people used the system competitively as they crossed the city, each trying to simultaneously "out-plant" the other by relaxing as they walked.

Five tours using Biomer Skelters ran through June and July. Questionnaires were distributed by LJMU to gauge public perceptions of the technology, which indicated that users enjoyed the experience of using this new form of technology and found the experience of regulating heart rate whilst walking to be challenging.

The tours elicited a number of comments from the public, a selection of which are reproduced below:

(a) "Very impressive and would recommend to others to try. Beautiful imagery."

(b) "This technology has an awful lot of implications for both medical and creative fields."

(c) "Perhaps I should reconsider going to the gym!"

By exposing the people to this emerging technology, it was clear that the public perceived a number of benefits associated with the technology as an aesthetic experience (a), therapeutic tool (b) and a health check (c).

The two artists who created Biomer Skelters (Tamiko Thiel and Will Pappenheimer) felt that collaboration with LJMU opened up possibilities of incorporating the body into the virtual spaces created by their art. Tamiko Thiel articulated the impact of the collaboration on her practice as follows: "My concern is to build AR artworks that really engage the viewer/user in an exploration of the physical site and with their own body in space. In this project the biosensor made these ubiquitous but often unconscious connections and feedback loops between us, the artworks and our environment tangible and perceivable as a complete system."

The Research and Innovations Manager at FACT felt that the collaboration with LJMU has already exerted a significant impact on their strategic direction as the foremost arts organisation in the north-west; in his own words: "the deployment of emerging tools introduced to FACT by the team at LJMU has helped initiate the emerging FACTLab model that places impactful scientific research as a touchstone for programming the public spaces of the building. Several projects have already emerged as part of this new approach including: a collaborative project with the RCA and Lancaster University called Rhythmanalysis that uses such tools to improve our understanding of work/life balance (to be exhibited in December 2013); a project called Your Tour with the Sainsbury Centre for Visual Arts in Norwich and the University of Central London University, that uses locative tools to push artwork specific content to visitors in a gallery setting and a commission with artist Daksha Patel that used psychophysiological sensors to create a live and public participatory performance artwork on the ground floor of FACT."

The impact of our research is based upon the capacity of emergent technology developed in academia to innovate digital art and influence the strategy of a major arts organisation in the UK. The show and tours raised public awareness about A2R technology as well as eliciting views from potential users about applications for wearable computing.

Sources to corroborate the impact

Human Sensors Conference at Hacky Birthday Event
http://www.fact.co.uk/news-views/2013/02/human-conference-sensors-at-hacky-birthday/

Announcement of Biosensor Training Session
http://www.fact.co.uk/whats-on/current/free-biosensing-training/

Description of Invisible ARTaffects
http://www.fact.co.uk/projects/turning-fact-inside-out/manifestar-invisible-artaffects/

Interview with Professor Fairclough about collaboration with ManifestAR
http://www.fact.co.uk/news-views/2013/06/creating-invisible-artaffects-with-ljmu/

Biomer Skelters Tour Announcement
http://www.fact.co.uk/news-views/2013/08/take-a-unique-bio-sensing-tour-of-liverpool/

Biomer Skelters video from Vimeo
http://vimeo.com/69094382

Article from BayTV
http://baytvliverpool.co.uk/vod/index.php?vid=PBV51b9b8c3bb4a9

FACT Inside Out article in Liverpool Daily Post
http://www.liverpooldailypost.co.uk/culture/arts/interview-what-expect-exhibition-turning-5397553

Review of FACT Inside Out in The Guardian
http://www.theguardian.com/artanddesign/2013/jun/17/indoor-fracking-installation-provoke-debate

Review of FACT Inside Out in Manchester Salon
http://www.manchestersalon.org.uk/turning-fact-inside-out.html