Summary of the impact

Music teachers, physiotherapists, museum curators and other practitioners have used the results of our research to improve their practice, with consequent benefits to individuals. For example, a violin teacher used our MusicJacket haptic guidance system to permanently improve pupil violin bowing technique. A neuroscience team made use of our Haptic Bracelet system in a novel form of gait rehabilitation with a patient recovering from a hemiparetic stroke, who reported improved posture and movement. Through public participation in events featuring our Haptic Lotus, such as theatre performances for blind and sighted people, as well as our engagement in schools and at festivals, we have stimulated public interest in technologically mediated approaches to issues of health, the arts and accessibility. This has led to informed public discourse through reports in national newspapers, magazines and the BBC.

Underpinning research

Researchers from the Pervasive Interaction Lab — Dr Janet van der Linden (Senior Lecturer), Dr Simon Holland (Senior Lecturer), Professor Yvonne Rogers (at the OU July 2006 - Sept 2011) and Dr Jon Bird (Research Fellow at OU from Sept 2008 - Dec 2009) worked together on the design, development and evaluation of systems that provide technologically enhanced senses. Key research questions were: "Can people use technological systems as if they are part of their bodies?" and "To what extent can these technologies support users in subtle, fluid ways that feel natural and intuitive?"

In 2009, the E-sense project AHRC funded, £207,000; 2008-2010 `Extending our sense and self through designing novel technologies' (Rogers, Holland, Clark)) developed the TVSS system - a `minimal tactile vision sensory substitution system' [3.1] in order to explore situations in which sense of vision might be replaced by sense of touch. In this system a camera tracked the direction of a ball being rolled towards a blind-folded person, who tried to catch the ball using camera-controlled tactile feedback on their stomach, provided by an array of vest-worn vibrotactiles.

Insights from the minimal TVSS inspired van der Linden in 2009 and 2010 towards the development of the MusicJacket [3.2, 3.3] to support the complex task of violin playing. A musician's visual and auditory systems are already heavily involved in the process of playing the violin, and this system explored the extent to which tactile feedback might provide an intuitive way to guide players' bowing, as well reducing cognitive load compared with visual or auditory feedback. The MusicJacket uses motion capture technology to record a person's movement and gives real-time feedback on the correctness of violin bowing through vibrotactiles. Using input from professional violin teachers, the system first records a person's perceived ideal trajectory for bowing. If, during playing, a person moves their arm inappropriately, they feel a gentle buzz to nudge them towards a more appropriate pattern of movement.

One strand of E-sense focused on technologically enhanced senses for timing and synchronisation, leading Holland to develop the Haptic Drum Kit [3.4] in 2009. Vibrotactiles are attached to each wrist and ankle to guide the playing on a drum kit of rhythmic patterns that require multi-limb coordination. Such rhythms are typically learned through notation, sight or sound.

However, notation is hard to learn, sight communicates timing poorly, and sound gives inadequate indication of which limb moves when. By contrast, touch can communicate timing well, and can directly signal what each limb does when.

Following this research, in 2012 Holland designed the Haptic Bracelets as self-contained, low latency wireless bracelets for wrists and ankles, with a larger dynamic range and controllable by smart phone, as proposed in [3.5]. These two projects demonstrated that beginning drummers were able to learn intricate drum patterns from haptic communication alone. More generally, this research showed that haptics could be used to support accurate multi-limb timing and synchronisation.

In 2010 van der Linden and Rogers investigated issues of accessibility through haptic technologies asking whether blind and sighted members of an audience could have a comparable sensory experience when engaging with a dramatic theatre performance. That is, instead of providing blind people with a compensatory alternative description, both blind and sighted people were given the same opportunity to explore a multi-sensory theatre experience set in the dark. At its core was the Haptic Lotus, a device designed in the form of a flower, which sits in the palm of the hand and changes its form in response to a person's journey through the dark [3.6]. This work was in collaboration with Oshodi from Extant UK (a performing arts company for and by blind artists) and Spiers, and was funded by the Technology Strategy Board Technology programme Creative Industries Fast Track (£35,000 for match funding) October 2009 - June 2010 (`Haptic Theatre project', Extant, Battersea Arts Centre and Open University).

References to the research

(key references in bold)

Details of the impact

MusicJacket

Music teaching practitioners used the MusicJacket to improve their practice, with consequent benefits to individuals. During 2010 the MusicJacket system was used over a period of two months in violin lessons with a group of 10 children ranging in age from 6 to 14 years. Three years later, the impact of the technology was still felt, as explained by one teacher: `This system was very successful for this student and altered her bowing patterns consistently and accurately. ... the MusicJacket had therefore been responsible for a permanent change in the student's bowing technique.' [5.1]

The MusicJacket research also impacted on teaching practice at professional and Masters level. In March 2011 the Utrecht Music Conservatorium invited van der Linden to take part in a continuing professional development course for violin teachers and to run a class with Masters students in Music. The introduction of the MusicJacket to a class of Master students gave the lecturer a technological tool to help stimulate reflection and debate about their styles of practising. The lecturer commented: `Seeing the external technology needed to chart a playing technique made the students value and understand their own internal workings. They understood better that it is not just doing but witnessing the playing that makes for accuracy. This required them to think about how their goal-oriented practising might be clouding their feedback mechanisms, preventing true advancement in study.' [5.2]

The system also created public interest in science and engineering approaches to creative areas and was shown on national television in the BBC series for young people, Bang Goes The Theory [5.3].

Haptic Bracelets

Haptic Bracelets made an impact on health, where they demonstrated they can alleviate problems with gait following a hemiparetic stroke. Stroke survivor JB commented on the effect of the Haptic Bracelets on her walking: `it makes you stand up straighter ... When I stand up straight my hips move better and I walk more smoothly and it's easier. ... This helps me to walk in time. It's just sort of having an even pace ... which helps me stand up straight and walk properly.' [5.4]

Awareness of the Haptic Bracelets as a new approach to gait rehabilitation has been spread through participative demonstrations with the professional Association of Chartered Physiotherapists with Interests in Neurology (ACPIN), which were attended by around 50 NHS physiotherapists [5.5].

A hands-on workshop for NHS physiotherapists at Wye Valley NHS Trust was held at the invitation of the Stroke Research Facilitator [5.6]. This workshop examined the role of the Haptic Bracelets in gait rehabilitation for stroke, Parkinsons and related conditions.

Haptic Lotus

A theatre event featuring the Haptic Lotus stimulated public interest and engagement in new technologically mediated approaches to accessibility for the arts. We provided the opportunity for some 150 participants to gain hands-on experience with the Haptic Lotus, during performances in Battersea Arts Centre in June 2010 [5.7]. The audience involved both blind and sighted people from a wide range of backgrounds and professions: accessibility, gallery and heritage professionals, artists, technology innovators, scientists and teachers.

This stimulated informed public discourse through reports in national newspapers and popular magazines. Guardian journalist Naomi Alderman, in her column on technologies and gaming, describes the event as `my standout theatre experience of the year' [5.8].

The Haptic Lotus was also discussed in Wired (the magazine for future science, culture and technologies), reviewed in Switched and featured in Philosophy Now, where it sparked a discussion on how philosophers from Locke to Schopenhauer, Wittgenstein, Magee and Milligan, had approached the issue of different kinds of knowledge, distinguishing between the simple ideas of our senses and the complex ideas of reflection [5.7].

This work also had impacts on awareness, attitudes and understanding. In 2012 during the planning of a new interactive Information Age gallery, an Accessibility Advisor to national museums, including the London Science Museum said: `awareness of this project, and other examples of innovative practice, has encouraged the museum to widen its approach to multi- sensory access. These projects have definitely inspired us to think more creatively about solutions to overcome sensory barriers and to explore more diverse approaches to interpretation' [5.9].

Other impacts on society, culture and creativity

Public interest and engagement in scientific and engineering approaches to technologically enhanced senses have been stimulated by the demonstration of our research to diverse audiences in informal settings. For example, during the Brighton Science Festival (21 February 2009) and at the London Museum for Science (23 June 2009), primary school age children interacted playfully with each other wearing the minimal TVSS system, thus getting direct experience of, and engagement with, sensory substitution [5.10].

Sources to corroborate the impact

[5.1] Violin teacher who took part in the evaluation of MusicJacket

[5.2] Alexander Technique Lecturer and Project Coordinator for `Musicians in Balance', Utrecht Music Conservatorium

[5.3] BBC Bang Goes The Theory, broadcast 2 May 2011, http://www.bbc.co.uk/programmes/b00lwxj1/broadcasts/2011/05

[5.4] Haptic Bracelet page on Music Computing website: http://mcl.open.ac.uk/MusicLab/86

[5.5] Head of Sensory Motor Neuroscience Centre, Birmingham University

[5.6] Stroke Research Facilitator, Wye Valley NHS Trust

[5.7] The Question, Battersea Arts Centre: http://www.thequestion.org.uk/ (their press page has links to press releases in Wired, Philosophy Now and Switched)

[5.8] The Guardian, `The player: videoless games can reveal skills we didn't know we had', Wednesday 10 November 2010; http://www.guardian.co.uk/technology/2010/nov/10/player- audiogames-reveal-skills

[5.9] Accessibility Advisor (advisor to national museums including Science Museum, London)

[5.10] Esense project website: http://mcs.open.ac.uk/esense/