Summary of the impact

This research developed brain-computer interfacing technology, which has enabled people with severe physical disability to interact with and create music. Professor Eduardo Miranda and his team built and trialled a proof-of-concept device for patients suffering from almost total paralysis of the body. Locally, this has significantly improved the quality of life for one individual and changed the attitudes of hospital staff. More widely, it is informing the further development of assistive BCMI technology and contributing to on-going and emergent impact in musical creativity that engenders new ways of thinking about the potential relationships between science, technology and music.

Underpinning research

The work of Professor Eduardo Miranda and his team has benefitted many communities through performance, analysis, and the provision of technological tools. It includes basic research into music cognition, the development of models and means for the analysis of music and brain imaging data, and new digital musical instruments and systems for composition. These have been both tested and disseminated through practice-led creative research projects, which have been commissioned and performed by ensembles such as the BBC Concert Orchestra and City of Birmingham Symphony Orchestra's Leo String Quartet. Projects were developed with partners across Europe, and funded by The Leverhulme Trust (£49,000; 2004-2007), European Union FP-6 (€1,950,000; 2005-2008) and European Union - Lifelong Learning Programme (€412,280; 2008- 2010).

From this range of research activity we choose one specific case of use in therapy for brain damage. The research was an outcome of the EPSRC-funded project "Learning the Structure of Music" (grant code EP/D062934/1: £385,338; 2006-10). The team comprised Miranda (PI, employed since 02/02/2003), Dr Torsten Anders (01/07/07-30/06/09) and Dr Alexis Kirke (as a PhD student from 2007 and post-doctoral research fellow since 10/03/10).

A brain-computer interface (BCI) is a system that interacts directly with the user's brain, by picking up tiny electrical impulses of neurons, in an electroencephalogram (EEG). Other teams worldwide have proposed BCI systems for music, but they have so far failed to impact upon special needs because they have been unable to manage the fundamental problem of voluntary control. Our research provided a solution based on a neurological phenomenon known as SSVEP (Steady State Visually Evoked Potentials). These natural responses to visual stimulation at specific frequencies produce signals that can be detected in the EEG. When a person with sensors placed on the scalp looks at different flashing lights at specific frequencies, it shows up in their EEG, and a computer can be programmed to infer at which icon they are staring. The team created musical algorithms that translate specific EEG signals associated with the light frequencies of different icons into distinct musical processes. Looking at one icon sounds a certain note or produces a certain rhythm; staring at another changes its pitch, and so on. The technique was developed with biomedical engineers at Essex University and therapists at the Royal Hospital for Neuro-disability, London. Therapists were then interviewed to assess practical needs in the field; they highlighted the necessity for the technology to be easy to use, quick to set up, reliable and not requiring constant input from technicians. Based on this feedback, a proof-of-concept system was developed between January 2008 and July 2010, which realised the possibility of conducting research outside of the laboratory setting. The system was trialled with hospital staff and a patient, who had locked- in syndrome after suffering a severe stroke.

Essex University's contribution to the project (1 staff and 1 PhD student) supported computer programming and implementation of the EEG analysis algorithms. The Royal Hospital for Neuro- disability (2 staff and nurses) provided consultancy, advice on health & safety, liaison with the patient and support during the trials.

References to the research

The research was published in the following international peer-reviewed journals:

Details of the impact

This project has transformed research that focuses on the therapeutic value of music technology, by taking it out of the controlled environment of a laboratory and applying it within in a hospital setting. It has brought together developments in brain scanning analysis and automatic music composition technologies, generating an effective new approach to the creation of assistive music technology for people with severe physical disability. The research conducted at the Royal Hospital for Neuro-disability had a profound impact upon the patient, raising her confidence and sense of wellbeing. The patient - who communicates to a carer via eye movement to select letters from an alphabet table - sent the research team a note after the first trial to say "how much I enjoyed the experiment yesterday. It was great to be in control again" (15/05/2010). Recalling the impact of the experience, the patient commented on the "freedom it gave me. There was never anyone to give me orders, and I was able to take my own decisions. It was very easy to follow, and although no Mozart, I soon forgot the passing of time" (05/06/2011). The technology enabled her to demonstrate skill and ability which the able-bodied people around her did not have (e.g. control of an eye gaze system).

By embedding innovative music technology into the hectic, busy schedule of a hospital routine, one of the major barriers of medical research was broken. According to Dr Wendy Magee, previously Head of Music Therapy at the hospital, this research "offered staff an opportunity to see cutting edge research first hand; enabled them to see the impact of research on patient benefit; offered opportunity for interdisciplinary collaboration, particularly in terms of advising on the clinical practicality of the equipment being developed and publication; brought science into the clinic; and offers models of collaboration between clinicians, designers and scientists which is best practice" (18/07/2012).

Hospital staff stated that "Theoretically, we knew that the system should be helpful for people with disability. Now we have the demonstrable proof" (18/03/2011). By working with staff throughout the process the researchers ensured this technology was suitable for the hospital environment. The approach helped to create a strong partnership and encouraged medical staff to embrace academic research to benefit their own work. The research introduced resident music therapists, nurses and carers to a new tool for communicating with patients and a way of thinking about using BCMI technology in their practice, by demonstrating the system with one of their patients in situ. As a member of staff commented: "What the technology is enabling is to put the complex and severe disability in the background and someone's creativity and expression in the foreground. Beyond the obvious enjoyment that this gives to the patient, it also enables the clinicians to interact with the patient in a new way opening up new avenues for therapy and recreational activities" (06/06/2011).

Following the publication of this research in Music and Medicine, it was featured in Nature, attracting the attention of the media worldwide (18/03/2011) including CNN and BBC World Service. The Independent hailed the research as "making medical history" (04/04/2011). This exposure provided the opportunity to communicate the research to the public on a global level and generated a great number of requests by institutions and individuals worldwide interested in adopting the technology. Enterprising companies (e.g., g.TEC, Austria and Grooveshark.com, USA) have made contact to discuss forms of exploitation. In 2013, The Plymouth Community Healthcare Trust is starting to use the equipment trialled originally at the Royal Hospital for Neuro- disability to support a patient with locked-in syndrome.

This research has been further developed as part of our current EPSRC-funded project with neuroscientists at Reading University, "Brain-Computer Music Interface for Monitoring and Inducing Affective States" (2011-2016). One of its first creative outputs, Miranda's Symphony of Minds Listening, premiered at the Peninsula Arts Contemporary Music Festival 2013. Miranda composed this piece using the new methods he developed to convert fMRI brain scans information into music. These methods, also the basis for the assistive technology for music-making by people with severe disabilities, are central to the implementation of the new generation of brain-computer music interfaces he is developing for inducing emotions.

Symphony of Minds Listening complemented the overarching theme of the Contemporary Music Festival, that of memory and consciousness. It was the best attended event of the Festival, which had a total audience of approximately 700 (of whom 70% were members of the general public). The work prompted debate and critical analysis in the national media, having been featured in publications such as The New Statesman (21/02/2013), The Sunday Times (19/05/2013) and The Telegraph (25/02/2013). Professor Miranda was also interviewed on national BBC radio. In its review, Gramophone commended: "Such experiments can often impress on a scientific level but disappoint aesthetically and artistically, but Miranda's Symphony of Minds Listening achieved a neat balance between both elements" (26/02/2013).

Sources to corroborate the impact

1. Statements are available from: the patient; hospital medical staff and former Head of Music Therapy, Royal Hospital for Neuro-disability, London, UK; Clinical Psychologist, Plymouth Community Healthcare CIC, UK
2. Peninsula Arts audit of audience attendance and composition for the Contemporary Music Festival 2013.

Web links to a small sample of press evidence demonstrating the reach and significance of worldwide media attention to the research at various stages:

3. Teknisk Ukebad (12/2005), in Norwegian: http://cmr.soc.plymouth.ac.uk/publications/teknisk_ukeblad.pdf
4. L'Espresso (23/04/2009), in Italian: http://espresso.repubblica.it/dettaglio/pensa-una-musica-e-il-pc-la-suona/2079776/15
5. Nature News (18/03/2011): http://www.nature.com/news/2011/110318/full/news.2011.113.html
6. BBC World Service (8/02/2011), programme available for streaming from this site: http://www.bbc.co.uk/programmes/p00dbw5m
7. CNN (29/03/2011): http://www.edition.cnn.com/2011/TECH/innovation/03/29/music.brain.power.therapy/index. html
8. The Independent (4/04/2011), includes interview with the patient: http://www.independent.co.uk/life-style/health-and-families/health-news/lockedin-woman- makes-medical-history-2261429.html
9. BBC World at One, on both therapeutic uses of BCMI and Symphony of Minds Listening (20/12/2012), programme available for streaming from this site: http://www.bbc.co.uk/news/uk-20799961
10. The Gramophone (26/02/2013): http://www.gramophone.co.uk/blog/concerts-and- events/peninsula-arts-contemporary-music-festival-2013