Summary of the impact

Augmentative and Alternative Communication (AAC) systems and devices assist people with little or no functional speech to communicate. Dundee research on pragmatics and conversational modelling to improve communication rates and support conversational flow has led to widespread use in AAC of re-usable phatic conversation (e.g. greetings, responses, farewells) and visual scenes (rapid access to conversational items). More recently, the application of natural language generation and sensor-based data-to-text technology has resulted in the automated generation of jokes and narratives to assist non-speaking people to engage in lively conversation. Symbolic and phonetic interfaces have been developed for children and adults with congenital and language impairments.

Underpinning research

Research on conversational-modelling for AAC was started by Arnott and Newell in Dundee, working with Ph.D. students Alm and Waller. Alm researched support for non-speaking people in informal dialogue through the use of conversational pragmatics, and extended this work into computer-based personal biography including storage and retrieval methods to manage anecdotal material [1].

Post 1993, Alm's funded collaboration ("PicTalk" project: Leverhulme, 1994, £23,600) with Todman (Psychology, Dundee) and File (Abertay) further developed the theme of conversational pragmatics in AAC, with Alm also contributing further HCI development. The concept of intersecting perspectives as a mechanism to enable natural incremental movement through conversational topics was proposed by Todman and Alm [2] at this time.

Alm, Arnott and Murray (all Computing, Dundee) developed the use of scripts and visual scenes in AAC [3],[7],[8]. Such use of scripts with the graphical visual scenes supporting them were novel contributions to AAC. Concepts from conversational pragmatics and script-based systems have since influenced most commercial AAC products in the form of reusable utterances and visual scenes.

Transatlantic collaboration between Todman, Alm (Dundee), File (Abertay), Higginbotham (SUNY, USA) and Enkidu Research Inc. (USA) resulted in the CONTACT project ("A whole utterance-based communication aid for people without speech: An office workplace implementation", EPSRC GR/R51353/01, £132,254 pre-FEC, 2001-2004), merging outcomes of their research to produce a prototype AAC system for use in the office workplace.

Waller and Newell (both Computing, Dundee) established the need for AAC to support a range of conversational modes, including predictable (chat, needs-based talk), formulaic (script-based talk, jokes), repeatable (conversational narrative) and novel (new vocabulary) conversation [4]. Collaboration with Ritchie (Computing, Aberdeen) and Pain (Informatics, Edinburgh) resulted in a joke generating system, STANDUP [5], for children with speech and language impairment [9].

Based on a social construct of conversation, the PROSE system supported interactive storytelling, allowing users to control the narration of a story instead of delivering a monologue [4]. Waller, Black (Computing, Dundee) and Reiter (Computing, Aberdeen) addressed the issue of automatic experiential data input by using sensor-based data-to-text technology to generate narrative texts for children with speech and language impairment [10],[11]. Environmental sensor data (location, people, objects, voice recordings) were transformed into narrative text using data-to-text technology [6]. Working with Grove (City University) on narrative use with people with speech and language impairments, symbolic narrative interface scaffolds have been designed by Waller to allow disabled children to narrate and embellish their experiences.

Key Dundee Computing Researchers with principal years of involvement since 1993: Alm (1993-present), Arnott (1993-2005, 2010-present), Murray (1994-2000), Newell (1993-2005), Waller (1993-present).

References to the research

(three most significant marked *)

Research Grants

[7] ALADIN: Advanced Language Device for Interaction", EU TIDE Project TP 1035, £224,360, 1994-1997.

[8] REACT: Real-time Communication Terminal", EU TIDE Project DE 4207, £337,841, 1997-2000.

[9] Facilitating language play in non-speaking children through computer-supported joke construction, EPSRC GR/S15419/01, £364,155 pre-FEC, Dundee share £181,218 pre-FEC, 2003-2007.

[10] How was School Today ...? Supporting narrative for non-speaking children, a feasibility study. EPSRC EP/F067151/1 Digital Economy Feasibility Studies, £102,258, 2008-2009.

[11] How was School Today ...? in the Wild, EPSRC EP/H022570/1 Digital Economy Research in the Wild, £156,721, 2010-2011.

Details of the impact

AAC research at Dundee is user-led with strong connections with clinical practitioners, end-users and industry.

Waller's invited presentations to clinical practitioners, e.g. the 22nd European Academy of Childhood Disability EACD2010 conference and the 2013 British Society of Paediatric Dentistry Conference, reflect impact on clinical practice. We have teaching arrangements with special schools and clinical training departments nationally and internationally. Waller and her team teach on the communication modules for medical (first, second, third levels) and dental (final year) students at Dundee University and biannually on the national community dentistry training CPD course run by NHS Education Scotland, training over 250 clinicians since 2006. We run clinical workshops in conjunction with the UK national AAC charity, Communication Matters, and have delivered clinical training to over 150 teachers and therapists since 2008. In a new initiative with Communication Matters we have established a forum which brings together researchers, practitioners and AAC companies in order to reduce the time taken to transfer technology research into the commercial sphere — "collaboration with Dundee's School of Computing has allowed us to develop and enhance the channels through which research is articulated to clinicians, research and design technologists employed within the communication technology (AAC) sector and then end users." [R1]

The STANDUP and "How was School Today?" projects, funded by EPSRC and Digital Economy, have been well received internationally and has resulted in collaboration with leading special education schools worldwide [R2]. Our work has resulted in schools adopting a narrative approach to intervention with speech impaired children and we have established a story telling group in an adult care centre. Our collaboration with Capability Scotland has contributed to their two schools being awarded the highest rating in recent HMI inspection: "Your department's continuing work with our schools in terms of augmentative and alternative communication assisted us in achieving the glowing report from Education Scotland in April 2013 which commended the leadership team at Stanmore House School for their commitment to the use of up to date technology. Your input is also recognised in our services where, for example, augmentative and alternative communication is used to support interactive customer led reviews and the development of personal support plans. By providing the necessary tools and using a mix of high tech communication aids, individuals are empowered and supported to take control and make informed life choices." [R3]

Working in school has resulted in identifying the need to support literacy learning for nonspeaking children. Supported by Capability Scotland, the PhonicStick™ has been evaluated by clinical departments in Manchester, Sweden, Ireland and South Africa. It provides a unique way for nonspeaking individuals to `play' with sounds and blend them into words. Capability Scotland funded this fundamental research and "recognise, as you [Waller] do, that whilst enabling speech to communicate basic needs and wants is important, true communication is far more complex and involved. That is why the Phonic Stick symbolises more than anything else the benefit of our long partnership. It goes beyond mere symbols or words to enable the formation of language and we were therefore proud to sponsor its development." [R3] The Dundee City Education Department are supporting the product development of the PhonicStick™ as a teaching tool because of its "potential to transform literacy opportunities for some of our most disadvantaged young people". [R4]

Waller has established a unique AAC usability review consultancy comprising adults who use AAC. Adults with severe physical and speech impairments volunteer their expertise in using AAC to support the development and evaluation of assistive technologies, undertaking commissions from industry to support the engineering of new devices; clients include top AAC developers. We have developed innovative methods for involving people with severe speech and physical disabilities in the design of assistive technology by adapting techniques to enable stakeholders of differing ages and abilities to engage in design and have developed recommendations for ISO design guidelines.

Close links with international AAC developers have resulted in effective technology transfer. We work with leading AAC companies (e.g. DynaVox, Toby Churchill Limited (TCL), Tobii Technology AB) to commercialise our IP. A patent has been registered for the PhonicStick®, a joystick-controlled phoneme-based word generation tool for children with language impairments, and the IP licensed to TCL in 2012. Waller's work on narrative, licensed to Don Johnston Inc. of Illinois in 1992, has provided much of the theoretical underpinning to the Improv™ AAC system, released in 2012. Don Johnston Inc. state that: "work at University of Dundee is instrumental in pushing the boundaries of AAC by taking a conversational discourse approach to supporting aided communication." [R5]

CONTACT project partner Enkidu Research Inc. merged with DynaVox in 2004. Conversation modelling and pragmatics techniques developed at Dundee were commercialised in the InterAACt language framework from DynaVox Mayer-Johnson of Pittsburgh, the largest AAC company in the world with 395 full-time employees and net sales of over $97 million (fiscal year to June 2012). Speech generating (i.e. AAC) products represented approximately 84% of these sales. "InterAACt is the exclusive language framework used on all DynaVox devices. It allows individuals with significant communication needs to successfully communicate, develop high-level language skills and express themselves, in everyday activities." [R6]

DynaVox products are promoted in the international AAC marketplace (DynaVox lists 25 countries) and are prescribed extensively by clinicians (e.g. speech and language therapists) for their clients. The InterAACt framework therefore reaches a wide client base of people who experience improved AAC support during conversation with others. Other beneficiaries are family members, carers and friends of the people who use InterAACt, as they have more fulfilling conversations with the InterAACt user, and clinicians and therapists who work with non-speaking and communication impaired people. The AAC industry benefits through a significant advance in augmentative communication technology coming to market. Concepts from conversational pragmatics and script-based systems have influenced most modern AAC systems in the form of re-usable utterances and visual scenes. [R7]

DynaVox Mayer-Johnson state that: "Dundee is one of a small number of centers whose ideas have defined the field of AAC as we know it. Your ideas have .... greatly improved the lives of tens of thousands of people. The research done at Dundee has directly generated significant economic value for many organizations in our industry. ... DynaVox's InterAACt language framework makes extensive use of techniques developed at the University of Dundee and which we license from you. This technology provides our customers with the opportunity for greater communication rates while retaining the crucial flexibility and context sensitivity needed so they can truly say what they want to say." [R7]

The projects have also experienced national and international news exposure [R8]. STANDUP has been downloaded more than 1000 times from its project webpage. "How was School today...?" [R9] won the 2010 TES Schools Award for Outstanding ICT Learning Initiative of the Year for project partner Capability Scotland (Corseford School): the "How was School today...?" project "signalled how assistive technology could really open up the world of communication to those who use our [Capability Scotland's] services. We were particularly delighted that it won the only Scottish award in the 2010 Times Educational Supplement's Awards for the School and University." [R10]

Sources to corroborate the impact

[R1] Factual Statement from Communication Matters, the UK Chapter of the International Society for Augmentative and Alternative Communication (ISAAC), September 2012.

[R2] Factual Statement from Lead Speech-Language Therapists, The Children's Institute: The Day Institute, Pittsburgh, PA, USA, February 2011.

[R3] Factual Statement from Chief Executive, Capability Scotland, Edinburgh, October 2013.

[R4] Factual Statement from Head of Education (Secondary and Support for Learners), Dundee City Education Department, May 2013.

[R5] Factual Statement from President, Don Johnston, Volo, IL, USA, October 2013.

[R6] DynaVox website (http://uk.dynavoxtech.com/interaact/).

[R7] Factual Statement from DynaVox Mayer-Johnson, Pittsburgh, PA, USA, January 2012.

[R8] "New software helps children speak". BBC TV news report, GMT Friday 5 June 2009. Shown throughout the day and on 10:00pm news. Corresponding BBC web site report: http://news.bbc.co.uk/1/hi/health/8084422.stm. Local copy (video file, mp4): http://ref2014.computing.dundee.ac.uk/aac/[R8]_BBC_News.mp4.

[R9] "How was my school day? Now disabled pupils can tell the story". The Times, 5 June 2009. Local copy available at: http://ref2014.computing.dundee.ac.uk/aac/[R10]_The_Times.pdf

[R10] The Times Educational Supplement website: Winner of the "Outstanding ICT Learning Initiative of the Year" (http://www.tes.co.uk/article.aspx?storyCode=6047969&s_cid=winners2010)