Summary of the impact

The primary impact of our research has been through systems that provide information for the deaf community via animated sign language. Tools exploiting our research into automatic generation of human motion, especially for sign language, are used in several ways:

• as a mechanism for face-to-face communication between deaf and hearing people;
• for the provision of public information in sign language in a semi-automatic way;
• to display the content of web sites in sign language;
• to provide sign language for TV programmes.

The same avatar (animated character) technology has also been applied in two other arenas: as a foreign language learning resource for primary school children and to add value to cultural heritage presentations.

Underpinning research

Interactive information in content using human gestures cannot be effectively generated from video material; this is particularly true of sign language. An attractive alternative approach uses 3D character animation techniques.

The Virtual Humans Group at UEA undertakes distinctive research bringing together computer scientists with a wide range of expertise: speech and language recognition, computational linguistics, AI, and computer graphics for 3D character animation. Over a period of 14 years, dating back to 1999, this Group has researched advanced techniques for general animation with speaking characters, including scripted animation of 3D characters supporting animation of deaf sign language. Such animations must not only appear convincing, but must be fully comprehensible to those using only the visual channel for communication.

Initially `Simon the Signer' (1999: Bangham, Glauert) explored translating text subtitles to animated sign sequences. Linguistic analysis drove the presentation of motion-captured signs in English language order, which resulted in natural-looking animations. `Simon the Signer' won two Royal Television Society awards in 1999. However, although the system was an important step forwards, British Sign Language (BSL), like all sign languages, has its own syntax and grammar which is only loosely related to the syntax of spoken English.

The TESSA project (1999: Cox, Lincoln) developed a speech recognition system to identify phrases spoken by a counter clerk for presentation to deaf customers in true BSL, which was an important advance [1]. The key insight was that the discourse between deaf customer and counter- clerk is highly constrained in topic and scope, so that the vocabulary and syntax required can be largely pre-defined [2]. TESSA won the BCS Gold Medal (2000) and was exhibited in the Science Museum, Kensington in July 2001.

This early work built sequences from motion-captured signs and achieved realistic animation for fixed phrases. However, signs translated from spoken language undergo considerable modification/inflection in context. These issues were addressed by techniques developed in the ViSiCAST project (2000: Glauert, Bangham, Cox, Kennaway, Marshall, Safar, Eliott), which produced what is still the most mature system anywhere for translating spoken language to sign language, using a Discourse Representation Theory semantic representation, and Head-Driven Phrase Structure Grammar rules for sign language generation, for three national sign languages [3]. A sophisticated treatment is made of linguistic features in sign language such as handling of plural verbs, management of signing space for anaphoric references, and use of non-manual movements (such as eyebrow position) to match sentence mode. The output was based on Hamburg Notation System (HamNoSys), a recognised `phonetic' transcription system for signs [4]. Two further EU-funded projects, eSIGN (2002 Glauert, Cox, Lincoln, Elliott) and Dicta-Sign (2009 Glauert, Elliott, Kennaway, Safar), developed novel algorithms to synthesise natural motion data for 3D characters from HamNoSys. Planning for hand shape, orientation, and location for the initial sign posture, leads to generation of movement and hand shape changes for subsequent transitions and postures making up a sign. Inverse kinematics generates the necessary arm positions and realistic dynamic trajectories create natural-looking motion [5].

These projects created a bespoke avatar development and animation toolkit to support several cross-platform applications and browser plugins for interactive real-time virtual signing [6]. Conventional 3D characters are enhanced with information on key body sites linked to abstract locations in HamNoSys transcriptions and morph targets for facial expressions used in signing. Avatars and animations can be exported to industry-standard tools (Maya and 3ds Max).

Key Research Personnel

Lead academics: Professors John Glauert; Stephen Cox; Andrew Bangham; Dr Ian Marshall; Dr Ralph Elliott

Research Associates: Richard Kennaway; Mike Lincoln; Eva Safar

References to the research

(UEA authors in bold)

External funding supporting the research

1. Simon the Signer
Bangham, Glauert; Independent Television Commission (1999-2000) £26K

2. TESSA
Cox; Post Office (1999-2002) £203K

3. ViSiCAST
Glauert, Bangham, Cox; EU FP5 (2000-2002) £563K with additional funding of £519K from ITC, Post Office, BBC

4. eSIGN
Glauert, Cox; EU eContent (2002-2004) £168K

5. LinguaSign
Glauert; EU Socrates Lingua 2 (2006-2010) £93K

6. Dicta-Sign
Glauert; EU FP7 (2009-2012) £516K

Details of the impact

The primary beneficiary of our research into the automatic generation of human motion has been the deaf community, in the UK and elsewhere. The systems developed at UEA have been applied in a variety of situations to improve both direct communication and information provision. In addition, the avatar systems have been used to provide a new way of teaching languages in primary schools and to add an additional dimension to cultural heritage sites.

1. Avatar Signing for the Deaf Community
   For many prelingually deaf people, their first language is natural sign language, with approximately 50,000 people in Britain having British Sign Language (BSL) as their first language. Our algorithms and 3D animation technology have been developed to enable rich natural sign language to be generated for communication with deaf people. Systems arising from this research provide deaf people with the means to access information and services in their chosen language, respecting the cultural and social needs of a recognised minority community.

Our systems allowing face-to-face communication between deaf and hearing people, with speech recognition and the subsequent automatic generation of signing were first used by the Post Office in 2000. During the current REF impact period, this technology has been taken forward by IBM through Say It Sign It (SiSi), a real-time system that allows sign language users to communicate with hearing people by converting speech and text into natural sign language. A further important development is the modification of this system to allow use on mobile devices, which involves Orange working alongside IBM. IBM have stated (corroborating source A):
`IBM chose the virtual signing technology developed at UEA as the most advanced and flexible system available.'

Semi-automatic signing based on pre-defined phrases for information provision is an important aspect of the impact of this technology for the deaf community. This has been used in train announcement systems and weather forecasting where the required phrases are from a limited set and the key feature is the ordering that is required to produce the appropriate information. The ViSiCAST project developed a tool to allow non-signers to build weather forecast summaries using drop-down menus to choose phrases covering a full range of general weather conditions. Weather phrases have been mapped to natural British (BSL), Dutch (NGT), and German (DGS) sign language, along with corresponding text.

Working with the Royal National Institute for the Deaf (now Action on Hearing Loss), we have developed resources that enable authoring of sign language content for websites. eSIGN has produced numerous websites utilising our technology to provide sign language as a key method of communication for non-hearing customers. Key examples include web sites on employment in the Netherlands (IvD), deaf services in the UK (Deaf Connections) and for the Federal Ministry of Labour and Social Affairs (BMAS) in Germany (see corroborating source B).

A closely related application is the provision of information for attendees at public festivals or events. A recent example is the successful use of the JASigning software, which is based on UEA research, at the Visual Festival organised by the Swiss Association of the Deaf in May 2013. The Festival organiser has stated (corroborating source C) that:
`the technology provided a novel and attractive way to communicate information effectively for use by deaf participants.'

With signed-TV, the aim is to use 3D animation techniques to supplement the use of on-screen human interpreters. For example, in conjunction with Gamelab London we developed Performing Hands, a range of interactive educational materials for the BBC which uses virtual signing to support learning via interactive games. This resource was a finalist for a BAFTA award in 2008. The same technology has been used for programmes commissioned by the British Sign Language Broadcasting Trust. Examples can be seen in on-line programmes such as Wicked Kids and Little Wicked (see corroborating source D). Gamelab (corroborating source E) say:
`these resources are unique animated materials for use by deaf children and all depend on the avatar technology developed at UEA.'

and the British Sign Language Broadcasting Trust (corroborating source F) state:
`production makes substantial use of the virtual signing avatar technology created by researchers at UEA.'

The production of signed-TV is on-going and our technology is used by the Netherlands Organization for Applied Scientific Research (TNO) to produce accessible programmes for deaf children for Dutch TV, such as the popular programme Het Zandkasteel. TNO have stated (corroborating source G):
`We identified the UEA technology as the most suitable for our needs as it enables us to produce cost-effective 3D animations that are attractive to children.'
2. Foreign Language Learning
   Our virtual signing technology has other applications where there is a requirement to combine gestures and speech. Resources for primary language education in the form of animated stories on DVD are available under the brand LinguaSign (see http://www.linguasign.com/). These stories combine standard animation for the main story line alongside speech. The story characters speak with lip-sync animation while performing gestures that reinforce the acquisition of vocabulary. The stories have been reproduced using matching phrases in English, Dutch, French, and Portuguese, voiced by native speakers. In 2013, more than 50 primary schools undertook a trial of the resources for Key Stage 2 children. 62% of respondents to the survey at the end of the pilot strongly agreed that the DVD was effective at improving pupil's speaking and listening skills in a new language (corroborating source H) whilst one teacher commented:
   `My class and I really enjoyed using it. We found it engaging and fun. A great way to prepare them for MFL [Modern Foreign Language Learning]at high school.'
3. Cultural Heritage
   In a further use of our technology, exploiting animation scripting algorithms from the virtual signing system, animated characters have been used to provide engaging stories to enhance the presentation of cultural heritage sites managed by the Heritage Economic and Regeneration Trust (HEART). The CEO of HEART has stated that (corroborating source I):
   `The technology developed by UEA, and utilised by us, has made a real difference to the way in which we can present our cultural heritage to a wide range of audiences and has enabled us to engage with particularly hard to reach groups.'

Sources to corroborate the impact

[A] Letter from IBM confirming that IBM and UEA are developing the SiSi system, held on file at UEA

[B] Letter from BMAS (German Federal Department for Social Affairs), held on file at UEA

[C] Letter from Visual Festival 2013, held on file at UEA

[D] Wicked Kids and Little Wicked programmes available to view on-line at:
http://www.bslzone.co.uk

[E] Letter from Gamelab London, held on file at UEA

[F] Letter from British Sign Language Broadcasting Trust, held on file at UEA

[G] Letter from Netherlands Organisation for Applied Scientific Research, TNO, held on file at UEA

[H] Analysis of SurveyMonkey results following completion of the primary schools pilot 2013, held on file at UEA

[I] Letter from Heritage Economic and Regeneration Trust (HEART) Chief Executive Officer, held on file at UEA