Summary of the impact

Augmented reality supplements real-world experience by placing graphics over a live camera feed in real-time. Smartphones are a popular and suitable device (ubiquity, portability, and sensor features). However embedded industry practices delivered an inferior experience limiting adoption. Maguire's research-informed innovative integration and implementation of visual effects enabled a high-end cinematic experience. This included integration of user's environment, interaction with characters, motion blur, 3D realistic character, facial animation, exponentially increased capacity for customized narrative and new audiences. The outcomes also expanded the aesthetic and genre of the narrative, sharing user experience and thus reaching significant new user demographics.

Underpinning research

Maguire joined the University of Ulster as Professor of Animation in March 2010 and in May that year identified potential to use the unit's research and 'art school' environment to investigate strategies and technologies in order to connect areas of animation that are unconnected in industry. Since their introduction, character based applications on smartphones have been captivating audiences worldwide. Maguire hypothesized that their novelty would eventually diminish as audience sophistication and expectations increased. His holistic research therefore aimed not just to develop solutions to technical challenges, but also to widen emotional vocabulary and increase realism. His initial research, leading to Fairy Magic in 2012, revealed:

• Unrealised capacity in the smartphone platform.
• Chasm between cinematic characters and those of mobile games.
• Burgeoning mobile market potential for new and neglected audiences.
• Limited character interaction and repeating narratives in existing products.

Although the mobile phone presented itself as good candidate for 3D animation, having an advanced graphics processor that supported the graphic standard OpenGL ES, most applications prior to his research in 2010 were rendered as simple 2D sprite animations or pre- rendered movie sequences. Hardware bandwidth of mobile devices restricted the direct implementation of cinematic technologies required for realism. He observed that visual effects on Avatar involved a technically complex off-line workflow and a large-scale computer infrastructure to generate one frame in 48 hours. Effective real-time phone graphics needed to deliver one frame every 0.01 seconds. Maguire developed new ways to replicate film industry effects using untapped real-time capabilities of the smartphone to create more appealing and engaging visuals. His research identified key areas of opportunity:

• Facial expression; although appropriate for animation techniques delivered by existing applications, was limited. In order to increase successful emotional engagement a more comprehensive and modular range of emotion would be required. Maguire employed his own research into the use of the Facial Action Coding System (Ekman, Friesen, 1978) in animation for film to create a system that enabled every facial movement possible in the mobile application, Fairy Magic.
• Real-time animation; existing character animations limited to pre-canned movies, restricted narratives and user interactions. Assessment of untapped capacity in mobile devices, together with skills and knowledge from two different domains, informed a completely different method, real-time 3D character rendering. In Fairy Magic, Maguire applied his new creative concept to prove that it was possible to bring 100's of modular based interactive animations to mobile apps.
• Image Based Lighting; Maguire recognised that contemporary computer lighting models that added more realism to film were beyond the hardware capabilities of the mobile platform. In existing augmented reality applications the graphic layer placed over the camera had no cognition of the background. In Fairy Magic, Maguire researched how to light real-time characters with an unpredictable background by replicating Image Based Lighting.
• Motion Blur; the research uncovered motion blur as a problem with fast moving animations. He investigated three existing techniques 1) real-time graphics delivery on desktop PCs, 2) Monte-Carlo stochastic sampling in rendering for film, and 3) traditional 2D Cel animation. His innovation drew on these unconnected practices to deliver on a method to mimic motion blur on mobile devices for repetitive motion. This was implemented in Fairy Magic and the underpinning research was shared with industry.

References to the research

Details of the impact

As the visual effects global talent pool increased and more favourable tax systems came into effect elsewhere, production moved from its roots in the USA to other parts of the world. Faced with an uncertain future, visual effects artists either lobbied the US government for trade protection or moved with production. This research sought to explore another opportunity; `Fairy Magic' became a vehicle to apply research and knowledge gained in visual effects to a different sector and illustrated an effective route to other markets.

Maguire used techniques appropriated from film deemed too computationally expensive for mobile by finding ways to mimic their visual properties and not their direct implementation. In Fairy Magic he implemented a facial animation system based upon FACS, a comprehensive, anatomically based system for describing all observable facial movement to expand the emotional vocabulary of previously existing applications creating more expressive character interaction. More empathy with a character would lead to more appeal and make a contribution to affective computing.

Prior to his research, mobile animations were sprite based or memory intensive pre-rendered movie files thereby limiting the narrative and character interaction. Maguire shifted from this accepted industry standard to a more flexible method and increased interaction and narrative by order of magnitude. In Fairy Magic, Maguire implemented real-time characters with 100 modular based animations that could be blended together using human interaction and controlled randomisation to create 1000's of combinations across multiple characters with a much smaller footprint. This enabled room for `hero' animations. One such sequence drew upon the first encounter between Tarzan and Jane in Walt Disney's animated classic. When the user holds their finger on the screen for a period of time, a fairy will come and gingerly touch then hold their finger.

Maguire recognised that Image Based Lighting (IBL), were each pixel of an image, typically a hemispherical environment, is treated as a light source to illuminate a scene with thousands of lights, was beyond the three light limit imposed by OpenGL ES. However, to embed the character within the camera, environmental lighting would need to be considered. Fairy Magic, recreates this visual aesthetic by sampling pixels either side of the cameras CCD and inputting their colour values into dynamically lighting the real-time characters. Covering the lens and observing the lighting change on the character can demonstrate this.

Motion blur, an artefact of mechanical film capture and now part of film lexicon is implemented in computer graphics using Monte Carlo stochastic sampling (Cook, Carpenter, Catmull, 1987]. It however, relies on interrogating vectors over several frames precluding its use on mobile. Motion blur on fast animations such as beating wings, were recreated by using multiple objects containing several static wing positions replaced in rapid succession to create this cinematic illusion.

In September 2012, one week after `Fairy Magic' was launched, game developer Pocket Gems, a game developer based in San Francisco with 2 of the top 3 grossing applications in the Apple App Store, reached out to Maguire to discuss collaboration to bring Fairy Magic research to their 80 million users. On November 8th 2012, the Fairy Magic team signed an agreement with Pocket Gems to co-develop a game using the research to be realised in 2014.

The expressive facial animation of Fairy Magic drew the attention of Nvidia Corporation, the world leading design and manufacturers of graphics processing units for workstations and the mobile market. Co-founder and CEO, Jen-Hsun Huang invited Maguire to disseminate his research during Huang's keynote presentation at GTC in March 2013. The outcome `New Dawn', showcased their cutting-edge GPUs and provided Maguire at University of Ulster with an opportunity to develop a realistic human character using FACS based shape targets at a range never previously accomplished.

NESTA concluded that the Creative Industries would become a key driver for the UK's recovery from recession taking centre-stage as a major, high-growth contributor to the UK economy over the next 5 years. Commenting on `Fairy Magic', Richard Williams, Chief Executive of Northern Ireland Screen, said `we need to grow that good practice and do a lot more of it' (CAL Enquiry 2013). `Fairy Magic' and `New Dawn' research sought to engage with visual effects artists between Europe and the USA, to expose the UK's growing animation workforce to a globally collaborative environment whilst equipping them with the right skills to innovate, create and communicate remotely.

Local industry benefited from the founding of an animation cluster, Toody Threedy. Initially, a vehicle to disseminate knowledge and research it has become a hub for Foreign Direct Investment as studios source talent for Game of Thrones (HBO, 2012), Dracula (Universal, 2014) and Edge of Tomorrow (Warner, 2014). With over 250 members it has increased opportunities and built trust between Northern Ireland's animators and their global peers. Flexible learning programmes delivered by the `Fairy Magic' crew's newly acquired knowledge, standardised NI's animators on a single platform enabling companies to interoperate and collaborate on projects. The media coverage (such as; The Arts Show (2012) TV, BBC, BBC2, 22 November 8pm) surrounding the launch of `Fairy Magic' promoted animation as a career opportunity to a wider audience including children aged 10-18 with the formation of 3D Dojo, an animation branch of James Whelton's Coder Dojo. These initiatives have embedded the universities research and dissemination of knowledge as a key driver of economic development. Games play a major role in learning and shaping our worldview. Board games are social devices enabling us to interact and role-play with others. Pong, the first commercially available video game had two paddles by default. As games developed with limited computing power single player games against the computer evolved they reduced our gaming to a solo experience. The internet enabled us to play with limited interaction but with a global audience. Augmented Reality by the nature of its camera view brings us full circle to a more social shared experience. Looking through `Fairy Magic's' camera you see your friends, your family, your environment to create a hyper-local experience strengthening social bonds.

During the examination and exploration of the smartphone platform and while reproducing a cinematic experience, it became apparent that the device could offer more than just a portable screen. With a host of sensors continually gathering data in real-time, including accelerometers and a compass it would be possible to place a real-time character anywhere in space. However, this could place the characters in an inaccessible location behind the user. A home location had to be developed to attract the character to a position in front of the phone enabling the user to always have the character in view. As the character could appear anywhere on the screen, a traditional user interface (UI) locked to screen space proved ineffective. The UI was therefore decoupled from screen space and forced to follow the characters centroid ensuring all navigation controls were relative to the character. E.g. Left became left of character and not left of screen. The screen was not treated as a portal within the application but a sheet of glass. The character's size was a consistent 10 cm across all devices and screen sizes. This contributed to the character's existence in our world.

An intimate connection, modelled after Tarzan and Jane's first meeting by Glenn Keane (Tarzan, 1999) was created when the user held their finger on the screen. The character would reach forward and touch the users finger as the smartphone's vibrate hardware was called upon to create a tickle.

The phones front camera was used to pull the character into the experience. When the device was laid flat, the camera input was switched from the user's point of view to the point at the user themselves. The image was projected onto a polygonal height field allowing the user to see themselves reflected in a pool along side the characters increasing the illusion of their existence.

Sources to corroborate the impact

The following are in the public domain- the citations recognise the innovation and impact.

1. The Guardian `Fairy Magic', `10 Best iPhone and iPad apps this week'. August 12^th. http://www.theguardian.com/technology/appsblog/2012/aug/20/best-iphone-ipad-apps-wwe- fleksy
2. Irish Technology Leadership Group `Fairy Magic' Winner Best Game 2012, Innovation in Entertainment Summit, Hollywood, CA, USA. http://www.siliconrepublic.com/new-media/item/29458-ireland-at-the-dazzling/
3. The Next Web Pick of the Day. (August 16^th 2012) TNW has 7.2M monthly visits and recognized `Fairy Magic is a spellbinding augmented reality iOS game for kids'. http://thenextweb.com/apps/2012/08/16/tnw-pick-day-fairy-magic-spellbinding-augmented-reality-ios-game-kids/
4. Venture Beat (5.5M monthly unique visitors) made public "Nvidia touts its latest graphical sorcery in `New Dawn' tech demo" that confirms the ground-breaking research and rapid adoption by industry. http://venturebeat.com/2012/07/27/nvidia-new-dawn-tech-demo/
5. Jen-Hsun Huang, CEO Nvidia Corporation, Keynote speech at GPU Technology Conference, San Jose, CA, USA (2013) presenting `A New Dawn". http://www.youtube.com/watch?v=CvaGd4KqlvQ

6. Director of Strategic Delivery, Department of Culture Arts and Leisure, UK. DCAL is the NI government lead on the creative industries — will confirm the impact the research has had on regional economic and cultural policy and strategy including Programme for Government.
7. Chief Executive, Northern Ireland Screen, UK — will confirm the new audiences and impact on open access technology. It's impact on building an indigenous sustainable creative industry and the attraction of Foreign Direct Investment to the UK.
8. Former Head of Global Operations, Lucasfilm Ltd, USA. (Now at Telltale Games) will verify the impact of the technology and the workflow and Will confirm that the research is a stunning example of the possibilities for modern mobile platforms.
9. Set Dress Lead, Pixar Animation Studios, USA Will confirm the innovative nature of the use of visual effects techniques being appropriated for a mobile platform and has "all the hallmarks of becoming a standard lighting model for all AR applications."
10. Animation Supervisor on New Dawn, Nvidia, USA will confirm they adopted the research in `New Dawn', how the research has changed the perception of, and realization of limitations of interactive experiences on mobile devices.