Summary of the impact

Researchers at Oxford Brookes University have shared their computer vision expertise to help enhance interactions in augmented reality (AR) systems, directly contributing to the development of Sony's Wonderbook. This has facilitated economic and cultural impacts demonstrated through delivery of AR books to the gaming community. Through the understanding and utilisation of the complex mathematical theory behind computer vision, the partnership were able to create a robust human hand tracker and segmenter that could calculate the position of the player's hand and segment it in a live video in real-time. The result was a display that looked more natural and quite literally enabled books to `come to life' on the screen in dramatic new ways that can be used equally for entertainment and education.

Underpinning research

Working with Sony Computer Entertainment Europe in a Knowledge Transfer Partnership (KTP),¹ Oxford Brookes University, led by Professor Philip Torr (Oxford Brookes University 2004-October 2013), worked to enhance the experience of Augmented Reality (AR). This research led approach also helped to provide a Natural User Interface (NUI) for interactions in AR systems. The target platform was the recently released Sony's Wonderbook, which is an AR book for PlayStation 3 (PS3).

In Wonderbook, the player interacts with the game by using a physical book which is tracked by utilizing an EyeToy camera (PS3's camera peripheral). Using the tracked location of the book and its pages, virtual game graphics are augmented onto the screen to create a virtual pop-up book where the player can see the blend of reality and a magical pop-up book. The research centres on work done for hand detection and tracking; efficiency and speed are key constraints in solving these problems in real-time for computer games applications.

The aim of the KTP was to create a robust human hand tracker and segmenter, which would calculate the position of the player's hand and segment it in a live video in real-time. This could then be used as a game input mechanism for the Wonderbook. The main technical challenges to overcome were;

• Tracking the player's hand in variable lighting, and variable skin colours,
• Detecting and then reinitialising if the tracking failed,
• Getting this running in under one 30th of a second per frame of video on a PlayStation3 platform.

Many existing AR systems are not able to identify human elements present in the places where they augment virtual content and hence, the graphics are usually overlaid on all the surfaces being tracked by the system. In case of Wonderbook, this means that if the player had their hand over the book in the physical world, it would appear on the screen that their hands are below the book (as the contents will be overlaid on top of the entire book surface including their hands). In order to enhance the experience of interacting with a virtual pop-up book, the researchers created a set of algorithms which are able to identify which parts of the image belong to player's skin and use those regions to prevent the augmented content to be overlaid. This results in a display which looks more natural, resembling what you would expect when using with a normal book.

The problem of identifying which parts of the image belong to player's skin is a very challenging problem, mainly because of the wide diversity in skin tones, as well as what the skin looks like under a range of illuminations in living rooms. In order to tackle this problem statistical machine learning and computer vision techniques were used to analyse videos of a diverse set of players interacting with Wonderbook under different illumination conditions. As this system needs to run in real-time and due to the large amount of data needed to be processed, thus algorithms were created which are very fast to evaluate and use the minimal amount of computation time and memory².

References to the research

1. KTP007878 March 2010 — March 2012 `To segment hands in a video image and recognise simple gestures to enhance Sony games' Technology Strategy Board & Sony Computer Entertainment Europe, £179, 236.
http://info.ktponline.org.uk/action/details/partnership.aspx?id=7878

Details of the impact

The research-led approach to enhance the experience of an AR application, including a Natural User Interface (NUI) for interactions, has demonstrated economic and cultural benefits through enabling the delivery of AR books on Playstation platforms, such as the Wonderbook.

By bringing in knowledge of how computer vision research in universities is conducted, Dr Amir Saffari was able to recommend structural changes to the SCEE team and a new set of tools for implementation. Sony Computer Entertainment Europe reported that they saved a significant amount of time and money in development and research efforts. The partnership also enabled the company to bring in more computer vision expertise which would not have been possible without this collaboration. At the end of the project they recruited the Associate, Dr Amir Saffari, to their permanent staff; making a total of three permanent researchers with PhDs in vision, and also funding three PhD students at the university.⁴

The Knowledge Transfer Partnership succeeded in creating a hand tracker which surpassed the performance of Sony's existing technology for solving this problem⁴. Wonderbook is a physical book that interacts with the PlayStation 3 via a camera, and allows the player to control the computer through natural hand gestures; part of maintaining the illusion of an AR book is knowing where the player's hands are when they cover the book so that content is able to interact with the hands and be occluded from view when covered. In 2011, Sony signed a deal with JK Rowling to extend the Harry Potter franchise into its own content. The first implementation of this was the `Book of Spells' from the World of Harry Potter series, complete with new writing by the author, released in November 2012.

Of her Wonderbook Book of Spells, J.K. Rowling said:

`It's the closest a Muggle can come to a real spellbook. I've loved working with Sony's creative team to bring my spells, and some of the history behind them, to life'⁵.

The Oxford Brookes vision technology was used in Book of Spells and Professor Torr and Oxford Brookes receive one of four external (to Sony) credits in the game book;

"The skin segmentation system in Wonderbook was developed through research by Professor Philip Torr and the Computer Vision Group at Oxford Brookes University" ⁶.

Wonderbook was announced in June 2012 by Dave Ranyard of SCEE at Sony's press conference during the 2012 Electronic Entertainment Expo (`E3')⁷, the World's biggest gaming conference, in Los Angeles with positive reviews "...one of the best uses of AR that I've seen to date." Andy Robertson, The Daily Telegraph 15 June 2012; "Wonderbook really does look like a storybook come to life, with an art style that makes everything look like it's made out of paper." Sarah LeBeouf, The Escapist 6 June 2012; "Another cool aspect of the tech is how it manages to display computer generated visuals below portions of the live footage." iWaggle3d.com, 9 June 2012; "...the book is an excellent piece of kit, as is the software that supports it,..." Grant Howitt, The Guardian GamesBlog, 16 August 2012.⁸

The technology developed from the research insights enabled through the Knowledge Transfer Partnership will also be applied in forthcoming Sony Wonderbook titles such as Walking with Dinosaurs and the next in the Harry Potter series `Book of Potions'⁹.

The insights rooted in the excellent research of Professor Philip Torr, with Dr Amir Saffari, have contributed to SCEE using enhanced AR technology in their products — "taking story-telling to a whole new level' and further demonstrating the value of complex mathematic theory in areas far removed from academia.

Sources to corroborate the impact

4. Corroborating contact 1. Head of London Studio Vision R&D at Sony Computer Entertainment Europe.
5. `Wonderbook: Book of Spells' http://www.sony.com/pottermore/us/book-of-spells
6. Wonderbook credits, shipped with every product (example available from Oxford Brookes University Research Support Office)
7. `E3: Playstation gets augmented reality Harry Potter book' BBC News 5 June 2012
   http://www.bbc.co.uk/news/technology-18327724
8. Selection of reviews of the Wonderbook at its AR technology

a. `E3 2012: Wonderbook preview', The Telegraph 15 June 2012
http://www.telegraph.co.uk/technology/video-games/e3/9333219/E3-2012-Wonderbook-preview.html

b. `E3 Preview: Wonderbook: Book of Spells' The Escapist Magazine 6 June 2012
http://www.escapistmagazine.com/articles/view/editorials/reviews/previews/9695-E3-Preview-Wonderbook-Book-of-Spells

c. `Wonderbook: Augmented reality storytelling' iWaggle3D.com 9 June 2012
http://www.iwaggle3d.com/2012/06/e3-2012-wonderbook-augmented-reality.html

d. `Wonderbook — hands-on preview' The Guardian, 16 August 2012
http://www.guardian.co.uk/technology/gamesblog/2012/aug/16/wonderbook-hands-on-preview-ps3

9. `Wonderbook Walking With Dinosaurs and Book of Potions trailered' VG24713 June 2013
   http://www.vg247.com/2013/06/13/wonderbooks-walking-with-dinosaurs-and-book-of-potions-trailered/