ICARUS – Interactive Construction of 3D Models from Digital Images

Submitting Institution

University of Manchester

Unit of Assessment

Computer Science and Informatics

Summary Impact Type

Technological

Research Subject Area(s)

Information and Computing Sciences: Artificial Intelligence and Image Processing


Download original

PDF

Summary of the impact

In the late 1990s, a significant barrier to the adoption of virtual reality software was the expense of manually creating models of real-world scenes. To address this, between 1998 and 2004, the ICARUS software system was developed, which enabled the creation of structured, 3D geometric models from a sequence of images or video. The system also pioneered improved methods of camera tracking. ICARUS was subsequently licensed and developed commercially, and became the foundation for video and film post-production products that are used worldwide in the film (e.g. Universal Pictures, Warner Bros, Paramount Pictures) and television (e.g. BBC) industries, underpinning a company with an annual turnover in excess of £1m.

Underpinning research

ICARUS was created to address the problem of building geometric models of real-world scenes directly from digital images. Camera calibration and tracking are central, not only to constructing 3D models, but also to video and film post-processing. Accurate camera calibration enables automation of special effects, such as the addition of virtual objects, removal of unwanted objects and artefacts, and compositing of separate sequences. It involves computing the camera's extrinsic parameters (position and orientation) for each image in a sequence and its intrinsic parameters (focal length, aspect ratio, principal point and skew). A key contribution of the research was a new hierarchical, two-pass tracking algorithm [1, 2, 5] based on three innovations (see [2] for details):

  1. The use of estimates of inter-frame camera motion to guide the feature tracker, greatly reducing the number of incorrectly tracked features.
  2. A reliable approach to projective reconstruction, using carefully selected subsequences of the video data, and hierarchical algorithms to merge subsequences into a single reconstruction. Previous methods used feature-matching algorithms that were sensitive to drift over a video sequence, leading to poor reconstructions, particularly for wide-baseline, or lengthy, motion sequences.
  3. The application of random sampling algorithms to the problem of self-calibration, allowing for more accurate upgrades from projective to metric geometry. The research showed that applying random sampling techniques to the problem of self-calibration significantly increased the usefulness and applicability of the basic linear algorithm, especially for longer video sequences.

The result of these innovations was an accurate and reliable camera-tracking algorithm that could be used in a wide range of applications (e.g., [3,4,6]).

Key researchers:

S. Gibson (RA, 1995 - 2004); now Research Director, The Pixel Farm Ltd).

J. Cook (RA, 1995 - 2004); now Senior Software Engineer, The Pixel Farm Ltd).

T.L.J. Howard (Lecturer, SL, Reader, 1988 - date).

R.J. Hubbold (SL, Professor, 1985 - 2010).

D. Oram (PhD student, 1997 - 2001).

A.J. West (Lecturer, 1988 - 2001).

A.D. Murta (Lecturer, 1990 - 2001).

References to the research

The research was published in internationally recognised conferences and journals, with several of the publications being well-cited.

Key references

[1] S. Gibson, R. J. Hubbold, J. Cook, and T. L. J. Howard. Interactive reconstruction of virtual environments from video sequences. Computers and Graphics, 27(2), April 2003, pp. 293-301. (Best paper award.) DOI:10.1016/S0097-8493(02)00285-6 (Google Scholar: 30 citations)

 
 
 
 

[2] S. Gibson, J. Cook, T. L. J. Howard, R. J. Hubbold, and D. Oram. Accurate camera calibration for off-line, video-based augmented reality. In IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2002), September 2002. Darmstadt, Germany. IEEE Press, ISBN: 0-7695-1781-1. DOI: 10.1109%2FISMAR.2002.1115068. (Google Scholar: 107 citations)

 
 
 

Other references

[3] S. Gibson, T.L.J. Howard and R.J. Hubbold. Flexible image-based photometric reconstruction using virtual light sources. Proc. Eurographics 2001, Computer Graphics Forum, 20(3), September 2001, pp. 203-214, Blackwell Scientific. DOI:10.1111/1467-8659.00513 (Google Scholar: 24 citations)

 

[4] A.D. Murta, S. Gibson, T.L.J. Howard, R.J. Hubbold, and A.J. West. Modelling and rendering for scene of crime reconstruction: A case study. In Proc. Eurographics UK 16th Annual Conference, pages 169-173. European Association for Computer Graphics, March 1998. ISBN 0-952 1097-7-8. (Google Scholar 16 citations)

[5] S. Gibson, J. Cook, T.L.J. Howard, R.J. Hubbold. ICARUS: Interactive Reconstruction from Uncalibrated Image Sequences, Proc. ACM SIGGRAPH 2002, Sketches and Applications, ACM Press, July 2002. ISBN 1-58113-525-4. DOI: 10.1145/965400.965427 (Google Scholar: 8 citations)

 

[6] S. Gibson, J. Cook, T.L.J. Howard, and R.J. Hubbold. Rapid shadow generation in real-world lighting environments. In Rendering Techniques 2003 (Proceedings of the Eurographics Symposium on Rendering 2003), Leuven, Belgium, June 2003. (Google Scholar: 73 citations)

Details of the impact

Context

Camera tracking, as provided by PFTrack, is central to modern film post-production, enabling the merging of real footage with computer-generated effects, and the removal of unwanted objects visible in the footage, such as unwanted street furniture, telegraph poles, etc. Accurate camera calibration enables a seamless inclusion of such changes. By extracting 3D structure from footage, background detail can be re-inserted where objects have been deleted. Tracking also underpins other areas of application, such as removal of scratches from archive film.

Design and development of ICARUS commenced in 1998. At that time, pioneering work on automatic tracking and reconstruction was underway at the University of Oxford in the UK (Zisserman, Hartley et al.), and the University of Leuven in Belgium (van Gool, Pollefeys). Whilst this work had laid the foundations, software capable of constructing real environments with structure (e.g. named parts), or reliably tracking commercial video and film sequences, did not really exist in a usable form.

Pathways to Impact

By 2002, considerable interest was being shown in ICARUS by the research community and by companies, and a decision was reached to release a free, binary-only version of the software for non-commercial use. In the first three months, this received over 10,000 downloads. In 2003, ICARUS was licensed to The Pixel Farm, a UK start-up company, for exclusive use in the film and video post-production industries, and the free version of the software was withdrawn. Initially, the University provided consultancy and software support to the company. In September 2004, Dr Simon Gibson (RA), the originator of many novel aspects of ICARUS, joined The Pixel Farm as their Research Director, and Jon Cook (RA) joined as a software engineer; both still work for the company.

Notable early successes were the use of ICARUS — by then rebadged by the company as PFTrack — in the Academy Award winning film Cold Mountain (Miramax Films, 2003) [A], and the film Fantastic Four (20th Century Fox, 2005).

Reach and Significance of Impact

Today, PFTrack is used extensively in industry in the USA and Europe. Recent film credits (within 2008-2013) include: Watchmen (Warner Bros, Paramount Pictures), the Harry Potter films (Warner Bros), Battle Los Angeles (Columbia Pictures), The Boat that Rocked (Universal Pictures), The Hunt for Gollum (Independent Online Cinema), and TV series such as Smallville (BB Studios, Canada for USA TV networks), Doctor Who (BBC), Grandpa in my Pocket (Dinamo for BBC), The Wrong Door (BBC), Kia Motors Pro_Cee'd TV advert (The Mill, UK). Details of these and other examples where PFTrack has been used can be found at [B]. PFTrack is also used at the UK's National Film and Television School in its Digital Effects MSc course [C]. Examples of high grossing films during the REF period in which PFTrack has been used include Battle: Los Angeles (over $200m) and Watchmen (over $185m) [D].

The Pixel Farm has grown to become a leading provider worldwide of this type of software within a highly competitive industry, employing 10 people, and with a turnover exceeding £1m annually [E]. Major partner companies include Cooke Optics and FilmLight [E].

It should be noted that this list is partial: many companies in the film industry are secretive about the post-production facilities and software that they employ and will not allow any publicity, even under a non-disclosure agreement.

Sources to corroborate the impact

Supporting material is available from the university for the corroborating sources below.

[A] http://www.4rfv.co.uk/industrynews.asp?id=22201 (dated November 2003, accessed 21/02/2013). Confirms the use of PFTrack on Cold Mountain, and provides details on how it was used.

[B] For details of projects using the PFTrack software, see the company's website: http://www.thepixelfarm.co.uk/userstory.php. Confirms the use of PFTrack in the films and programmes listed.

[C] Flyer for National Film and Television School Digital Effects MSc Course. Confirms use of PFTrack.

[D] Details of gross income from films is available on sites such as http://www.imdb.com and http://www.boxoffice.com. The examples quoted have featured in User Stories at [B], and supporting material includes the specific examples given.

[E] Letter from Director of Research, The Pixel Farm Ltd. Confirms the role of the university research in creation of products, scale of the business, and examples of partners.