ICARUS – Interactive Construction of 3D Models from Digital Images
Submitting InstitutionUniversity of Manchester
Unit of AssessmentComputer Science and Informatics
Summary Impact TypeTechnological
Research Subject Area(s)
Information and Computing Sciences: Artificial Intelligence and Image Processing
Summary of the impact
In the late 1990s, a significant barrier to the adoption of virtual
reality software was the expense of manually creating models of real-world
scenes. To address this, between 1998 and 2004, the ICARUS software system
was developed, which enabled the creation of structured, 3D
geometric models from a sequence of images or video. The system also
pioneered improved methods of camera tracking. ICARUS was subsequently
licensed and developed commercially, and became the foundation for video
and film post-production products that are used worldwide in the film
(e.g. Universal Pictures, Warner Bros, Paramount Pictures) and television
(e.g. BBC) industries, underpinning a company with an annual turnover in
excess of £1m.
ICARUS was created to address the problem of building geometric models of
real-world scenes directly from digital images. Camera calibration and
tracking are central, not only to constructing 3D models, but also to
video and film post-processing. Accurate camera calibration enables
automation of special effects, such as the addition of virtual objects,
removal of unwanted objects and artefacts, and compositing of separate
sequences. It involves computing the camera's extrinsic parameters
(position and orientation) for each image in a sequence and its intrinsic
parameters (focal length, aspect ratio, principal point and skew). A key
contribution of the research was a new hierarchical, two-pass tracking
algorithm [1, 2, 5] based on three innovations (see  for details):
- The use of estimates of inter-frame camera motion to guide the feature
tracker, greatly reducing the number of incorrectly tracked features.
- A reliable approach to projective reconstruction, using carefully
selected subsequences of the video data, and hierarchical algorithms to
merge subsequences into a single reconstruction. Previous methods used
feature-matching algorithms that were sensitive to drift over a video
sequence, leading to poor reconstructions, particularly for
wide-baseline, or lengthy, motion sequences.
- The application of random sampling algorithms to the problem of
self-calibration, allowing for more accurate upgrades from projective to
metric geometry. The research showed that applying random sampling
techniques to the problem of self-calibration significantly increased
the usefulness and applicability of the basic linear algorithm,
especially for longer video sequences.
The result of these innovations was an accurate and reliable
camera-tracking algorithm that could be used in a wide range of
applications (e.g., [3,4,6]).
S. Gibson (RA, 1995 - 2004); now Research Director, The Pixel Farm Ltd).
J. Cook (RA, 1995 - 2004); now Senior Software Engineer, The Pixel Farm
T.L.J. Howard (Lecturer, SL, Reader, 1988 - date).
R.J. Hubbold (SL, Professor, 1985 - 2010).
D. Oram (PhD student, 1997 - 2001).
A.J. West (Lecturer, 1988 - 2001).
A.D. Murta (Lecturer, 1990 - 2001).
References to the research
The research was published in internationally recognised conferences and
journals, with several of the publications being well-cited.
 S. Gibson, R. J. Hubbold, J. Cook, and T. L. J. Howard. Interactive
reconstruction of virtual environments from video sequences. Computers
and Graphics, 27(2), April 2003, pp. 293-301. (Best paper award.)
(Google Scholar: 30 citations)
 S. Gibson, J. Cook, T. L. J. Howard, R. J. Hubbold, and D. Oram.
Accurate camera calibration for off-line, video-based augmented reality.
In IEEE and ACM International Symposium on Mixed and Augmented Reality
(ISMAR 2002), September 2002. Darmstadt, Germany. IEEE Press, ISBN:
0-7695-1781-1. DOI: 10.1109%2FISMAR.2002.1115068.
(Google Scholar: 107 citations)
 S. Gibson, T.L.J. Howard and R.J. Hubbold. Flexible image-based
photometric reconstruction using virtual light sources. Proc. Eurographics
2001, Computer Graphics Forum, 20(3), September 2001, pp. 203-214,
Blackwell Scientific. DOI:10.1111/1467-8659.00513
(Google Scholar: 24 citations)
 A.D. Murta, S. Gibson, T.L.J. Howard, R.J. Hubbold, and A.J. West.
Modelling and rendering for scene of crime reconstruction: A case study.
In Proc. Eurographics UK 16th Annual Conference, pages
169-173. European Association for Computer Graphics, March 1998. ISBN
0-952 1097-7-8. (Google Scholar 16 citations)
 S. Gibson, J. Cook, T.L.J. Howard, R.J. Hubbold. ICARUS: Interactive
Reconstruction from Uncalibrated Image Sequences, Proc. ACM SIGGRAPH
2002, Sketches and Applications, ACM Press, July 2002. ISBN
1-58113-525-4. DOI: 10.1145/965400.965427
(Google Scholar: 8 citations)
 S. Gibson, J. Cook, T.L.J. Howard, and R.J. Hubbold. Rapid shadow
generation in real-world lighting environments. In Rendering
Techniques 2003 (Proceedings of the Eurographics Symposium on Rendering
2003), Leuven, Belgium, June 2003. (Google Scholar: 73 citations)
Details of the impact
Camera tracking, as provided by PFTrack, is central to modern film
post-production, enabling the merging of real footage with
computer-generated effects, and the removal of unwanted objects visible in
the footage, such as unwanted street furniture, telegraph poles, etc.
Accurate camera calibration enables a seamless inclusion of such changes.
By extracting 3D structure from footage, background detail can be
re-inserted where objects have been deleted. Tracking also underpins other
areas of application, such as removal of scratches from archive film.
Design and development of ICARUS commenced in 1998. At that time,
pioneering work on automatic tracking and reconstruction was underway at
the University of Oxford in the UK (Zisserman, Hartley et al.),
and the University of Leuven in Belgium (van Gool, Pollefeys). Whilst this
work had laid the foundations, software capable of constructing real
environments with structure (e.g. named parts), or reliably tracking
commercial video and film sequences, did not really exist in a usable
Pathways to Impact
By 2002, considerable interest was being shown in ICARUS by the research
community and by companies, and a decision was reached to release a free,
binary-only version of the software for non-commercial use. In the first
three months, this received over 10,000 downloads. In 2003, ICARUS was
licensed to The Pixel Farm, a UK start-up company, for exclusive use in
the film and video post-production industries, and the free version of the
software was withdrawn. Initially, the University provided consultancy and
software support to the company. In September 2004, Dr Simon Gibson (RA),
the originator of many novel aspects of ICARUS, joined The Pixel Farm as
their Research Director, and Jon Cook (RA) joined as a software engineer;
both still work for the company.
Notable early successes were the use of ICARUS — by then rebadged by the
company as PFTrack — in the Academy Award winning film Cold Mountain
(Miramax Films, 2003) [A], and the film Fantastic Four (20th
Century Fox, 2005).
Reach and Significance of Impact
Today, PFTrack is used extensively in industry in the USA and Europe.
Recent film credits (within 2008-2013) include: Watchmen (Warner Bros,
Paramount Pictures), the Harry Potter films (Warner Bros), Battle Los
Angeles (Columbia Pictures), The Boat that Rocked (Universal Pictures),
The Hunt for Gollum (Independent Online Cinema), and TV series such as
Smallville (BB Studios, Canada for USA TV networks), Doctor Who (BBC),
Grandpa in my Pocket (Dinamo for BBC), The Wrong Door (BBC), Kia Motors
Pro_Cee'd TV advert (The Mill, UK). Details of these and other examples
where PFTrack has been used can be found at [B]. PFTrack is also used at
the UK's National Film and Television School in its Digital Effects MSc
course [C]. Examples of high grossing films during the REF period in which
PFTrack has been used include Battle: Los Angeles (over $200m) and
Watchmen (over $185m) [D].
The Pixel Farm has grown to become a leading provider worldwide of this
type of software within a highly competitive industry, employing 10
people, and with a turnover exceeding £1m annually [E]. Major partner
companies include Cooke Optics and FilmLight [E].
It should be noted that this list is partial: many companies in the
film industry are secretive about the post-production facilities and
software that they employ and will not allow any publicity, even under a
Sources to corroborate the impact
Supporting material is available from the university for the
corroborating sources below.
(dated November 2003, accessed 21/02/2013). Confirms the use of PFTrack on
Cold Mountain, and provides details on how it was used.
[B] For details of projects using the PFTrack software, see the company's
Confirms the use of PFTrack in the films and programmes listed.
[C] Flyer for National Film and Television School Digital Effects MSc
Course. Confirms use of PFTrack.
[D] Details of gross income from films is available on sites such as http://www.imdb.com and http://www.boxoffice.com.
The examples quoted have featured in User Stories at [B], and supporting
material includes the specific examples given.
[E] Letter from Director of Research, The Pixel Farm Ltd. Confirms the
role of the university research in creation of products, scale of the
business, and examples of partners.