Computer Vision for Stereo 3D Film Production
Submitting Institution
University of SurreyUnit of Assessment
Electrical and Electronic Engineering, Metallurgy and MaterialsSummary Impact Type
TechnologicalResearch Subject Area(s)
Information and Computing Sciences: Artificial Intelligence and Image Processing, Computation Theory and Mathematics
Summary of the impact
The introduction of computer vision algorithms that allow creation of
stereo 3D content in film and
broadcast using conventional 2D monocular video cameras has permitted
fundamental advances
in the 3D reconstruction of complex real-world dynamic scenes from video,
enabling the rendering
of stereo views from a single camera view. Technology introduced by Surrey
has been used for 3D
scene analysis by the UK company, The Foundry, in their film
post-production for major Hollywood
3D film productions (Avatar, Harry Potter, Planet of the Apes, Tron, The
Hobbit). Since 2007, The
Foundry has transformed to become the leading provider of 3D stereo
production software tools to
the global visual effects industry, growing from 20 to over 200 employees.
Underpinning research
Stereo 3D production requires the use of a high-precision camera rig to
align a pair of moving and
zooming cameras. This introduces a number of significant limitations:
(1) all stereo parameters (inter-ocular distance, convergence, zoom) are
fixed at the time of
acquisition;
(2) distortion between the stereo views occurs due to zoom and colour
differences;
(3) camera movement is restricted due to the stereo rig. Correction of
distortion and adjustment of
stereo parameters in post-production is highly labour intensive and
restricted to the captured
camera views.
Research conducted at Surrey in collaboration with The Foundry (www.thefoundry.co.uk)
a world-leader
in film post-production tools and the BBC, supported by Royal Society, TSB
and EU funded
projects (2007-11), has introduced a number of significant advances in
robust through-the-lens 3D
scene analysis. The advances greatly increasing the flexibility in stereo
3D production and allow
stereo image pairs to be produced from monocular camera views. The
fundamental research
advance at the University of Surrey underpinning this technology is the
introduction of robust
computer vision algorithms for 3D reconstruction from video of complex
natural scenes [1,2]. The
foundations for this technology are based on early research [5-6] which
pioneered the use of
multiple view images and video to capture 3D shape, and introduced the use
of this technology for
content production.
A significant research challenge faced by Surrey researchers has been the
reconstruction of highly
dynamic elements, such as people, whilst maintaining video-quality stereo
rendering. This has
been achieved by pioneering research introducing a framework for joint
segmentation and
reconstruction from multiple view video [2] which robustly solves two
classical problems in
computer vision. Information across views is exploited to overcome the
inherent ambiguity in single
view video analysis. This framework provides the basis for both
manipulation of stereo 3D
acquisition in post-production and creation of stereo image pairs from a
single moving monocular
cameras as used in conventional production. This research has recently
been extended [3,4] to
introduce the first integrated pipeline for 3D reconstruction of both
background scene and
foreground live action. This pipeline was used by film production company
BUF in EU project
i3Dpost and demonstrated in the productionof The Midas Touch (2011).
This research was led by Prof. Adrian Hilton in Surrey's Centre for
Vision, Speech and Signal
Processing (CVSSP) working with post-doctoral researchers Dr. Jean-Yves
Guillemaut (appointed
Lecturer in 3D Computer Vision, April 2012). Early research at Surrey [4]
was conducted by Dr.
Jonathan Starck as a PhD and post-doctoral research before he joined The
Foundry to lead their
research team. The Foundry has exploited Surrey research advances in 3D
video analysis to
extend their software tools (NUKE) from 2D compositing to full 3D video
editing. This has led to
NUKE becoming the industry leading 3D stereo production tool widely used
through the global film
industry.
References to the research
1. Hilton, A., Guillemaut, J.-Y., Kilner, J.J., Grau, O. and Thomas, G.,
3D-TV Production from
Conventional Cameras for Sports Broadcast, IEEE Transactions Broadcasting,
57(2): 462-476,
2011 (Grau, Thomas are co-authors from BBC Research)
2. Guillemaut, J.-Y. and Hilton, A., Joint Multi-layer Segmentation and
Reconstruction for
Free-Viewpoint Video Applications, International Journal of Computer
Vision, 93(1):73-100,
2011
3. Kim, H., Guillemaut, J.-Y., Takai, T. and Hilton, A. Dynamic 3D Scene
Capture and
Reconstruction for Outdoor Production in IEEE Transactions on Circuits and
Systems for
Video Technology, 22(11): 1611----1622, 2012
5. Starck, J., Nobuhara, S., Maki, A., Hilton, A. and Matsuyama, T. The
Multiple Camera 3D
Production Studio, IEEE Transactions on Circuits and Systems for Video
Technology
19(6):856-869, 2009 (with international collaborators Nobuhara, Maki,
Matsuyama from
Univ. of Kyoto, Japan)
6. J. Starck and A. Hilton. Surface Capture for Performance-Based
Animation. IEEE
Computer Graphics and Applications, 27(3):21—31, 2007.
Details of the impact
"Surrey, collaborating closely with The Foundry across two funded
research projects, have
made advances in 3D computer vision which have greatly assisted us in
the development of
our industry-leading tools for stereo-3D film production."
Founder and Chief Scientist, The Foundry.
Introduction of a computer vision framework allowing robust
reconstruction and analysis of real-world
scenes has had a direct impact on stereo 3D production in film and
broadcast. Significant
contributions include:
- Pioneering computer vision technology allowing stereo 3D production
from a single moving
camera. This allows stereo production from the principal film camera view
without the need for a
stereo camera rig.
- Full control of stereo parameters in post-production for film and
broadcast allowing correct
rendering for different screen formats and avoiding the need for manual
correction of visual
distortion. This greatly increases the flexibility of stereo production
for both film and broadcast.
- Stereo production of stadium sports using conventional broadcast
cameras avoiding the need
for expensive stereo camera rigs and making possible simultaneous stereo
3D production
alongside conventional broadcast. Prototypes of this technology have been
used in conjunction
with the BBC for a variety of stadium sports (football, rugby, ski-jump).
- Integration with The Foundry of robust visual analysis in industry
leading film post-production
software to increase flexibility in stereo production (Ocula) and video
editing for visual-effects
(NUKE).
- Stereo 3D film production and post-production through integration of
computer vision
technologies in The Foundry's NUKE product which has been used on major
stereo film
productions (Avatar, Harry Potter, Pirates of the Caribbean, Batman)
Advances in robust visual reconstruction have been directly exploited by
UK industry collaborations
in broadcast (BBC), film post-production tools (The Foundry) and film
post-production (Framestore,
DoubleNegative). Framestore and DoubleNegative are the two largest UK film
post-production
companies with 500-1000 employees working on visual-effects for major film
releases (Avatar,
Harry Potter, Batman) and have both received Academy Awards (Oscars) for
their work.
Exploitation of the robust visual reconstruction and stereo 3D rendering
in film post-production has
been realized though The Foundry's NUKE software. This research has
extended NUKE
functionality from a 2D compositing package to a 3D post-production tool
which is established as
the market leader in stereo post-production. This is evidenced by use of
NUKE as the major tool in
the recent wave of blockbuster stereo 3D films (Avatar for example) and
recognition of The
Foundry as one of the fastest growing UK technology companies.
The Foundry has expanded from c. 20 people in 2007 to >200 in 2013.
Dr. Jonathan Starck who
transferred across to The Foundry in 2008 from the University of Surrey to
work on NUKE is now
their Lead Researcher heading up the company's research team. The Foundry
has been listed in
the Sunday Times Microsoft Tech Track top 100 each year from 2010-13 as
one of the UK's
fastest growing technology companies.
The technology introduced in collaborative projects is licensed to The
Foundry and BBC under the
terms of the i3Dlive and i3Dpost collaborative agreements. The IP is
primarily realized as
mathematical methods, software and know-how and therefore not directly
applicable for patent
protection.
Following the success of NUKE for stereo 3D production, The Foundry has
become one of the
leading software providers to the global visual effects industry; their
revenue increased from £6.1M
in 2009 to £14.9M in 2010 with a tripling of personnel from 35 to more
than 100. The Foundry was
acquired in March 2012 by the Carlyle Group for an undisclosed sum and
ranked 10th in the 2012
Sunday Times Deloitte Buyout 100.
Sources to corroborate the impact
C1. Founder and Chief Scientist, The Foundry. Contact details
provided.
C2. BBC Lead Researcher. Contact details provided.
C3. BBC Blue Peter, broadcast February 2011, use of 3D
reconstruction technology for ski-jump
Guinness book of records attempt
http://www.bbc.co.uk/blogs/researchanddevelopment/2011/03/rd-and-blue-peter--ski-rossend.shtml
C4. Ocular product launched by The Foundry in 2008 now a leading
film industry tool for
stereo post-production http://www.thefoundry.co.uk/products/ocula/
C5. The Foundry were listed in the Sunday Times Tech Track 100
from 2010-13 as one of
the UK's fastest growing technology companies:
http://www.thefoundry.co.uk/articles/2012/09/18/411/the-foundry-places-in-tech-track-100/
http://www.thefoundry.co.uk/articles/2012/03/23/330/the-foundry-ranks-10th-in-the-sunday-times-buyout-track-100/