Summary of the impact

Professor Kautz and his team have developed two photo manipulation and processing methods (Exposure Fusion and local Laplacian filtering) that are used to produce well-exposed photographs with tuneable local contrast. Both are robust and consistent without requiring any per-image parameter tuning. Due to its reliability and effectiveness, Exposure Fusion is now considered the standard method for blending multiple photographs into a single well-exposed photograph, and is used by a large number of commercial and non-commercial products. Local Laplacian filtering was chosen by Adobe Systems Incorporated to be the default tool for image enhancements in Adobe Lightroom and Adobe Camera Raw. As a result, these methods are now in the hands of hundreds of thousands users, who use them to create and manipulate well-exposed digital photographs.

Underpinning research

Image processing methods and how they can be applied to digital photographs have been researched for many years. However, most of the proposed methods require per-photograph parameter tuning, which makes their use cumbersome in practice. Furthermore, many of the more advanced image processing techniques are computationally expensive.

Since 2007, Professor Kautz and colleagues have been looking at methods for advanced image enhancements that do not require such parameter tuning and which are efficient nonetheless. The problem that his team first addressed was how to avoid over- and under-exposed areas in digital photography. The dynamic range of natural scenes commonly exceeds the dynamic range that can be captured by a digital camera. In order to reduce these over- and under-exposed areas, Professor Kautz proposed Exposure Fusion, a method for combining multiple images into a single well-exposed image. This work was done in collaboration with a visitor (Tom Mertens) from Hasselt University, with the main idea invented by Professor Kautz [1][2]. The key idea is as follows: The user takes multiple differently exposed photographs of a scene. This ensures that each part of the scene is correctly exposed in at least one of the photographs. Exposure Fusion takes this stack of images and fuses them into a single image, where each part of the scene is correctly exposed. Technically, this is done using multi-scale blending to fuse the images. The method automatically deduces which areas should come from which exposure and fuses them accordingly. The main benefit of the method, besides computational efficiency, is that it achieves consistent results for a large variety of photographs, and essentially never fails. This is in contrast to almost all other techniques that can be employed in this context.

This work used so-called Laplacian pyramids as the underlying data structure, which decomposes an image into multiple subsampled images that recursively represent the difference to the finer pyramid levels. This enables operation on multiple scales and also supports multi-scale blending. Given the success of using this pyramidal representation, Professor Kautz and his team (again with international collaborators) expanded upon it in 2011 and demonstrated that pyramids are also useful for efficient but high-quality compositing. This type of compositing traditionally requires solving the Poisson equation, which is computationally expensive. Here, it was demonstrated that pyramidal processing can achieve results indistinguishable from Poisson blending, but at a small fraction of the computational cost. Again, no parameter tuning was necessary [3].

In 2012, in collaboration with Adobe (Sylvain Paris) and the Toyota Technical Institute (Sam Hassinoff), Professor Kautz demonstrated that the Laplacian pyramids can be modified to work more locally, which in turn enables very robust, high-quality edge-aware filtering [4]. Professor Kautz led this research effort. Edge-aware filtering is the building block for many image-processing methods, and the team demonstrated its use for tone-mapping, detail enhancement, and image abstraction. The main insight of the work is that processing each coefficient in the pyramid using its local point-wise filtering operation yields artefact-free edge-aware filtering. This was rigorously evaluated on dozens of images and showed that consistent results across images are achieved. Code was publicly released for local Laplacian filtering, in order to ensure an uptake in the community. The work was funded through several unrestricted gifts by Adobe Systems Incorporated.

All the research was conducted jointly with an international team of collaborators. In both cases, UCL provided the core ideas. For Exposure Fusion, UCL's particular contribution was the idea to fuse multiple images directly to avoid under- and over-exposed areas. For local Laplacian filtering, Professor Kautz's work on Laplacian filtering served as the starting point for the joint research with Adobe and TTI. UCL's particular contribution was the idea to use Laplacian pyramids, which is the crux of the methods.

References to the research

[1] and [4] best demonstrate the quality of the research.

Details of the impact

UCL's work has introduced two new processes into digital photography, enhancing the ability of photographers to process their photos in a much more robust and consistent way.

The methods were first disseminated through conference and journal publications. Both Exposure Fusion and local Laplacian filtering were made available as open source, in 2007 and 2011 respectively, to be used freely by anyone. Releasing code in this way was the crucial component in influencing users and beneficiaries, as it was easy for software companies and open source developers to test and validate the methods.

As both approaches achieve high-quality results, they were taken up very quickly by a variety of tools. Exposure Fusion was incorporated into Photomatix (commercial software), Bracketeer (commercial software), Enfuse (open source, the first tool with Exposure Fusion, in 2009), Tufuse (freeware), and mobile phone cameras [a]. A professional photographer outlines the benefits the method brings to photographers: "Exposure fusion results in noise reduction (contrary to local tone-mapping which amplifies noise) — this makes it perfect for night and long-exposure `HDR' photos; images have more natural look [and] are free of halo artifacts; using exposure fusion might be easier because it has fewer parameters to set — also it is more intuitive as many photographers are familiar with notion of blending images." [b]

The key beneficiaries to date are photographers who want to avoid over- and under-exposed areas in their photographs. This encompasses hobbyist photographers, real estate agents taking pictures of interiors [c] and special effects companies for creating backdrops, For example, in creating the backdrop for the great hall scene in Harry Potter and the Deathly Hallows: Part II, released in 2011, Exposure Fusion reduced the amount of work and time that was needed to achieve desired backdrops because it reduced the amount of noise on the film plate [d]. Noise would normally need to be removed by filtering or time-consuming and painstaking edits.

Real estate agents can also now create pleasing photographs without much manual intervention and without spending much of their time on them. Previously, there was no easy way for them to create interior photographs that could depict the interior as well as the exterior in a consistent way (often the outside would be overexposed or parameters would need to be tweaked to achieve a good photograph). One professional architectural photographer comments: "Most of my exposure blending is done through a process called exposure fusion. My experiments with HDR were less than ideal, looking over-processed which is why I moved to exposure fusion. Architecture and real estate photography requires a realistic, natural look so as not to misrepresent the subject. I use Photomatix' exposure fusion mode which, when set properly, helps create realistic looking images. I do charge more for HDR real estate shoots since it really improves the image quality and helps create more attractive real estate photos." [c]

The user base for this particular method is vast; the aforementioned commercial and free tools are very commonly used (e.g., the Photomatix group on Flickr alone has more than 400,000 photographs [e] [text removed for publication]). It is difficult to get an exact number of installed software, as software vendors are reluctant to disclose this information, but an estimate is that millions of users have used Exposure Fusion.

The local Laplacian filtering work was demonstrated to Adobe product teams in 2011 and they were immediately convinced that UCL's proposed method was superior to any other method they had tested. Starting from the publicly released prototype implementation, the Adobe Lightroom and Camera Raw team integrated the UCL method into their products (released in April 2012). Professor Kautz's Laplacian filtering technique now forms the main image-editing tool (for contrast, shadows, highlights, etc.) in Adobe Lightroom and Camera Raw. The new processing tools based on UCL's work have been reviewed in the popular press numerous times and the quality of the processing is always highlighted. For example, Digital photography review site, dpreview: "[A]fter processing dozens of images in PV2012, I find I am consistently getting pleasing results in fewer discrete steps."; The Guardian: "Pros: new processing algorithm, powerful and mostly intuitive to use"; Technology news site, Ars Technica: "It looks like Adobe's made some real progress with this, in both quality and ease of use." [g]

Adobe describes the effect of local Laplacian filtering: "The Lightroom and Camera Raw team has been very pleased with all of the positive feedback on the new image processing (PV2012) available in the Lightroom 4 beta... The ability to recover shadow and highlight detail with a straightforward set of controls without introducing artifacts or over-the-top faux-HDR effects is a huge leap forward in image processing. I thought Scott Kelby summed it up quite well when he said, `Your photos look better processed in Lightroom 4. Period.'... The team would like to share the praise that we're receiving for the new processing controls with the authors of this research paper [output [4], above]" [h].

Since the method changes the basic manipulation tools (contrast, etc.), it has affected virtually every user of Adobe Lightroom and Camera Raw, of which there are millions. While it is difficult to get reliable numbers, the beta-version of Lightroom 4 was downloaded 300,000 times [i]. The full version has more users, and in combination with Camera Raw, Adobe estimates there are at least one million users [j]. The method is also in use in the latest version, Lightroom 5, released in June 2013 [k].

Sources to corroborate the impact

[a] Exposure Fusion's use in Tufuse http://www.tawbaware.com/tufuse.htm, Enfuse: http://enblend.sourceforge.net/enfuse_details.htm, Bracketeer: http://www.pangeasoft.net/pano/bracketeer/index.html

[b] For a professional photographer's review of Exposure Fusion, see: http://www.hdrone.com/2013/01/exposure-fusion-in-photomatix-for-ultra-natural-photos/

[c] Quote from architecture photographer on benefits of using Exposure Fusion, including that the higher-quality work it produces helps them receive higher pay: http://blog.daminion.net/interview/with-architectural-photographer-scott-dubose/

[d] Correspondence from a VFX photographer confirms the benefits of using Exposure Fusion on the Harry Potter and the Deathly Hallows film. Available on request.

[e] The Flickr Photomatix group, with 400,000 photos: http://www.flickr.com/groups/photomatix/

[f] [text removed for publication]

[g] Reviews of Lightroom that mention its new processing feature: http://www.guardian.co.uk/technology/2012/apr/25/adobe-lightroom-4-review; http://www.dpreview.com/articles/7481161037/lightroom-4-review/6 and http://arstechnica.com/gadgets/2012/04/ars-reviews-adobe-lightroom-4/

[h] Adobe's corroboration of its use of local Laplacian filtering, and its positive feedback from reviewers: http://blogs.adobe.com/lightroomjournal/2012/02/magic-or-local-laplacian- filters.html

[i] Confirmation of the 300,000 downloads of the Lightroom 4 public beta, John Nack on Adobe, http://blogs.adobe.com/jnack/2012/03/lightroom-4-arrives-at-a-great-new-price-too.html

[j] Adobe confirms it has at least 1 million users of its Creative Cloud suite of programs, which includes Lightroom: http://blogs.adobe.com/creativecloud/one-million-members-one-million- thank-yous

[k] A research scientist at Adobe can corroborate the involvement of the UCL team, and the use of local Laplacian filtering in the Adobe technology platform, including in Lightroom 5. Contact details provided separately.