Hanlin Goh published a note.
Light Field Photography (Part 2: Perfect Focus)
In my previous note, I described a technical overview of what light field photography is and what the Lytro camera records in the raw form. The extra information by recording the entire light field, specifically the displacement of pixels between the micro lenses helps to estimate the depth of the scene, which eventually leads to many interesting applications. In this note, I will explore the idea of focusing a light field image. I will use my photo of Wooden Vehicles, for most of the examples here [Click Here to View the Living Picture].
Depth of the Scene
Lytro saves the light field information in two files. A raw version, as shown in the previous note and a processed version, which I will now explore. (Again, I need to add a "venture beyond here at your own risk" disclaimer).
A major part of the processed file is a set of jpeg images that have varying focus. They have been individually refocused at different depths from the raw file (I will not go through the whole mathematical theory of how these images are created). Each of them has an assigned focused depth. Three of these images (focused near, middle and far) are shown here:
A smaller, but no less important, part of the processed file reveals the depth of the scene arranged in a 20 by 20 grid. I made a program that estimated the depth information more precisely. Here's a visualization of my estimated depth map overlaid on the original image, where the portion closer to the camera in blue and the further portions in red. Notice the colours hug around the objects quite nicely.
Image Refocusing
Currently, Lytro provides an interface where can we click on the image to refocus it. In this part, I aim to recreate this capability.
The combination of the depth information and the set of images make it easy to refocus the image at anywhere we want. At the spot we want to focus on, we simply need to retrieve its depth from the depth map and display the image has a depth that is closest. Here's an example of selecting the focus point in the foreground, middle and background. The top row is a visualization of the focus of different parts of the image with respect to the selected point, where blue indicates the areas that will be in focus and red are portions that are most blurred. The bottom row is the resulting image that is shown.
Demo Time!
Click the following links to play with the image refocusing demos I programmed:
Focus Stacking
The lenses of conventional cameras captures only a limited depth of field, so not every part of the scene can be in focus. The depth of field can be increased by "physical tricks" such as using a smaller aperture or a wider-angled lens. For a light field image, it is also governed by the same laws of optics and may not be able to capture in a single image the full range of depth. However, because it the light field is recorded, it is possible to enhance the depth of field through "virtual manipulations" after image capture. Lytro already has plans to include this feature in their update later this year.
Since we can refocus the image to wherever we want, then by extension, it is also possible to synthesize an image that is completely focused. I programmed a method in Matlab that assigns a value to each pixel based on the probability of each map being in focus. Saving you from all the technical details, here is the before and after picture after running the program.
Selective Focusing (Miniaturization Effect)
Selective focus within a scene helps a photographer draw attention and tell the story, so we do not always want the entire photo to be in focus. Selective focusing is also used to enhance macro photographs by blurring the background or inducing a toy-like miniaturization effect (currently physically achieved by tilt-shift lenses).
The opposite of focus stacking is to achieve a depth of field that is shallow (possibly shallower than physically recorded). Using depth information I wrote a program to sharpen pixels at a selected depth, while applying a Gaussian blur to the other layers with increasing standard deviation as the distance to the selected depth increases. Currently, because it is less accurate to estimate the depth of a scene that is already mostly in focus, results may vary - to that end, Lytro probably has to put in some additional work. After ironing out the creases and some colour enhancements in photoshop, this is the result of miniaturizing a picture posted on 19 May 2012:
Part 3 will be coming out this weekend. Stay tuned ~ ciao!
