Hanlin Goh published a note.
Light Field Photography (Part 1: Technical Introduction)
Photography has always been exciting for me because it is a challenge of both hemispheres of the brain. The left brain understands the science of recording light; the right brain brings creativity. In this series of notes, I will explore some technical details and applications of digital light field photography using the Lytro.
Thanks to a friend in US, I received my Lytro light field camera on 12 March 2012. And as some of you probably noticed, I have been posting a photo a day in my Living Picture Year project. So far, it has been more of an artistic journey. Recently, my left brain called out for attention to understanding the technical aspects the camera. I'll try to make the notes as comprehensible as possible, but some of you will find it too technical... so just enjoy to pictures =)
What is Light Field Photography?
Light field photography involves two steps: 1) recording of light rays using a plenoptic camera and 2) synthesis of the recorded image. A plenoptic camera is almost like a conventional camera, with lenses and a sensor to record the light coming through the lenses. However, in addition, in front of the sensor it has a layer of tiny microscopic lenses or "microlenses". The microlenses diffuses each ray of light at a microscopic scale, so that the photo sensor can record all the directional details of the light rays. It is like having many tiny pinhole cameras in front of the sensor. Later, objects have not been focused by the main lens can be virtually refocused through image processing.
The Lytro "science" page has a simple explanation of the camera. Those interested in the full details can refer to the thesis of Ren Ng, Lytro's founder.
Understanding the Raw Light Field Image
The camera sensor is a 3280 x 3280 BGGR bayer array, giving you about 11 Megapixels (or mega-rays as they call it). However, because each "pixel" records a light ray from a different direction, each image is only 1080 x 1080 (or 1.2 Megapixels). The light field image file is in a new format, currently one can only use Lytro's proprietary software (currently only for Mac) that comes with the camera to transfer, open and share the captured images. This makes editing/photoshopping impossible (at least for now).
On the computer, the pictures are stored in a library. In the library, each raw image (.lfp format) is a 16MB file, while a compressed (and processed) version is about 1MB. If you are a computer science/engineering student or a hacker, it is not difficult to extract the information inside each file (disclaimer: do not attempt past this point if you don't know what you are doing). However, Nirav Patel has already written a lfpsplitter, which is perfect for the lazy me. With some quick programming in Matlab, I was able to parse the raw file.
Let's use my photo of Hulk and Stitch on 25 May 2012 as an example [Click Here to View the Living Picture]. The image with the preset focus is:
I extracted the raw sensor information and here is a 100% crop of half of Stitch's face:
We can see that it is a grid of circles - within each circle the sensor records the rays from one micro lens (like a mini photo). It is almost like placing thousands (or even millions) of cameras side-by-side, each capturing a tiny part of the scene. We imagine that one circle is used to synthesize one small part in the final image (although there's actually more to this).
Notice that some circles have a uniform colour (centre of Stitch's eye and forehead), while other circles have two colours (between the eye and skin), and there are even some with even more complex details (teeth). These variations are due to each pixel recording a ray in a slightly different direction from its neighbour. The more different the object looks at various focusing planes, the more complex the information in the circle. For example, a uniformly coloured object looks the same when focused or blurred, so it produces a with only one circle.
This detailed recording of light rays is the key to the technology for light field photography. The extra information is exploited to produce various image renderings for different purposes, like refocusing. In the next few notes, I will try to program some light field related applications just for fun... stay tuned. Meanwhile, enjoy the photos from my Living Picture Year project.
