Advertisement
Behind every great image taken by a camera phone there's a huge amount of electronic / optical / mechanical magic taking place. Camera users are normally oblivious to this magic since it happens so quietly and unobtrusively. Here we discuss the challenges of creating a good image from a CMOS sensor inside a camera phone.
Advertisement
How an Image is Created
Advertisement
In a film camera, light captured through an optical system strikes a piece of film, which is exposed and then developed using a chemical process. In a digital camera, that light still travels through an optical system comprising a multi-element lens, and a barrel, but now the light strikes a digital sensor array of rows and columns, made of millions of tiny picture elements or pixels. Figure 1 is a mechanical overview of a digital camera.
Fig. 1: Mechanical overview of a digital camera. The optics are virtually identical to those in a conventional film camera.
When light strikes the pixel array it goes through a color filter array that ensures that only blue, red or green light actually hits the appropriate pixel. At each pixel an analog signal is created, which goes through an ADC (analog to digital converter) to become a digital signal. This signal is then sent through what we will call the Image Pipe (or I-Pipe) that comprises a series of electronic filters that make the signal look like a real picture.
The I-Pipe adjusts the white balance, the color, and “reverses” certain anomalies introduced into the picture by the nature of the capture method. Examples include lens shadows, geometric distortion, reduced picture focus away from the center of the lens, and digital sensor noise. The Agilent I-Pipe also compresses the image using JPEG to create a small, accurate, compressed image that can quickly be written into a storage medium.
View this block diagram full size
Pre-Processing the Light
An absorptive or reflective infrared filter is used to pass the visible part of the spectrum and block infrared radiation above 780 nm. This ensures that the sensor focuses only on what the eye will see, and optimizes the integrity of the colors. If infrared light is not cut off in this manner it can cause blurriness and decrease the sharpness in the image formed by the lens.
A microlens is also used to pre-process the falling light so it is refracted appropriately into the pixel in as vertical a direction as possible. This microlens enhances optical sensitivity of the pixel and usually sits right above the color filter array.
Color Filter Array – Bayer Filter
Photodiodes are sensitive to brightness and not to colors. Therefore, some mechanism must be used to artificially make them sensitive to specific colors so those colors can eventually be represented to the human eye. A color filter array is used to ensure that each sensor pixel receives light of just one color: typically red, blue and green.
Fig. 2: The human eye is twice as sensitive to green as it is to red and blue. The Bayer color filter alternates a row of red and green filters with a row of blue and green filters with twice as many green pixels as blue and red combined.
There are different patterns that can be used for the color filter array. Because of the way the human eye perceives color and the fact that the human eye is twice as sensitive to green as it is to red and blue means that to emulate the human eye perception, the camera needs more green pixels. The Bayer pattern (Fig. 2) alternates a row of red and green filters with a row of blue and green filters with twice as many green pixels as blue and red combined. The raw output from the Bayer filter is a mosaic of blue, green and red pixels varying in intensity depending on the degree of light shone on a particular pixel.
continued on next page
Demosaicing and White Balance
When an image is created by the color filter array, four separate pixels determine the color of a single pixel. This forms a mosaic of separate colors that don’t look like a real image until a demosaicing algorithm takes the average of color values from the pixels closest to the target pixel to approximate what the true color of the target pixel should be.
With no correction, a picture taken in fluorescent light may appear too green and one taken outdoors at sunset may seem too orange. Automatic white balance (AWB) correction ensures that the white in an image appears as true white to the viewer.
Image Restoration: Removal of Unwanted Artifacts
In CMOS or CCD sensors there are several sources of noise added to the image that must be removed or at least mitigated. These are:
1) Fixed pattern noise produces the same noise pattern in every picture. This fixed pattern noise can be mitigated by the camera taking a “dark” reading (an exposure with no light) and then subtracting that result from a normal exposure. The dark level output current is the average output level produced when there is no illumination and will contain the leakage current of the photodiode.
2) Random noise, which can be caused by the ambient temperature. A higher temperature will generally cause more electrons to leave their orbits and create random noise signals in the sensor. It is exacerbated by heat dissipation from the sensor circuitry. If the camera phone is left in a car during summer, the pictures it produces will be much nosier than if the camera was inside a cool building.
3) Pixel cross talk, where the light entering one pixel crosses over into an adjacent pixel creating a “muddying” due to say a red pixel letting red light into an adjacent blue pixel causing an unwanted increase of signal in the blue pixel and lost image information in the red one.
Fig. 3: A microlens is used to refract light into the pixel in as vertical a direction as possible. Higher-end sensors use a second microlens to bend the light again, further down the pixel, to minimize the chances of it crossing over into adjacent pixels and causing crosstalk noise.
Higher-end sensors use a second microlens to bend the light again, further down the pixel, to minimize the chances of it crossing over into adjacent pixels and causing crosstalk noise (Fig. 3).
The number of pixels is only one measure of information capture capability. A bigger pixel, will, in general, have a higher signal-to-noise ratio than a smaller one because it will have more area to gather light and therefore captures more photons and creates a bigger signal relative to the overall noise that exists.
On the Fly Defect Correction
There are always some pixels on a sensor that do not survive the manufacturing process without optical blemishes or electrical defects. This can produce visual imperfections that result in the production of inconsistent or non-linear responses to incident light.
The I-Pipe determines whether a pixel is defective or not by measuring its output and then comparing it to the average values of neighboring pixels. If the difference is greater than a certain tolerance threshold, that pixel is “marked” as bad and its output is no longer valid. The value for the output at that target pixel location can be interpolated by using the output values of the adjacent pixels and averaging them to produce a simulated output at the bad target pixel.
Enhancing Degraded Image Resolution or Sharpness
So far, we have seen how the light travels through an optical lens, a Bayer matrix, and one or more microlenses. It is then re-created as an image familiar to us by the use of demosaicing, interpolation, anti-vignetting, defect correction, etc. Performing so many “unnatural” electronic acts causes the sharpness of the final image to be less than it should be. This sharpness can be achieved by adding a part of a high-pass signal (high frequencies only) to the output, corrected image. Noise sharpening can be reduced or increased depending on how much high-pass signal is added.
continued on next page
Vignetting Correction
Vignetting is the shading or darkness created at the edges of the picture due to the lens and its barrel. Vignetting correction corrects this aberration by normalizing the illumination ratio of the corner to that of the image center.
Post Processing Enhancements that Improve the Image
The image can be enhanced by enhancing contrast in bright and dark areas to improve the perceived fullness and color quality. Agilent uses this technique, called adaptive tone mapping, within its CMOS image sensors to create truer, richer colors by extending the dynamic range. The bottom image uses adaptive tone mapping and enhances brightness and contrast by automatic adjustment of tone maps. This enhances contrast in bright and dark areas, shifts under exposed and over exposed images and results in brighter, more vivid color reproduction.
The Complete Imaging System
In a complete imaging system (Fig. 4) you can see where the I-Pipe image processing phases occur as the signal is cleaned, shaped, and enhanced before it is output to a display or stored in memory.
Fig. 4: Block diagram of a complete imaging system, showing the various forms of processing applied to the raw output from the image sensor.
About the author:
Feisal Mosleh holds a B.Sc.(Hons) from Imperial College, University of London and an M.Sc. in Electronics Engineering from the University of Durham, England. He is currently the worldwide strategic marketing manager for Agilent’s Sensor Solutions Division based in California.
0 comments on “Camera phone image processing “magic,” revealed!”