updated 11:48 pm EDT, Thu May 8, 2014
Multiple pics as camera shakes blended into 'super resolution' image?
A new patent application discovered by AppleInsider may hint at future technology that would be used in iPhones or other camera-based products. Using a concept called "optical image stabilization" (OIS), Apple's patent application proposes a method of capturing multiple samples from a camera taken nearly instantaneously and blending them intelligently into a high-density, "super-resolution" image.
The idea comes from the inescapable fact that people more often hold cameras in their hand to take pictures than us a tripod or other stabilizing tool. This often results in blurry pictures that aren't usable but still contain all the resolution and much of the picture information that is captured in a well-taken image. In something of the same manner as differently-exposed photos can be combined to produce High Dynamic Range (HDR) images, multiple nearly-instantaneous shots could be intelligently composited into a very-high-resolution image where software has intelligently blended the best parts of all the images.
A forerunner to this idea is already in practice on the iPhone 5s, in the "auto image stabilization" feature that is currently handled by software and hardware. When a user takes a picture, the camera actually takes four images, then analysis is done on the pictures by a image signal processor (ISP) that "takes the best parts of those photos [and combines them] into one image with as little noise, subject motion, and hand shake as possible," says Apple.
Optical stabilization is preferable to digital stabilization, as it results in a better original image. For some time now, iMovie has featured a form of digital stabilization for video, cropping the original image to create stability in pans and walking or other motion recordings where the camera was not steady. The technique of optical stabilization has been around for quite a while and already features in many smartphones, but the new Apple implementation is different of the usual methodology.
With traditional optical stabilization, small movements in the camera can be compensated for with actuators that shift the camera components slightly in an attempt to nullify the effect of the movement. Apple's approach plays off a technique used by the company in the iPhone 5s for making panoramas -- in addition to using traditional techniques to stabilize the image, the 5s camera stitches many high-resolution images together (combined with the ISP features such as dynamic auto exposure) to create an image that's not only clear and bright, but also large -- 16MB in size at 10,800 x 2376 in resolution.
The patent application describes a method by which the camera would take multiple images and use the iPhone's built-in systems -- presumably including the M7 motion co-processor, the gyroscope, the accelerometer, the ISP and other factors -- to intelligently blend the images into one super-high-resolution image, suitable for creating larger prints or using in 4K high-resolution videos among other possibilities.
"In one embodiment, an electronic image sensor captures a reference optical sample through an optical path. Thereafter, an optical image stabilization (OIS) processor adjusts the optical path to the electronic image sensor by a known amount," Apple says in its patent. "A second optical sample is then captured, along the adjusted optical path, such that the second optical sample is offset from the first optical sample by no more than a sub-pixel offset. The OIS processor may reiterate this process to capture a plurality of optical samples at a plurality of offsets. The optical samples may be combined to create a super-resolution image."
The fusion of the two techniques -- traditional OIS and advanced image processing techniques -- allow Apple to take all usable picture information and resolution together to create super-resolution images. The challenge behind the patent is to combine optical image stabilization and the ISP hardware without making the camera module unduly bulky. This is exactly why software image stabilization is much more commonly employed.
The patent application lists just two inventors, Richard L. Baer and Damien J. Thivent, and was originally filed on November 8, 2012.