Refocusable camera in a phone? A good idea. If it's not too low resolution.
***
// published on The Verge - All Posts // visit site
Apple patents Lytro-style refocusable camera
Apple appears to have some interest in Lytro-style refocusable cameras. In a patent filed in 2011 and granted today, Apple describes a camera that would be able to switch between capturing two types of images: low-resolution images that could be refocused after capture, and high-resolution photographs that can't be changed. To do both, the camera would include an adaptor with a "microlens array" that would slide between the camera's lens and sensor when capturing a refocusable image, and slide away when capturing a traditional photo. The image above depicts the cross section of such a camera while the adaptor's microlens array — labeled "430" — is positioned to shoot a refocusable image.
Adding API access to the camera would be a very good idea. A natural way for google to break out of androids camera doldrums. Potential applications could be very exciting.
Google is working on a new Android camera API, supports Camera RAW
The Nexus 5 camera was a huge disappointment, especially after comments from high-ranking Googler Vic Gundotra stating that "we are committed to making Nexus phones insanely great cameras. Just you wait and see."
That was nine months ago. We waited and saw, and what showed up on the Nexus 5 wasn't very good. There may be an explanation for this, though. According to commits in the public Android source code, which were first spotted by Josh Brown on Google+, Google is working on a new camera API for Android. Work on the new API started in December 2012, which would make it seem targeted for KitKat, but about a month before the new OS's release, the API was pulled from Android's framework code. The commit that removed the API from the release Android code is here, with the comment saying:
DO NOT MERGE: Hide new camera API. Not yet ready. Bug: 11141002
This commit was pushed on October 11, about a month before the release of KitKat. A month before release was probably "feature freeze" time, where work on new features stops and everyone focuses on fixing bugs in time for release. The camera revamp didn't make it and was replaced with the original camera API.
Moto X Update Dramatically Improves Camera Quality
In my Moto X review, I characterized camera performance and quality as very bimodal. In some scenes it could take great photos, in others it seemed to struggle and either produce images that looked somewhat washed out with weird white balance or aggressive noise reduction. In low light scenes where I expected the clear pixel to make a dramatic improvement, photos were a bit splotchier than I had hoped also from noise reduction. On paper the Moto X should’ve been a solid performer – Motorola went for a relatively large 1/2.6" format sensor, with 1.4µm pixels, a unique RGBC color filter array with single clear pixel, and an F/2.4 optical system.
The good news is that Motorola is dramatically improving the Moto X camera experience with an upcoming OTA update that’s rolling out today to T-Mobile Moto X owners, and hopefully eventually to other operators after testing and approval is completed.
Imaging performance improves dramatically indoors and out with this update. The update changes the tuning of the camera by improving exposure in outdoor and backlit scenes, white balance and color accuracy across the board, and reducing noise in low light scenes. I got the chance to play around with a Moto X with this update loaded on and of course brought along a Moto X without the update to compare side by side in my dual-camera bracket.
I have to say that the changes Motorola makes to the Moto X with this update are nothing short of the biggest I’ve ever seen come across in an OTA update. There’s a lot of performance that comes from properly tuning a system, and it’s obvious that the imaging team has retuned a lot of the imaging pipeline in the Moto X with this update, as a lot of things are fixed.
That white haziness that used to cloud so many outdoors photos is completely gone, instead replaced by tuning that yields more contrasty results without that same kind of haze. White balance also improves outdoors, sometimes images had a blueish cast to them, this is now a bit warmer when appropriate. Colors also seem to pop a lot better. Outdoors the Moto X really performs a lot better thanks to improvements to auto exposure which now no longer randomly overexposes some scenes. The noise reduction algorithm that was running has also been turned down dramatically, leaving a lot more high spatial frequency detail in images, which is visible in trees and bricks especially in my sample images. I definitely prefer camera tuning that passes more detail at the expense of also passing more chroma noise, it seems that Motorola has gone that way as well with this update.
In low light the Moto X shows much of what it does outdoors – fixed white balance even under challenging sodium light sources, dramatically less noise reduction which passes through a lot more detail. Images look less like oil paintings, in the sample photo of the test scene more detail on the book makes it through, including those narrow lines which previously blurred together. There’s more chroma noise but again I like this tradeoff.
Overall I’m hugely impressed with the improvements that Motorola made to the Moto X camera with this update. I've been carrying the Moto X as my daily for a while now and lacking imaging performance was my only major concern anymore, with this update, the Moto X moves up quite a bit in my mind. It’s great to see the Moto X move a lot closer to the imaging performance that I expected given the impressive specs and emphasis that clearly was put on that axis of performance.
Some Thoughts about the iPhone 5S Camera Improvements
One of the big improvements that comes with the iPhone 5S is a new camera system, with a faster aperture for more light throughput, bigger 8 MP sensor with correspondingly bigger pixels, dual LED color temperature matching flash, and improved ISP. I thought it worth going over some of the changes before we look at it in the review since there’s already honestly quite a lot that I’ve gathered about the 5S system just from examining EXIF on the images from Apple’s uploaded sample images gallery, what I saw in the demo room on a 5S, and what has been said publicly.
Camera has become one of the major axes of differentiation in the smartphone space. It doesn’t take much inspection to see that it has become an emphasis everywhere. Nokia has had halo devices for a long time now which establish its position with dominant smartphone camera leadership, HTC recently pushed its imaging emphasis very far with the One’s camera system and arguably the system before it for the One X and S, and even the recent Moto X made an attempt to do something fundamentally different with a color filter array including a clear pixel. Obviously the iPhone sits somewhere in that fray if nothing else since Apple really started demonstrating something above normal camera competency around the 3GS and 4 by shipping a system of its own design and specification. Since then, Apple has continued to push parts of its imaging chain further with a custom Sony CMOS starting with the 5 and custom ISP with the 4S. Obviously the statistic that Apple will cite is that the iPhone cameras take the top 3 spots on Flickr for camera popularity, making them arguably some of the most important cameras out there. Given my optical engineering background, watching smartphone cameras evolve and move in different directions is of course particularly interesting since it’s a place in the camera world with unusual constraints. As a photographer I’m also interested in finding ultimately what device has the best usability to camera tradeoff ratio, and Apple has historically made sound choices with its system.
With the iPhone 5S camera, Apple makes a number of interesting changes that I already touched on. Rather than march down the pixel size roadmap to 1.1µm (and beyond that the upcoming 0.9µm step) to either increase pixel count with an equivalent sized sensor, or shrink the sensor and optical stack and maintain the same number of pixels, Apple has chosen a strategy like HTC’s and gone the other way. Apple has increased pixel size to 1.5µm (from 1.4µm), while keeping pixel count the same, thus creating a larger sensor. I remember telling Anand and a number of other people that if Apple even just stayed at 1.4µm for the 5S, it would mean validation of everything I ever said about the 1.1µm size and beyond in my prior smartphone imaging and optics presentation.
Obviously seeing Apple put up a "bigger pixels = better picture" slide during the keynote and move the opposite direction by bucking the trend just like HTC did makes me a lot more comfortable about the future of Apple’s smartphone imaging roadmap. Increasing pixel size is arguably the better but simultaneously harder direction for the industry to increase image quality, low light sensitivity, and SNR. Keep in mind that I’m talking about pixel pitch here, so Apple’s increase from 1.4µm to 1.5µm pixels really means a 14.8 percent increase in pixel area and thus integration area. If you do the math out, Apple moves from the relatively standard 1/3.2“ CMOS sensor size to around 1/3” in size, similar to the HTC One. Increasing sensor size and pixel size really will make a measurable difference in low light sensitivity, noise, and dynamic range. I don’t know any specifics but given Apple’s ability to get Sony to make a one-off custom CMOS for the iPhone 5, I wouldn’t be surprised to see Sony supply a sensor of Apple’s specification this time for the 5S, one of the nice things about having their kind of volume.
At the same time, the optical design has improved to increase the amount of light passed through the system. Apple’s 4S and 5 systems were F/2.4, the iPhone 5S system moves down a quarter stop to F/2.2. I was hoping for F/2.0 like we see a number of other OEMs shipping (HTC, Nokia) but F/2.2 might be a logical tradeoff to keep aberrations down and not run into some of the stray light issues that I’ve seen crop up in those other systems. The 5 had stray light issues that famously resulted in some purple fringing already at F/2.4, which people falsely attributed to the sapphire cover glass (which is actually colorless). It remains to be seen whether the lower F/# on the 5S increases them, though I wager the attention given to this issue probably resulted in some correspondingly better anti reflection coatings and stray light management.
Focal length changes slightly, from 4.10 mm in the 5 to 4.12 mm in the 5S. The difference in sensor size and thus crop factor gives a 35mm equivalent focal length of around 31mm in the 5 and 29.7mm in the 5S. Shorter focal length has generally been a tradeoff for a while to decrease the z-height of the module further, and you see a lot of smartphone vendors hovering around 28mm or slightly lower, which is relatively wide. Marketing will then turn around and market the wider angle like it’s some positive thing, which is hilarious. Apple has thus far been steadfast at staying around 30mm, but there’s no denying that the 5S will indeed be a bit wider angle than the 5 in practice. Ideally I’d love to have a smartphone module with around a 35mm focal length in 35mm equivalent numbers.
The most visible and talked about change for the 5S imaging system thus far is the dual LED “true tone” flash, which really is a system with two LEDs of different color temperature. Spectral output of LEDs tend to be a bunch of narrow bandwidth spikes, which results in weird color rendering (analogous to the sometimes poor color rendering index of LED lightbulbs or CCFLs). In addition, white LED flashes just end up having a blue tint compared to Xenon. The solution is to add another color temperature alongside, then mix the two LEDs to get the desired color temperature that matches the scene. I actually saw NVIDIA doing this dual color temperature LED flash mixing on their Tegra 4 reference tablet back before MWC, and this is something other OEMs have been talking about, Apple just has the first device to market with it.
I played around with the flash in the demo area, and the 5S system pre-flashes, computes the correct flash amplitude to match scene color temperature, and then fires a flash during capture with the right color temperature.
The result is a photo that does look better in terms of color rendering and temperature. Personally I still have a deep seated dislike of on-camera or direct flash, and only use it when natural light is absolutely insufficient to get a photo. I don’t think this will really change my feelings about on-camera flash, but if you do have to use flash to light a dark scene, this makes it a lot less terrible.
Finally there are improvements to ISP and video encode. The 5S includes a further improved ISP, though I have no way of knowing what’s new inside beyond what Apple stated during the keynote, and in general SoC ISP remains a closely guarded secret for all of the silicon players. Apple purports their new ISP has improved AWB, AE, new local tone mapping, new autofocus matrix metering with 15 focus zones, and automatic selection of the sharpest image in the capture buffer when you finally do press capture. Of course there’s also a new burst capture mode which activates when holding down the capture button and captures at 10 FPS. I held this down in the demo area and took around 500 images without the speed slowing down. Apple is probably buffering these in DRAM and writing them to NAND at the same time.
On the video side the 5S now includes a 120FPS capture mode. The camera UI has a “slow-mo” option beyond the video mode, which captures 720p120 video. The framerate in this mode in the camera preview is visibly faster, capture works like normal, but inside the playback UI are two scrubbers which let you play back that 120FPS video at 30 FPS, making it look 4x slower. I’m unclear whether the 5S makes the raw 720p120 video available, but I really hope that’s the case. I do wish that there was 1080p60 capture, but that might be exposed through the video capture APIs somewhere, again it’s still unclear.
What’s missing from the 5S is OIS. I didn’t ever think it was in the cards for the 5S, so its absence isn’t a surprise, but it’s a substantial improvement for video and longer exposures. The reality is that an increasing number of players are including it – Nokia, HTC, and LG, and that list will only continue getting larger. Its absence isn’t the end of the world, but the stronger OIS implementations make a substantial difference for both videos and still images. Apple’s going the electronic and computational route with further improvements to its EIS auto image stabilization, which combines the sharp parts of multiple images together to get a single sharp picture. Almost every smartphone camera system now has a back buffer of images coming from the sensor, Apple purports that it is able to do some computational analysis, grab sections of the last few images and combine them to produce a sharp result. This will help in good lighting where the system can grab a lot of images quickly with good exposure, but obviously doesn’t fundamentally solve the low light problem where the exposures themselves are still longer. Grabbing photographs without blur remains a challenging problem for everyone, obviously OIS doesn’t help with scenes where the subject is moving either.
I’ll add that I’m still unclear whether the 2x2 pixel binning remains in place with the 5S camera. I would be surprised if it wasn’t in place however, especially given the 720p120 recording mode which would require it to get the higher sensitivity required for higher framerate. Higher framerate video capture means less integration time per frame to gather light.
Selected EXIF from Apple's iPhone 5S Sample Photos:
Make : Apple
Camera Model Name : iPhone 5s
Exposure Time : 1/1866
F Number : 2.2
Exposure Program : Program AE
ISO : 32
Metering Mode : Spot
Flash : Off, Did not fire
Focal Length : 4.1 mm
Focal Length In 35mm Format : 30 mm
GPS Altitude : 15.8 m Above Sea Level
GPS Latitude : 38 deg 1' 22.22" N
GPS Longitude : 122 deg 31' 22.23" W
GPS Position : 38 deg 1' 22.22" N, 122 deg 31' 22.23" W
Image Size : 3264x2448
A few iPhones ago, Apple started posting sample images from their new iPhone camera in full resolution taken straight from the device on their own site right after the announcement. That has turned into common practice now, and the 5S is no exception. I’ve gone over the images each time and looked at EXIF for whatever info is available (and the GPS tags that generally were left on), and did the same for the 5S. Interestingly enough Apple only left location tagging on for one of the 6 images (the skateboard pic) this time around.
5S: 1/2740, ISO 32
The photo samples look very good. None of the photos go over ISO 100, in fact most of them are at a very low ISO 32 which keeps noise down. Exposure times are also very fast. Only image 3 is somewhat low light looking. I was hoping Apple would include a few samples showing low light performance indoors with and without flash, but I guess I’m not surprised Apple would pick scenarios that show off the best quality rather than situations that might strain it. For that we’ll have to wait.
Image number one which is a macro shot of a flower has a nice looking blurry background effect without distracting or artificial looking bokeh. For a smartphone camera this looks very good.
Image number two is a top down shot of some chilis on a wood table, relatively planar. What’s awesome here is it lets me see immediately that there’s good sharpness across the entire field of view – the edges don’t get soft like a number of other smartphones do. Maintaining good MTF across the entire field of view is difficult, especially at extreme field angles like the corners. While there’s definitely some falloff, it’s very controlled on this 5S sample photo. In addition there’s no weird distortion, the horizontal grains in the wood remain horizontal for example.
Image number three is of a jellyfish, and is the only somewhat lower light photo, though I’m not convinced this is what I’d consider low light. It looks good but moreover illustrates that the auto exposure algorithm in the 5S isn’t just blindly taking exposure over the whole scene (the black aquarium) – spot metering mode is noted in the EXIF indicating the user tapped the jellyfish.
Image four is an impressive sunset, although here the bokeh is a bit more distracting and weird looking in the background. Sharpness is retained however in the grass close to the sunset bloom, without washing out. I also don’t see any fringing.
Image five is of some kids in a pool, skin tones look great here against the water, sharpness is also great.
Image six has action nicely frozen and was probably captured using burst mode, if there’s any opportunistic image recombination here for the scene, I can’t detect it in the output result.
The camera UI in iOS 7 also gets a substantial overhaul. Gone is the still image capture preview which is fit to the long axis of the iPhone 5/5C/5S display, which crops off the top and bottom of the image. Thankfully, Apple has come to its senses and changed the UI so the full field of view of the image is visible, and what you see is now what you get. If only Google would now follow suit and do the same to the AOSP/stock Android UI.
The camera UI has completely different iconography and styling from the old one. Gone is the video toggle, and in its place is a mode ring which switches between slow-mo, videos, photo, square, and panorama. This eliminates some of the feature cruft that was piling up in the “options” button from the old UI. There’s also the filters option which shows a live preview grid of some filters on the image – think photo booth for iOS. My only complaint is that whereas the previous iOS camera UI had more visual cues that made it easy to confirm the camera detected proper portrait or landscape orientation, the iOS 7 camera really doesn’t. Only the thumbnail and flash/HDR/front camera icons rotate. Further, the text ring switcher doesn’t rotate, which adds some mental processing when you’re shooting in landscape (which you should, especially for video).
I explored the iOS 7 camera UI and slow motion features on the iPhone 5S a lot in our hands on video which I’d encourage you to check out.
Overall the improvements to the 5S camera system are very positive and I’m very happy to see Apple going the direction of bigger pixels rather than marching down the pitch size roadmap and trading off sensitivity. Larger pixels and bucking that trend is absolutely positively the right direction to go. If Apple went the other way I’d start getting concerned about the camera team over there. Obviously the choices made in the 5S do a lot to put me at ease and reassure that there’s still some sanity in the smartphone imaging space. Likewise the dual color temperature LED flash system is another low light improvement, even if I still dislike on-camera flash in general and will probably never use it – every iPhone I’ve ever owned has had flash set to off rather than auto. Faster F/# is a big improvement along that same axis as well, letting in more light and possibly giving more shallow depth of field at some focus positions. High framerate video coming to the 5S was a no brainer given the new APIs and features added to iOS 7, I just assumed it’d be 1080p60 rather than 720p120, although there’s still a chance that 1080p60 is an option through the capture API rather than the camera UI. Finally I’m very happy that Apple has sorted out the aspect ratio cropping issues in the camera UI that really frustrated me with the iPhone 5’s display ratio change.
Most of my time getting hands on with the iPhone 5S in the town hall demo room was spent playing with the camera UI, but there’s only so much you can really tell given a few minutes. Either way the iPhone 5S looks to be another big improvement to Apple’s very popular camera system.
