Mobiles @ NCSU

Apple appears to have some interest in Lytro-style refocusable cameras. In a patent filed in 2011 and granted today, Apple describes a camera that would be able to switch between capturing two types of images: low-resolution images that could be refocused after capture, and high-resolution photographs that can't be changed. To do both, the camera would include an adaptor with a "microlens array" that would slide between the camera's lens and sensor when capturing a refocusable image, and slide away when capturing a traditional photo. The image above depicts the cross section of such a camera while the adaptor's microlens array — labeled "430" — is positioned to shoot a refocusable image.

While a number of companies have attempted...

Continue reading…

One of the big improvements that comes with the iPhone 5S is a new camera system, with a faster aperture for more light throughput, bigger 8 MP sensor with correspondingly bigger pixels, dual LED color temperature matching flash, and improved ISP. I thought it worth going over some of the changes before we look at it in the review since there’s already honestly quite a lot that I’ve gathered about the 5S system just from examining EXIF on the images from Apple’s uploaded sample images gallery, what I saw in the demo room on a 5S, and what has been said publicly.

Camera has become one of the major axes of differentiation in the smartphone space. It doesn’t take much inspection to see that it has become an emphasis everywhere. Nokia has had halo devices for a long time now which establish its position with dominant smartphone camera leadership, HTC recently pushed its imaging emphasis very far with the One’s camera system and arguably the system before it for the One X and S, and even the recent Moto X made an attempt to do something fundamentally different with a color filter array including a clear pixel. Obviously the iPhone sits somewhere in that fray if nothing else since Apple really started demonstrating something above normal camera competency around the 3GS and 4 by shipping a system of its own design and specification. Since then, Apple has continued to push parts of its imaging chain further with a custom Sony CMOS starting with the 5 and custom ISP with the 4S. Obviously the statistic that Apple will cite is that the iPhone cameras take the top 3 spots on Flickr for camera popularity, making them arguably some of the most important cameras out there. Given my optical engineering background, watching smartphone cameras evolve and move in different directions is of course particularly interesting since it’s a place in the camera world with unusual constraints. As a photographer I’m also interested in finding ultimately what device has the best usability to camera tradeoff ratio, and Apple has historically made sound choices with its system.

With the iPhone 5S camera, Apple makes a number of interesting changes that I already touched on. Rather than march down the pixel size roadmap to 1.1µm (and beyond that the upcoming 0.9µm step) to either increase pixel count with an equivalent sized sensor, or shrink the sensor and optical stack and maintain the same number of pixels, Apple has chosen a strategy like HTC’s and gone the other way. Apple has increased pixel size to 1.5µm (from 1.4µm), while keeping pixel count the same, thus creating a larger sensor. I remember telling Anand and a number of other people that if Apple even just stayed at 1.4µm for the 5S, it would mean validation of everything I ever said about the 1.1µm size and beyond in my prior smartphone imaging and optics presentation.

Obviously seeing Apple put up a "bigger pixels = better picture" slide during the keynote and move the opposite direction by bucking the trend just like HTC did makes me a lot more comfortable about the future of Apple’s smartphone imaging roadmap. Increasing pixel size is arguably the better but simultaneously harder direction for the industry to increase image quality, low light sensitivity, and SNR. Keep in mind that I’m talking about pixel pitch here, so Apple’s increase from 1.4µm to 1.5µm pixels really means a 14.8 percent increase in pixel area and thus integration area. If you do the math out, Apple moves from the relatively standard 1/3.2“ CMOS sensor size to around 1/3” in size, similar to the HTC One. Increasing sensor size and pixel size really will make a measurable difference in low light sensitivity, noise, and dynamic range. I don’t know any specifics but given Apple’s ability to get Sony to make a one-off custom CMOS for the iPhone 5, I wouldn’t be surprised to see Sony supply a sensor of Apple’s specification this time for the 5S, one of the nice things about having their kind of volume.

At the same time, the optical design has improved to increase the amount of light passed through the system. Apple’s 4S and 5 systems were F/2.4, the iPhone 5S system moves down a quarter stop to F/2.2. I was hoping for F/2.0 like we see a number of other OEMs shipping (HTC, Nokia) but F/2.2 might be a logical tradeoff to keep aberrations down and not run into some of the stray light issues that I’ve seen crop up in those other systems. The 5 had stray light issues that famously resulted in some purple fringing already at F/2.4, which people falsely attributed to the sapphire cover glass (which is actually colorless). It remains to be seen whether the lower F/# on the 5S increases them, though I wager the attention given to this issue probably resulted in some correspondingly better anti reflection coatings and stray light management.

Focal length changes slightly, from 4.10 mm in the 5 to 4.12 mm in the 5S. The difference in sensor size and thus crop factor gives a 35mm equivalent focal length of around 31mm in the 5 and 29.7mm in the 5S. Shorter focal length has generally been a tradeoff for a while to decrease the z-height of the module further, and you see a lot of smartphone vendors hovering around 28mm or slightly lower, which is relatively wide. Marketing will then turn around and market the wider angle like it’s some positive thing, which is hilarious. Apple has thus far been steadfast at staying around 30mm, but there’s no denying that the 5S will indeed be a bit wider angle than the 5 in practice. Ideally I’d love to have a smartphone module with around a 35mm focal length in 35mm equivalent numbers.

The most visible and talked about change for the 5S imaging system thus far is the dual LED “true tone” flash, which really is a system with two LEDs of different color temperature. Spectral output of LEDs tend to be a bunch of narrow bandwidth spikes, which results in weird color rendering (analogous to the sometimes poor color rendering index of LED lightbulbs or CCFLs). In addition, white LED flashes just end up having a blue tint compared to Xenon. The solution is to add another color temperature alongside, then mix the two LEDs to get the desired color temperature that matches the scene. I actually saw NVIDIA doing this dual color temperature LED flash mixing on their Tegra 4 reference tablet back before MWC, and this is something other OEMs have been talking about, Apple just has the first device to market with it.

I played around with the flash in the demo area, and the 5S system pre-flashes, computes the correct flash amplitude to match scene color temperature, and then fires a flash during capture with the right color temperature.

The result is a photo that does look better in terms of color rendering and temperature. Personally I still have a deep seated dislike of on-camera or direct flash, and only use it when natural light is absolutely insufficient to get a photo. I don’t think this will really change my feelings about on-camera flash, but if you do have to use flash to light a dark scene, this makes it a lot less terrible.

Finally there are improvements to ISP and video encode. The 5S includes a further improved ISP, though I have no way of knowing what’s new inside beyond what Apple stated during the keynote, and in general SoC ISP remains a closely guarded secret for all of the silicon players. Apple purports their new ISP has improved AWB, AE, new local tone mapping, new autofocus matrix metering with 15 focus zones, and automatic selection of the sharpest image in the capture buffer when you finally do press capture. Of course there’s also a new burst capture mode which activates when holding down the capture button and captures at 10 FPS. I held this down in the demo area and took around 500 images without the speed slowing down. Apple is probably buffering these in DRAM and writing them to NAND at the same time.

On the video side the 5S now includes a 120FPS capture mode. The camera UI has a “slow-mo” option beyond the video mode, which captures 720p120 video. The framerate in this mode in the camera preview is visibly faster, capture works like normal, but inside the playback UI are two scrubbers which let you play back that 120FPS video at 30 FPS, making it look 4x slower. I’m unclear whether the 5S makes the raw 720p120 video available, but I really hope that’s the case. I do wish that there was 1080p60 capture, but that might be exposed through the video capture APIs somewhere, again it’s still unclear.

What’s missing from the 5S is OIS. I didn’t ever think it was in the cards for the 5S, so its absence isn’t a surprise, but it’s a substantial improvement for video and longer exposures. The reality is that an increasing number of players are including it – Nokia, HTC, and LG, and that list will only continue getting larger. Its absence isn’t the end of the world, but the stronger OIS implementations make a substantial difference for both videos and still images. Apple’s going the electronic and computational route with further improvements to its EIS auto image stabilization, which combines the sharp parts of multiple images together to get a single sharp picture. Almost every smartphone camera system now has a back buffer of images coming from the sensor, Apple purports that it is able to do some computational analysis, grab sections of the last few images and combine them to produce a sharp result. This will help in good lighting where the system can grab a lot of images quickly with good exposure, but obviously doesn’t fundamentally solve the low light problem where the exposures themselves are still longer. Grabbing photographs without blur remains a challenging problem for everyone, obviously OIS doesn’t help with scenes where the subject is moving either.

I’ll add that I’m still unclear whether the 2x2 pixel binning remains in place with the 5S camera. I would be surprised if it wasn’t in place however, especially given the 720p120 recording mode which would require it to get the higher sensitivity required for higher framerate. Higher framerate video capture means less integration time per frame to gather light.

Selected EXIF from Apple's iPhone 5S Sample Photos:

Make : Apple

Camera Model Name : iPhone 5s

Exposure Time : 1/1866

F Number : 2.2

Exposure Program : Program AE

ISO : 32

Metering Mode : Spot

Flash : Off, Did not fire

Focal Length : 4.1 mm

Focal Length In 35mm Format : 30 mm

GPS Altitude : 15.8 m Above Sea Level

GPS Latitude : 38 deg 1' 22.22" N

GPS Longitude : 122 deg 31' 22.23" W

GPS Position : 38 deg 1' 22.22" N, 122 deg 31' 22.23" W

Image Size : 3264x2448

A few iPhones ago, Apple started posting sample images from their new iPhone camera in full resolution taken straight from the device on their own site right after the announcement. That has turned into common practice now, and the 5S is no exception. I’ve gone over the images each time and looked at EXIF for whatever info is available (and the GPS tags that generally were left on), and did the same for the 5S. Interestingly enough Apple only left location tagging on for one of the 6 images (the skateboard pic) this time around.

5S: 1/2740, ISO 32

The photo samples look very good. None of the photos go over ISO 100, in fact most of them are at a very low ISO 32 which keeps noise down. Exposure times are also very fast. Only image 3 is somewhat low light looking. I was hoping Apple would include a few samples showing low light performance indoors with and without flash, but I guess I’m not surprised Apple would pick scenarios that show off the best quality rather than situations that might strain it. For that we’ll have to wait. Image number one which is a macro shot of a flower has a nice looking blurry background effect without distracting or artificial looking bokeh. For a smartphone camera this looks very good. Image number two is a top down shot of some chilis on a wood table, relatively planar. What’s awesome here is it lets me see immediately that there’s good sharpness across the entire field of view – the edges don’t get soft like a number of other smartphones do. Maintaining good MTF across the entire field of view is difficult, especially at extreme field angles like the corners. While there’s definitely some falloff, it’s very controlled on this 5S sample photo. In addition there’s no weird distortion, the horizontal grains in the wood remain horizontal for example. Image number three is of a jellyfish, and is the only somewhat lower light photo, though I’m not convinced this is what I’d consider low light. It looks good but moreover illustrates that the auto exposure algorithm in the 5S isn’t just blindly taking exposure over the whole scene (the black aquarium) – spot metering mode is noted in the EXIF indicating the user tapped the jellyfish. Image four is an impressive sunset, although here the bokeh is a bit more distracting and weird looking in the background. Sharpness is retained however in the grass close to the sunset bloom, without washing out. I also don’t see any fringing. Image five is of some kids in a pool, skin tones look great here against the water, sharpness is also great. Image six has action nicely frozen and was probably captured using burst mode, if there’s any opportunistic image recombination here for the scene, I can’t detect it in the output result. The camera UI in iOS 7 also gets a substantial overhaul. Gone is the still image capture preview which is fit to the long axis of the iPhone 5/5C/5S display, which crops off the top and bottom of the image. Thankfully, Apple has come to its senses and changed the UI so the full field of view of the image is visible, and what you see is now what you get. If only Google would now follow suit and do the same to the AOSP/stock Android UI. The camera UI has completely different iconography and styling from the old one. Gone is the video toggle, and in its place is a mode ring which switches between slow-mo, videos, photo, square, and panorama. This eliminates some of the feature cruft that was piling up in the “options” button from the old UI. There’s also the filters option which shows a live preview grid of some filters on the image – think photo booth for iOS. My only complaint is that whereas the previous iOS camera UI had more visual cues that made it easy to confirm the camera detected proper portrait or landscape orientation, the iOS 7 camera really doesn’t. Only the thumbnail and flash/HDR/front camera icons rotate. Further, the text ring switcher doesn’t rotate, which adds some mental processing when you’re shooting in landscape (which you should, especially for video). I explored the iOS 7 camera UI and slow motion features on the iPhone 5S a lot in our hands on video which I’d encourage you to check out. Overall the improvements to the 5S camera system are very positive and I’m very happy to see Apple going the direction of bigger pixels rather than marching down the pitch size roadmap and trading off sensitivity. Larger pixels and bucking that trend is absolutely positively the right direction to go. If Apple went the other way I’d start getting concerned about the camera team over there. Obviously the choices made in the 5S do a lot to put me at ease and reassure that there’s still some sanity in the smartphone imaging space. Likewise the dual color temperature LED flash system is another low light improvement, even if I still dislike on-camera flash in general and will probably never use it – every iPhone I’ve ever owned has had flash set to off rather than auto. Faster F/# is a big improvement along that same axis as well, letting in more light and possibly giving more shallow depth of field at some focus positions. High framerate video coming to the 5S was a no brainer given the new APIs and features added to iOS 7, I just assumed it’d be 1080p60 rather than 720p120, although there’s still a chance that 1080p60 is an option through the capture API rather than the camera UI. Finally I’m very happy that Apple has sorted out the aspect ratio cropping issues in the camera UI that really frustrated me with the iPhone 5’s display ratio change. Most of my time getting hands on with the iPhone 5S in the town hall demo room was spent playing with the camera UI, but there’s only so much you can really tell given a few minutes. Either way the iPhone 5S looks to be another big improvement to Apple’s very popular camera system.

Hari Gottipati, writing at GigaOM:

Apple has avoided NFC, and all the rumors about NFC getting added to iPhone 4 and iPhone 5 are turned out to be false. Instead of NFC, Apple worked on alternatives using Wi-Fi and Bluetooth. During the introduction of iOS 7′s AirDrop at WWDC in June, Apple’s mobile development chief Craig Federighi said, “There’s no need to wander around the room, bumping your phone,” referring how NFC phones need to be very close to transfer the data.

iBeacon has flown a bit under the radar thus far. Could be huge combined with Touch ID.

I got this one wrong.

I fixed my thinking by this week, but as of a month ago, I had it wrong when I wrote “The Case for a New Lower-Cost iPhone”.

Here’s the thing. The iPhone 5C has nothing to do with price. It probably does have something to do with manufacturing costs (which are lower for Apple), but not price. Apple’s years-long strategy hasn’t really changed. They offer three phones:

1. This year’s, with the latest technology.
2. Last years’s, starting $100 lower.
3. The two-year-old model, with meager storage, free on contract, $200 lower unsubsidized.

It’s just that instead of putting the year-old iPhone 5 in slot #2, they’ve created the 5C to debut in that slot. The 5C is, effectively, an iPhone 5. Same A6, same camera, same just about everything — except for the most obvious difference, its array of colorful plastic shells. This is not an iPod Touch with a cellular antenna (the iPod Touch, which was not updated today, still has an A5 chip and roughly 4S specs). The prices of the iPhone tiers remain the same as last year. What changes with the 5C is that the middle tier is suddenly more appealing, and has a brand of its own that Apple can promote apart from the flagship 5S.

In marketing, what looks new is new.

Yes, it’s plastic, but there’s nothing cheap about it. It has a far better fit and finish, and feels way better in your hand, than Apple’s previous foray into plastic iPhones, the 3G and 3GS. The 5C feels like a premium product.

This move is about establishing the iPhone as a two-sibling family, like how the MacBooks have both the Airs and the Pros. Think of the 5C as the Air, and the 5S as the Pro. Or iMac and Mac Pro. The iPhone is growing up as a product family.

This is the first year when last year’s specs remain good enough to serve as the mass market new iPhone. Take a look at apple.com today and note which new iPhone appears first: the 5C, not the 5S. Which phone did they show a commercial for during the event? The 5C. Part of this too is that the 5C is going to be available in greater numbers sooner. Apple is taking pre-orders for the 5C but not the 5S because, I have reason to believe, they expect the 5S to be in constrained supply. That’s not surprising — plastic is easier to manufacture than aluminum, and the 5C’s components are all a year old. And it makes sense to promote the phone that you can actually fulfill demand for.

Schiller repeated, almost mantra-like, that the 5S was Apple’s “most forward-thinking iPhone”. In his wrap-up, Tim Cook echoed that line. This isn’t about downplaying the 5S, but rather, I think, about establishing the 5S as the top tier in what is now a two-tier lineup. The Lexus to the 5C’s Toyota; the Banana Republic to the 5C’s Gap. (The 4S is Old Navy.) Soon enough, all iOS devices will have 64-bit CPUs, motion-tracking sub-systems, fingerprint sensors, and point-and-shoot caliber cameras. But you get those things first in the iPhone 5S.

Some other thoughts and hands-on experiences from today’s event:

iPhone 5C

The default wallpapers, color coordinated to match the device’s hardware, make for a neat effect. iOS 7’s home screen parallax effect seems designed for this — it’s like a software version of the old see-through iMac hardware, as though your home and lock screens let you see through to the back of the device. (All the wallpapers are available to all iOS 7 users. Interesting that Apple chose not to default to the dynamic/animated ones, but there are color coordinated versions of those too.)

As stated above, the 5C hardware feels solid. And the hardware colors are exactly in line with the new color palette of iOS 7. Looks like an iPod-style “fun” design, way less serious than the 5S or any previous iPhone. I think iOS 7 looks great on the iPhone 5 and 5S, but it looks perfect on the 5C.

One interesting note: the 5C and 5S have a price overlap at $199. At that price, you can get either a 32 GB 5C or a 16 GB 5S. For those who don’t care about technical specs, I think the $199 5C is going to win.

iPhone 5S

Apple’s pitch on the 5S is remarkably simple.

First, it’s faster (seemingly way faster, far faster than the mere “S” tacked onto the end of its name would imply). They’re calling it “desktop caliber” performance, and I don’t think they’re exaggerating. Now that they’ve gone 64-bit, I’ve got to start wondering about ARM-based MacBooks in the near future.

Next, it has an intriguing “motion coprocessor”, which I think pretty much means you can use your 5S as a fitness tracker with almost no effect on your battery life. Fitness apps don’t need to be running in the background — instead, the M7 tracks your motion and the apps can read the logged data when you launch them. That’s huge — and perhaps bad news for dedicated fitness tracking hardware like Fitbit.

The camera is seriously upgraded. The slow-motion video mode is simple and fun, and the burst mode for stills is terrific. When you take a burst batch, after you pick which one (or ones) you want to keep, you can delete the remainder of the burst batch in one action. So you can take 30 photos in a three-second burst, pick two, and delete the other 28 all at once. Seems perfect. With no hyperbole, I think Apple is gunning to obviate the point-and-shoot camera industry. They’re on the short end of the stick when it comes to optics — the iPhone camera is small, and size (lens, length, sensor) matters in photography. But they’re working wonders on that end, and when it comes to software, I don’t see how traditional camera companies like Canon, Nikon, and Fuji can compete.

Lastly, there’s the Touch ID sensor. It’s fairly quick to train, and once trained, it is really fast, and in my brief hands-on testing, very accurate. The optimal way to use it to unlock your phone seems to be to tap the home button once to wake the phone, and then just keep your thumb or finger on the button for just another moment. Boom, unlocked. It’s very impressive technology. I already feel silly tapping in my passcode to unlock my iPhone.

When it comes to colors, the silver model seemed more or less unchanged from last year’s silver 5. The black one, which Apple is calling “space gray”, definitely has a different anodized color. It’s lighter and more metallic looking. I like it, and I think it will age better when you get wear and tear along the edges.

Then there’s goldie. I can’t say it’s my cup of tea, but it does look good, and not at all cheesy. Maybe a little blingy, but my hunch is that it’s going to prove wildly popular. If one color 5S proves harder to get than the others, it’s going to be the gold one I think.

iPods and iPads

No news at all is the only news regarding iPods. Apple has never held a “music event” without at least some new iPod models, but this year, nothing. Maybe next month, alongside new iPads — or maybe they’re going to stick with last year’s lineup for another year. I’m not sure what to expect.

If they’re holding new iPods until next month, my guess it’s because they wanted the iPhone 5C to be the only “available in an array of five bright colors” product in the spotlight today.

Speaking of next month, last year Apple held events on September 11 (iPhone 5, iPods) and October 23 (iPads). I feel pretty sure I’ll be flying back out next month.

As expected, Apple has taken the wraps off of its latest flagship smartphone at its Cupertino media event today. The new iPhone 5S is an upgrade in the vein of the previous iPhone 3GS and iPhone 4S—it is outwardly similar to last year's iPhone 5, but it includes a few key upgrades to keep the design going for another year.

The 5S still has a four-inch 1136×640 display and a Lightning connector, and it retains the iPhone 5's taller and slimmer profile relative to older models like the iPhone 4 and iPhone 4S. However, it includes a new 64-bit "desktop-class" system-on-a-chip (SoC) dubbed the A7 that boosts CPU and GPU performance over the A6 in the iPhone 5.

The chip has twice the general-purpose and floating point registers of its predecessor and is up to twice as fast at performing CPU tasks. The phone supports OpenGL ES 3.0, and Apple claims that the graphics performance in the 5S is 56 times better than in the original iPhone released six years ago.

Read 12 remaining paragraphs