On his blog, Crow explains the improved editing capabilities that are possible using high dynamic range wide gamut image formats (and editing applications that support these formats.) In this image comparison of an 8-bit JPEG and a 16-bit HD Photo file, Crow demonstrates how you can pull out a wider range of details. Notice that the top yellow balloon is translucent in the HD Photo file, a detail that gets lost in the JPEG.
There’s nothing that makes photographers want to run in the other direction more than the idea of switching to a new image file format. But the claims that Microsoft is making for its recently developed HD Photo format (formerly called Windows Media Photo) may be strong enough to make you turn around and take a look.
HD Photo is designed to combine the convenience and space efficiency of the aging JPEG format with much of the flexibility of RAW images. By retaining and decoding data in ways that a JPEG can’t, this new file type could facilitate a broad range of improvements and innovations in the way we shoot, edit, print, and display images.
Microsoft’s HD Photo program manager, Bill Crow, sat down with me after his demonstration at the company’s recent Pro Photo Summit to talk about the advantages of HD Photo, the innovations it might enable, and why it might succeed in making its way into cameras where JPEG 2000 failed.
Jump to a section:
• HD Photo Development
• Software and OS Compatibility
• HD Photo versus JPEG
• HD Photo versus JPEG 2000
• HD Photo versus Raw
• HD Photo and Printing
• Camera Manufacturer Adoption
• Likely Users
• HD Photo and HDR Images
HD Photo Development
Aimee Baldridge: How long has HD Photo been around?
Bill Crow: I’ve been working on it since we really formed the idea of a file format. A lot of the core technology, the compression technology, probably goes back as much as 10 years in Microsoft research. That’s the basis for it. We first started talking publicly about it probably about a year ago. We presented a paper at the Windows Hardware Engineering Conference, called WinHEC, last year, and I spoke here at this event [Microsoft's Pro Photo Summit] last summer about it. We were providing advanced versions of developer kits for our partners that far back. But the official release of HD Photo was really tied to the release of Vista. It’s all built into Windows Vista.
Software and OS Compatibility
AB: When you say it’s built into Windows Vista, what does that mean exactly?
BC: Windows Vista recognizes lots of different file types, standard file types. For images it recognizes JPEG and TIFF and DNGs and BMP files — and HD Photo files. So the photo experience that’s built into Vista — the fact that you can see thumbnails of images in a folder, the whole Photo Gallery image organizer and editor — that all fully supports HD Photo, along with the other formats. And the HD Photo codec is available for any application developer for Windows Vista through a standard set of APIs, the same APIs they would use to access JPEG or TIFF or the other formats.
AB: Is HD Photo available in image-editing programs?
BC: We’ve just really made it available this year. We have available a plug-in for Photoshop. That’s a free download from Microsoft. Microsoft has an image-editing program, a full graphic arts program that’s built into our Expression family of products. Microsoft Expression is a suite of applications. One of those apps is called Expression Design, and Expression Design supports HD Photo. There is at least one free image-editing program that’s available called Paint.NET, and they’ve added HD Photo support. I get random e-mails from other people telling me they’re putting it in programs I’ve never heard of, so we’re seeing that start to grow.
HD Photo versus JPEG
AB: What are the advantages of HD Photo as opposed to other file formats, like JPEG?
BC: I love JPEG. I really do. We wouldn’t be here today if it wasn’t for JPEG. I think there are three big advantages: The first is the fact that we support a wide range of pixel formats, including high dynamic range, wide gamut formats. JPEG is always used as an 8-bit per channel image format, and so all the information that comes from a camera’s sensor gets subsetted, if you will. It gets clipped to the limits of a JPEG file when it is stored in JPEG mode, which is the principal reason that so many photographers switch to raw — not to suffer that limit, that loss of data. So they have more control over the editing decision as opposed to the camera making automatic decisions when it makes a JPEG file.
With HD Photo we can operate in that same mode. It can be used in 8-bit mode. But we also support higher bit depths, using fixed-point or floating-point numbers. It allows, for example, a camera to make an automatic exposure adjustment decision and produce a ready-to-use file like a JPEG file, an RGB file. The adjustments have been made, but that file preserves the data that would otherwise have been clipped at the gamut and dynamic range limits of the JPEG file. It’s still numerically encoded in the file. So that allows [me to] make an adjustment to the photo after the fact and recover lost highlights or recover colors that might otherwise have been outside of the gamut.
[HD Photo provides] support for lots of different pixel formats. We looked at a demonstration of one of those pixel formats, but we also support CMYK formats and monochrome formats; we support alpha channels. We actually have an n-channel mode where we can have up to eight channels where it’s up to whatever the application is to decide what those channels mean. We tried to build a great deal of flexibility into the format.
The second big advantage is the compression efficiency. HD Photo’s compression algorithm is on average about twice the efficiency of JPEG. That means the same quality in a file that’s half the size. Or, as I demonstrated today, a file that has a lot more content, 16-bit per channel content В… still the same file size.
HD Photo also supports lossless compression, for very demanding applications where you want the highest quality. But it’s all done in the exact same algorithm. Lossless compression is simply our highest-quality mode. When you set the quality to the upper limit it will be mathematically lossless. JPEG still has an upper limit to how good the quality can be from a compression standpoint, just based on the algorithm. We don’t have a limit.
So, flexible pixel formats including high dynamic range is number one. Much better compression technology in a variety of dimensions is number two. And the third [advantage] is the ability to progressively decode an image. You really saw the examples of that not so much from my demonstration but from the Photosynth demonstration and from the HD View gigapixel image viewer demonstration. Both of those were different types of examples where [someone had] gigapixels of image content up on a server and wanted to be able to display them on a PC very smoothly, fluidly, without delay. In the case of Photosynth, those gigapixels were made up of two or three hundred 8-megapixel images. In the case of the gigapixel viewer it was one big 4.3-gigapixel image, but both of those use HD Photo as the underlying file format.
Both of them are based on a similar premise: It doesn’t matter how much content I have up on the server. At any given time I can only use so many pixels to fill the screen; the screen is only so big, and that’s fixed. I could have just terapixels of information on the server but at any given point in time I can only use so many of those pixels. The key is, can I get to the pixels that I need? If I’m using JPEG files, I really have no choice but to download an entire JPEG file. Even if I only need a small portion of it, or I need to see that whole file as a small thumbnail, I have to either have multiple versions of the file that I choose from, or I download the entire file, then I crop it and resize it and that’s what I put on the screen. That’s hugely inefficient. Photosynth could never work if that’s what was required behind the scenes.
The organization of an HD Photo file makes it possible for me to decode from this big stream of compressed data just the portion of the compressed data I need for any given region of the file or any given resolution of the file. The big innovation here is how we organize that compressed data inside the file that lets me index into it and extract subsets of it. We call that progressive decoding.
To give credit where credit is due, a portion of this progressive decoding capability is available in JPEG 2000. So a lot of those same things could be done using JPEG 2000, but there are a bunch of other reasons that we’ve just never seen JPEG 2000 achieve any success for broad consumer use. So while it solves that problem, there are other aspects of the spectrum of digital photography requirements which the industry has just not found JPEG 2000 to be a viable solution for.
HD Photo versus JPEG 2000
AB: If JPEG 2000 wasn’t adopted widely, why will HD Photo be adopted?
BC: I think HD Photo does meet the requirements that aren’t being addressed by JPEG 2000. Clearly one of the challenges that JPEG 2000 has faced is that it does offer certainly much better compression, and it does provide some of these progressive decoding capabilities, but it’s actually a fairly complex algorithm from a mathematical standpoint. To achieve its quality requirements, it runs in a floating-point mode. That’s very efficient — or can be reasonably efficient — on a PC, because a PC has a whole floating-point coprocessor. But that is very difficult to implement in a small, low-power device like a digital camera. So, we have never seen any camera manufacturer ever put support for JPEG 2000 in a camera.
HD Photo versus RAW
AB: How does HD Photo measure up to the capabilities that you get with RAW processing?
BC: I don’t think HD Photo is ever a complete replacement for RAW, but I believe the reason that the majority of photographers who shoot RAW today use RAW is for two very important capabilities, and they’re really just two ways to express the same capability: to be able to recover highlights that may have been lost, to have that extra one, one-and-a-half, maybe even two stops of latitude that is not represented in a JPEG file, and to be able to adjust the white balance after the fact.
By using HD Photo in a high-dynamic-range mode, a 16-bit fixed-point mode for example, I can preserve all the information that came from the camera’s sensor and make it possible to make a white balance change after the fact, to make exposure adjustments that are based on having all the sensor data available.
HD Photo and Printing
There’s another end of the equation where HD Photo becomes an important component, and that’s printing. There are entire books written today on printing workflows, and I’m sure your magazine runs articles regularly explaining the painful step-by-step process. And why do people do that as opposed to just right-click, print on their JPEG file? Well, it’s because they get fundamentally different and better-quality results. And the reason for that is dealing with the fact that the printer has its own color space, which is different from the sRGB color space that I’m working with.
The main reason we go through these complex workflows is to avoid [the photo] getting reduced down to a least common denominator sRGB before it gets handed to the printer, to try to preserve the whole color range.
If I have a file format that captures and holds onto the stuff that might otherwise go out of the limits of the visible range, then I believe it’s absolutely possible that you can have that HD Photo file that looks great on your screen and you can right-click and print from the desktop and get the same really high-quality print that today requires that much more complex workflow.
Camera Manufacturer Adoption
AB: Is there anything you can say about camera manufacturer adoption of HD Photo?
BC: No, I obviously can’t speak for anything that those manufacturers haven’t announced. Cameras come from manufacturers that we’re all familiar with, whether it’s Canon or Nikon or Fuji or Kodak. But most of those cameras use chipsets that come from another whole collection of companies — names that we’re not that familiar with — Sunplus and MediaTek and Novatek and Benq. So the first step toward adoption for HD Photo in devices is actually to be supported in those chipsets. There have been some announcements from at least two or three of those chipset manufacturers that they now support HD Photo and have it available in their chipset. Every one of those chips is a custom chip based on the specs of one of their customers, camera manufacturers.
There is a company I’m familiar with in the U.S. called Picture Elements. And Picture Elements is implementing HD Photo in Verilog. We’re getting into the gory technical depths. Verilog is a language that’s used for companies that make chips. It’s how you describe an algorithm as a set of gates. A Verilog system lets you build the whole thing in software, simulate it before you start actually stamping out chips. So that’s another one of the steps along the road that we have to follow before it actually gets into a camera. It just doesn’t show up in a camera tomorrow because a manufacturer wanted it.
Likely Users
AB: Assuming it does come to market, who would you expect to use HD Photo, from snapshot photographers and general consumers to professionals?
BC: We designed HD Photo to address all of those needs. If there can be just one format which is the digital photo format and it has different ways it can be used for all the different users’ needs, that’s the real goal we’ve been going for with this.
We have believed for a while — and one reason I’m very much engaged with this whole community — that serious amateur photographers, prosumer photographers are potentially going to be an early adopter for this format. They understand the limitations of JPEG and they’re looking for better solutions. They tend to be an audience that’s ready to try out the latest new thing and learn about it and take advantage of it.
Maybe it’s the cell phone manufacturer who’s just making smaller files. If you can cut down the size of the file so the amount of bandwidth I use to transmit the picture over my infrastructure is cut in half, so the amount of battery power that’s needed to run the transmitter of the phone to transmit that file is cut in half, so the amount of time the user has to wait until their file is sent and they can send the next one or use the phone for something else is cut in half В… maybe that’s going to be one of the early adopters.
HD Photo and HDR Images
AB: Could you go back to what you were saying about high dynamic range and explain how HD Photo images might look to someone shooting in a format that is capable of high dynamic range?
BC: When a high dynamic range image comes out of my camera, it’s going to look just like the JPEG image. The difference is when I want to make a change to it.
I think there are all kinds of neat, innovative ideas that could be done. One of the things many photographers do today is shoot bracketed exposure sequences. They shoot a range of photos and Photoshop has this capability built in to build a real high dynamic range image. Well, the camera could potentially do that in the camera. I’m not a camera expert. I don’t know how to do this, but I imagine a camera manufacturer can probably invent ways to do this. When they open that shutter and expose that sensor, maybe they can capture that information multiple times at the sensor at essentially different ISO settings or actually just dynamically change the aperture and scan the sensor multiple times and essentially create a bracketed sequence, but go ahead and assemble that as high dynamic range data in the camera. HD Photo gives them a format that would make that possible, whereas JPEG certainly can’t do that, and RAW files don’t do that because RAW files are just the RAW sensor data.
I know we will have been successful with HD Photo when a camera manufacturer or a software manufacturer comes out with something that the format made possible that I never would have thought of. That to me is the big payoff.
Read the rest of this entry »