Apple Embraces HEVC: What Does it Mean for Encoding?
From a codec perspective, the key announcement at Apple's Worldwide Developer Conference (WWDC) 2017 was the inclusion of HEVC into HTTP Live Streaming (HLS). Streaming Media's Dan Rayburn covered the significance of that announcement to the market as a whole in his article, "Why Apple's HEVC Announcement Is A Big Step Forward For The Streaming Media Industry." You can tell from the title that Rayburn thinks it's a big deal, and I agree. In this article, I'll focus on exactly what Apple announced regarding HEVC and its implications from a technical perspective. I'll also discuss the new subtitle format supported in HLS, and a new image format that Apple launched.
HEVC Support in High Sierra (macOS) and iOS 11
I gathered much of the information presented below from WWDC Session 504, "Advances in HTTP Live Streaming," by Roger Pantos, the original author of the HLS specification, and Anil Katti, both AV Foundation engineers.
Here's what I learned. First, Apple will support HEVC in High Sierra, the next MacOS, and iOS 11, and both will be available as free upgrades this fall. Figure 1 identifies all devices that will be compatible with iOS 11.
Figure 1. iOS devices compatible with iOS 11
As shown in Figure 2, Apple will support HEVC via a combination of hardware and software decoding, and you should expect software decoding to consume more power and shorten battery life. For example, tests on a Dell notebook performed by TechSpot found a 39% drop in battery life when playing HEVC without hardware support as compared to H.264 with hardware support. Until proven otherwise, I would expect similar decreases in battery life with Apple mobile devices and notebooks using software-only HEVC decode. On a positive note, on devices with HEVC and H.264 hardware support, the TechSpot article found very little difference in battery life when playing the two formats.
Figure 2. Hardware vs. software support for HEVC in iOS 11 and MacOS
Which devices does this impact? Looking at iPhones only, according to Wikipedia, the iPhone 5s uses the A7 CPU, the iPhone 6 and 6 Plus use the A8 CPU, so playback will be in software. The iPhone 6S models use the A9 CPU, while the iPhone 7 uses the A10 Fusion, which presumably will also include HEVC hardware-based decode. So, sending HEVC streams to the iPhone 5s, iPhone 6, and iPhone 6 Plus will likely significantly reduce their battery life; for newer devices, HEVC should have minimal effect.
Though not listed on the slide, at 4:15 in the presentation, Pantos states that the Apple TV will also receive a software update that enables HEVC playback, which you can confirm at Apple's tvOS Guide.
A different presentation, Session 515, the "HLS Authoring Update," noted that all software and hardware decoders support up to Main 10 Profile, Level 5.0, and the High Tier. This presentation also discusses many additional changes to the HLS specification and associated tools, and is worthwhile for anyone creating HLS content. Yet another presentation, "Introducing HEIF and HEVC" (Session 503), presented by Gavin Thomson and Athar Shah, details that beyond HLS, Apple's HEVC decoders can play HEVC encoded video in either a QuickTime or MP4 wrapper.
Other HEVC Implementation Details
Figure 3 covers several additional implementation details. First, and most important, HEVC must be packaged in the fragmented MP4 format Apple adopted when it added support in HLS for the Common Media Application Format (CMAF) at last year's WWDC. Second, Apple will continue to use CBC (cipher block chaining) encryption, which has been adopted by Google in Widevine, but not (so far) by Microsoft in PlayReady. So, while you can reuse your encoded fMP4 fragments for DASH and HLS, you'll still need two delivery payloads for encrypted content—one for CBC, and the other for Counter Mode (CTR) used in PlayReady.
Figure 3. Other implementation details
The rest of the slide deals with notification requirements for HEVC encoded content in the HLS manifest files, which is straightforward.
Mixing HEVC and H.264 in a Master Playlist
Figure 4 details how to combine HEVC and H.264 in a single HLS presentation defined by a master playlist files. As you can see, you can mix the two codecs in a single playlist just as you could include video encoded using different H.264 profiles. The key requirement is that you designate the codec as discussed above (and as you had to do when mixing H.264 profiles).
Essentially, this means that the player will have to seamlessly switch from playing H.264 encoded video to HEVC encoded video; it will be interesting to see how smoothly this works, particularly on devices and computers using software decode.
Figure 4. Mixing HEVC and H.264 in a Master Playlist
Note that unlike HEVC, which must be fMP4, H.264 HLS content can be formatted as either fragmented MP4 files or MPEG-2 transport streams. This means that you don't have to transmux your streams to mix them with HEVC, a nice convenience factor. The last line details that Apple updated their HLS Authoring Guidelines for HEVC (the official document is called the HLS Authoring Specification for Apple Devices). Let's see what the updated guidelines look like.
40 Percent? Who Said 40 Percent?
Figure 5 shows the new encoding ladder, and it's interesting that the highest bitrate saving is 29% and never gets to the 40% number that Apple bounced around in multiple sessions. It's also interesting that Apple hasn't yet extended its guidelines beyond 1080p, despite the iPhone 7 and many other devices supporting 4K playback.
Figure 5. The updated encoding ladder from the Apple HLS Authoring Specification
No matter. You should be able to achieve 35-50% savings at 1080p resolution, and 30-40% savings at 720p. Beyond those resolutions, H.264 has never really been a good option, so comparisons are not useful.
What About Subtitles?
Apple also debuted support for IMSC1, which is a WC3 recommendation for TTML captions that are optimized for streaming delivery, and is the baseline subtitle format for CMAF. It's also mandatory for the Interoperable Master Format specification to which content publishers like Netflix are committed.
According to the Apple documentation, the IMSC1 captions are carried as XML text inside fMP4 segments, and identified in the Master Playlist file. Note that IMSC1 captures are in addition to the WebVTT captions already supported by HLS, not instead of.
What the Heck is HEIF?
In the aforementioned presentation entitled "Introducing HEIF and HEVC," Apple also debuted an image format called the High Efficiency Image File Format, or HEIF, which is an image standard developed by the Moving Picture Experts Group (MPEG) for storage and sharing of images and image sequences. As you can see in the Figure 6, the HEIF format as used by Apple uses HEVC compression, which the presenters state provides up to 2X better compression than JPEG.
Figure 6. Details about the new HEIF format
Support for HEIF image decode looks a lot like HEVC decode; that is, in hardware for devices with the A9 chip and (presumably) higher, and software for all others. This raises two concerns. First, that HEIF decoders will be subject to royalties, though presumably, Apple has taken care of that for its own updates. Second, that HEIF images will be a proprietary format that you can't easily share or utilize. It may be that you'll first have to decode the image into JPEG with an Apple supplied or other decoder to use it outside the Apple ecosystem.
About those Royalties
According to multiple reports, Apple currently has about $250 billion in cash, so certainly it can afford any royalties associated with HEVC. Interestingly, Apple is listed as a licensee (and licensor) of the MPEG LA pool, but not as a licensee in the HEVC Advance pool. I checked with HEVC Advance, which confirmed that Apple was not currently a licensee, though they expect this to change when Apple actually issues the updates. The newest pool, from Velos Media, hasn't yet set terms.
What am I Hearing from the Industry?
As you would expect, the industry is very positive about the Apple move. Bitmovin already supports HEVC in HLS, and JW Player plans to once the updates are available and demand for the format begins.
The one publisher I spoke with expressed concern about the costs of adding a completely new set of encodes to its CDN footprint, and shared it they probably would wait until iOS devices with hardware decode comprised a high percentage of all supported iOS devices.
Related Articles
Adding HEVC to your HLS streams is looking like a no brainer, but if you decide to do so, you may not want to take Apple's HEVC encoding recommendations verbatim. You'll deliver noticeably higher quality video if you follow the advice detailed below.
21 Nov 2017
If you're adding HEVC to your HLS video, you're likely concerned about the playback frame rate and battery live on the iPhones, iPads, and computers to which you're delivering. We tested a range of devices, and found the CPU impact to be negligible on most of them.
15 Nov 2017
Why the move to HDR matters for more than just 4K—and why OTT services and devices are going to lead the way
09 Aug 2017
Hardware acceleration and field programmable gate arrays may be the answer to the rising costs of encoding for multiple codecs including H.264, H.265, VP9, and soon AV1
03 Aug 2017
Apple adding HEVC support to HLS this fall is a huge step forward for the codec. An announcement from MainConcept makes file creation easier.
24 Jul 2017
This isn't exactly the summer reality show many are looking for. The way HEVC patent pools are growing is likely to make AV1 an especially popular option.
27 Jun 2017
Frost & Sullivan analyst Avni Rambhia assesses the key factors content owners face today in determining whether to stick with AVC or move to HEVC.
24 Nov 2016
Frost & Sullivan analyst Avni Rambhia identifies the key inhibiting factors that are impeding the widespread migration from AVC to HEVC.
10 Nov 2016
Frost & Sullivan analyst Avni Rambhia explores the "three 'Rs'" of HEVC adoption: Revenue, Resolution, and thRoughput.
07 Nov 2016
Speedy adoption of HEVC has been delayed by squabbles over licensing. Here, the developers of x265 propose steps to end the gridlock and move forward.
31 Aug 2016
Should you be delivering HEVC? It depends on what you're delivering and who you're trying to reach. Frost & Sullivan analyst Avni Rambhia breaks down the key issues of sticking with AVC vs. migrating to HEVC in this clip from Streaming Media East 2016.
04 Jul 2016
While the x265 codec is making big strides in quality and file size, the major browsers are in no rush to support H.265. With interest in 4K growing, maybe it's up to Flash to save the day.
27 Apr 2016
Companies and Suppliers Mentioned