The State of Video Codecs 2023
Welcome to your codec update for 2023. I’m here to help you decide whether it’s time to go all in on AV1, VVC, LCEVC, or EVC or whether it’s better to stick with H.264, VP9, and HEVC. As an overview, I’ll focus on four aspects of codec utility:
- Can you use the codec? You can’t use the codec if it doesn’t play on your target platforms. So, we’ll review this data first.
- Should you use the codec? You shouldn’t use the codec unless it delivers substantial benefits like bandwidth savings over your existing codec. So, we’ll examine this next.
- How should you use the codec? If a codec plays on your target platforms and delivers substantial benefits, it still may not be usable for your selected application, whether it’s low latency, live origination and transcoding, or HDR. So, we’ll examine each codec’s suitability for these distinct roles.
- What does it cost to use the codec? We’ll also look at royalties.
Along the way, I’ll include 2022–2023 updates to identify significant changes that have occurred since last year.
Can You Use the Codec?
To mangle a truism from the sports world, the best codec ability is playability. Table 1 (below) highlights several aspects of this.
Table 1. Quality and encoding time with x265 as the baseline
Column 1 shows the percentage of available platforms that support the codec as reported by Can I use, which “provides up-to-date browser support tables for support of front-end web technologies on desktop and mobile web browsers.” While it’s possible to work around this native support, most smaller publishers rightfully take the easy route and publish where native support is available.
By far, the most significant move in 2022 was Chrome initiating support for HEVC. However, there are issues with Chrome’s HEVC playback, which works only with devices with HEVC hardware support and has limited DRM support. Still, for publishers who don’t use DRM, playback in Chrome pushes HEVC support to 84.57%, about 12% more than AV1.
This advantage over AV1, by the way, is largely thanks to Alliance for Open Media (AOMedia) founding member Apple, which still doesn’t support AV1 on any platforms. There were rumors in 2022 that Apple was initiating AV1 support. However, as reported in the excellent Bitmovin post, The State of AV1 Playback Support: 2022, “while their recently released M2 processor does not support AV1 hardware decoding, it will hopefully offer some form of software accelerated decoding." So, the Can I use numbers appear valid.
A couple of points about the Can I use data: VVC isn’t listed yet, which means little or no platform support. Given the relatively slow frame rate on my i7-based workstation, it’s unlikely that software-only playback will become a reality for VVC in the near term. As a counterpoint, however, several VVC patent owners have deployed VVC on mobile phones and report much higher frame rates.
Even assuming the viability of VVC software playback performance, the lack of native browser and OS support would still create licensing issues that most publishers couldn’t work around. That’s because the VVC patent pools are targeted toward encoder and decoder vendors, not content owners. So, until the decoder vendors support VVC, you can’t ship content to those devices.
In contrast, LCEVC appears more than capable of software-only playback, and its royalty policy allows publishers to license LCEVC and deploy the appropriate decoders on all platforms. As such, while the lack of platform support reported by Can I use numbers is a likely bar for VVC/EVC deployments, it’s not for LCEVC.
Why Hardware Decoding Matters
Now, let’s look at mobile hardware support. Several AOMedia members not named Apple have deployed AV1 for software playback on mobile devices. For example, Meta started distributing AV1-encoded video to iOS and Android phones but had to implement a testing protocol to identify which Android phones could play the streams smoothly.
In an article titled Enhancing SVT-AV1 With LCEVC to Improve Quality-Cycles Trade-Offs and Enhance the Sustainability of VOD Transcoding, the authors show why the testing protocol was needed. Specifically, testing on a range of Android devices, the authors demonstrate that a high percentage of frames dropped with software-only AV1 playback with the admittedly aggressive 60 fps El Fuente sequence (Figure 1, below). Using LCEVC with AV1 as a base layer provided much smoother playback. Other data in the article show how AV1 playback drained battery life much more quickly than LCEVC/AV1 playback and, presumably, hardware-based playback.
Figure 1. Playback performance of AV1 and LVEVC/AV1 on a range of Android devices
This article was obviously designed to demonstrate that LCEVC with AV1 as a base layer is a better choice than AV1, and it appears to do so. However, in early 2023, ScientiaMobile released a report showing that 86.6% of mobile devices supported hardware-based HEVC playback, while only 2.52% supported hardware-based AV1 (Figure 2, below).
Figure 2. Scientamobile data showing that HEVC hardware playback is available in 86.6% of mobile devices compared to 2.52% for AV1
This data presents publishers with two alternatives as it relates to mobile. The first is to implement AV1, which comes with three caveats:
- It requires a testing protocol to identify
which devices can play AV1 smoothly. - It will reduce battery life over hardware playback.
- It requires an iOS app.
The other option is to implement HEVC, which has three advantages over AV1:
- It will work on all devices with hardware playback.
- It will preserve battery life.
- It will play in the browser on iOS and Android
or, of course, in an app on either mobile OS.
What about the HEVC content royalty bogeyman? I’ll cover that in the final section.
Deploying to Smart TVs
H.264, HEVC, and VP9 enjoy near-universal support for living room devices. As I reported in The State of Video Codecs 2022, AV1 support for these devices was sufficient for Netflix to start distributing AV1 to the living room, although this was without HDR support.
Again, LCEVC could be deployed on some living room devices as a pure software upgrade, but VVC and EVC will require hardware support, which will take 3–5 years to develop into a market worth addressing. In this regard, VVC is showing much more progress than EVC, with a VVC-enabled TV announced at IBC and VVC added to LG’s webOS Smart TV spec for 8K Ultra HD models (in addition to AV1 and HEVC). As I reported in last year’s “The State of Video Codecs,” VVC appears to be “codec next” for smart TVs due to its inclusion in the DVB core specification as well as being newly selected as the “main” codec for Brazil’s TV 3.0. Note that TV 3.0 also includes LCEVC as the video enhancement codec.
The final two columns in Table 1 summarize data from the Bitmovin Video Developer Report. The penultimate column shows deployments in 2022 for VOD/live, while the final column shows planned deployments for 2023 for VOD/live. Survey participants are from a broad range of companies in the streaming media ecosystem, not just publishers, which explains how codecs like VVC and EVC are being deployed long before there’s a critical mass of available players. Still, it’s useful to see that even codecs like EVC, which appear to have little momentum, are on the radar of some ecosystem providers.
Bandwidth Savings
Next, let’s look at the question of whether you should use the codec and if it delivers sufficient bandwidth savings to justify the deployment cost. Table 2 (below) shows comparative quality figures for the selected codecs with three sources of information.
Table 2. Codec suitability for different codec tasks
The first column is from Moscow State University’s (MSU) MSU Video Codecs Comparison 2021 report, which probably will be updated by the time you read this article. Note that MSU reviews multiple codecs from multiple sources, including sources that aren’t available commercially and some that can only be accessed via cloud encoding facilities. This makes the report a great snapshot of overall quality, but not necessarily quality that you can actually produce for yourself. The numbers shown in Table 2 reflect quality comparisons using the Video Multimethod Assessment Fusion (VMAF) metric.
The Streaming Media numbers are from an article I did in December 2021 with codec testing comparisons, which also used VMAF. I used FFmpeg codecs to produce results for H.264 (x264), HEVC (x265), and AV1 (libaom-AV1), so these results should be achievable by most producers. Sources for the other codecs are listed in that article. I mixed in VP9 results from Bitmovin’s HEVC vs. VP9: Modern Codecs Comparison, because MSU tested SVT-VP9 rather than FFmpeg’s libvp9, with terrible results, and because I didn’t test VP9 in the Streaming Media article.
All data suggests the following:
- Continuing to use H.264 costs you at least 50% more from a top-line bandwidth perspective.
- VP9 and HEVC are about even in terms of quality.
- AV1 is about 35%–38% more efficient than HEVC.
- VVC is 3%–5% more efficient than AV1.
- LCEVC quality will vary with the base layer.
The Streaming Media tests show that LCEVC with x265 as a base layer was about 22% more efficient than x265. - The EVC Baseline codec, designed to be a royalty-free option for H.264, proved to be 45% more efficient than H.264.
- The EVC Main codec, designed to compete with HEVC with a lower royalty, performed very well, proving to be 42% more efficient than x265, although the encoding speed was very slow.
Note that these are top-line savings and that your actual bandwidth savings will depend on which rungs of your encoding ladder you predominantly distribute. For example, if you’re an IPTV (Internet Protocol Television) service that delivers the top rung of your encoding ladder to 99% of viewers, switching from HEVC to AV1 should deliver close to the indicated 35%–38%. However, if you distribute mostly middle rungs of your encoding ladder to mobile viewers, switching to a more efficient codec will swap a 2Mbps HEVC stream for a 2Mbps AV1 stream, resulting in a higher-quality video file but no bandwidth savings.
How Should You Use the Codec?
Just because a codec delivers substantial savings in VOD-oriented trials doesn’t make it ideal for other encoding tasks. Table 3 (below) contains several common codec use cases and rates each codec for suitability.
Table 3. Royalty policies for each codec
When I refer to “live origination,” I mean professional, primarily hardware-based encoders for broadcasters, not single-instance GPU-based encoders attempting to send a single stream to Twitch. H.264 enjoys the most hardware support, followed by HEVC and AV1, which now boasts several live encoders from vendors like Intel, NETINT, and NVIDIA. Very few vendors support VP9 origination or transcode.
LCEVC rates highly because V-Nova, its primary developer, has long shipped contribution encoders and has garnered hardware support from multiple encoding vendors, including AMD, Intel, Harmonic, and NETINT. In contrast, VVC has very little support for live origination, and EVC has none.
The live transcode column is similar to live origination. H264 and HEVC have the most support, with AV1 and LCEVC next and VCC and EVC nowhere close.
For low latency, I considered both technological availability and actual existing deployments. H.264 is the dominant low-latency codec used for adaptive streaming, while H.264 and VP9 are widely used for WebRTC. HEVC and AV1 are certainly feasible for low-latency applications with broad hardware support but aren’t widely used at this point. Low latency is feasible for LCEVC, but not widely used, with neither VVC nor EVC in the conversation.
The 4K column reflects compression efficiency and should be straightforward. Except for H.264, which is downgraded because of compression efficiency, HDR focuses primarily on support for dynamic metadata standards and the maturity and extent of available playback devices. By these measures, HEVC is by far the best option for HDR, with support in Dolby Vision, HDR10+, and HLG and near-ubiquitous playback support.
Few producers other than YouTube use VP9 for HDR, but that level of usage pushes VP9 into the Very Good range. Beyond VP9, understand that any 10-bit codec that supports ITU-T T.35 metadata can support HDR10 and HDR10+, although Dolby Vision support is limited to HEVC and H.264. So, while AV1 technically supports HDR10+, when Netflix started distributing AV1 video to smart TVs, it was SDR only. VVC, LCEVC, and even EVC might all technically support HDR10+, but there are few smart TVs that support these codecs.
What’s It Going to Cost to Use the Codec?
The short answer to the cost question is that if you’re a video publisher distributing free content, the only codec you’ll pay a license for is LCEVC (and that’s a good thing). While H.264 imposes minor charges for pay-per-view, subscription, or television broadcasting, and Access Advance charges low royalties on VVC and HEVC-encoded video distributed on physical media, none of the remaining patent pools charge for free internet video. This includes HEVC (see patent attorney Robert J.L. Moore’s comments).
Table 4 (below) provides a high-level overview of how and what the different pools charge for codec usage and deployment. Nothing’s changed with H.264, MPEG-LA charges for each encoder/decoder over 100,000 units, and there are minor fees for subscription (again over 100,000) or pay-per-view content and free television distribution, but not for free internet content.
Sisvel has patent pools for VP9 and AV1 that charge for playback only on consumer devices, with no royalties on encoding or content. Google appears to dispute the VP9 pool, stating in WebM’s FAQ page, “While open source, royalty free and/or no charge technologies are widely offered, there is no way to prevent third parties from demanding licensing fees on these types of technologies.”
AOMedia issued a similarly vague comment, in which it acknowledged being aware of the Sisvel pool. It also discussed its goals and procedures for AV1, but didn’t state that Sisvel’s claims were invalid. Neither Google nor AOMedia indemnify any user if Sisvel’s claims are proven valid, although AOMedia did establish a patent defense fund.
Regarding HEVC, 2022 saw the closing of the Velos Media pool, leaving the MPEG-LA and Access Advance pools. Significantly, Velos Media was the only pool that left the door open to charging for free content, so its shuttering should remove the last of these concerns. That said, Velos Media continues to license some 400 patents directly, with minimal details about what’s royalty-bearing on its website, and its owner, Marconi, has advocated for content royalties in the past. When asked, neither Velos Media nor Marconi would provide details about the Velos Media program and wouldn’t indicate whether they planned to charge royalties on content. Of the two remaining pools, MPEG-LA charges for encoders and decoders over 100,000 units but not content. Access Advance charges for encoders and decoders, with minor charges for HEVC-encoded video shipped on disks or other media.
MPEG LA and Access Advance both have VVC pools. MPEG LA again charges for hardware and software over 100,000 units, but not content. Access Advance again charges for encoders, decoders, and VVC-encoded video shipped on physical media. Access Advance also offers a Multi-Codec Bridging Agreement that reduces the combined royalty rate for licensees of both pools.
Where other pools license encoders and decoders, V-Nova, as LCEVC’s primary developer, charges a low per-user fee on the streaming service actually using the codec and is free for encoder and decoder vendors. V-Nova’s royalty involves different prices for different types of entities, like subscription and advertising-based services and TV Everywhere, all capped at $3.7 million per year.
As mentioned, if you’re a publisher considering LCEVC, this structure is a good thing; for one low payment, you can license every possible encoder or decoder. In contrast, those seeking to use other codecs would have to pay a per-unit royalty for each decoder, which for large services would be cost-prohibitive.
As of this mid-February writing, the patent owners that own the intellectual property incorporated into EVC—Samsung, Huawei, and Qualcomm—haven’t announced royalty terms.
Related Articles
It's been 3-plus years since the MPEG codec explosion that brought us VVC, LCEVC, and EVC. Rather than breathlessly trumpet every single-digit quality improvement or design win, I'll quickly get you up-to-speed on quality, playability, and usage of the most commonly utilized video codecs and then explore new codec-related advancements in business and technology.
04 Apr 2024
The case for moving from ubiquitous older video codecs like H.264 to newer codecs like AV1 or HEVC (H.265) is typically expressed in terms of encoding efficiency that translates to bandwidth and cost savings. For major content companies like Warner Bros. Discovery that have adopted H.265, how much has their experience borne out that hypothesis?
08 Nov 2023
Following on the news that Via Licensing has acquired codec patent pool administrators MPEG LA to form Via LA as a "unified entity," Via LA President Heath Hoglund discusses how the pool will operate, as well as resulting synergies including streamlined IP distribution and economies of scale in this interview with Streaming Learning Center's Jan Ozer.
08 May 2023
One of the premier pre-conference events at the upcoming Streaming Media East conference explores the readiness of the Versatile Video Codec (VVC) for deployment and implementation. Presented on Wednesday, May 17, 2023, from 9:00 AM to 12:00 PM at Boston's Westin Copley Place, the session includes multiple speakers, many from companies already shipping VVC-related products or services.
02 May 2023
Companies and Suppliers Mentioned