How to Deliver Resilient Streams at Scale
Guaranteeing a satisfying end user experience, whether you’re delivering content live or VOD, requires resiliency, ensuring that the stream doesn’t break down regardless of the scale, bursts, or other fluctuations in delivery demands. The challenges are different for live and VOD, with live proving significantly more challenging in most instances.
Mark de Jong, Chairman, CDN Alliance, discusses these key challenges and how to address them with Michael Demb, VP, Product Strategy, TAG VS, and Bob Hannent, Principal Architect: Technology Operations, DAZN, in this clip from Streaming Media Connect 2023.
De Jong asks Demb to discuss some of the most important elements of resilient live streams beyond taking a multi-CDN approach.
Demb emphasizes that VOD streaming is not at all like live streaming because when streaming live, there is very little room for mistakes. “If I have to choose my top three differentiators from a pure distribution perspective,” he says, “one will be the latency. In VOD, there is no such thing as latency. [Then] scalability, and of course quality assurance. How do you make sure that you're delivering the right quality of video? So to ensure that live streaming works smoothly, real-time monitoring is key. You need to know exactly what is happening with the streams at every moment. If something starts going wrong, you need to catch that fast and fix it even faster. That's critical. So in terms of scalability, live events, especially sports…they can be huge and unpredictable. Spikes in viewership can start from 10 viewers and grow to 10,000 and maybe even 10 million. And you don't know how many users you're going to have. So that means you need a really flexible setup, real-time performance measurements, and that can quickly identify those bottlenecks and make the switching decisions.”
Demb says that there can be embarrassing and unsatisfying user experiences if there is too much latency during live streams, which can result in occurrences such as the final scores of a match appearing on the phones of users before they actually see the winning goal. “So our customers invest a lot of effort in reducing their streaming latency and ensuring it stays low,” he says. “And TAG is helping them to measure those latencies and make those decisions. TAG just recently rejoined the Streaming Video Technology Alliance (SVTA) to help identify the best methods of measuring latency in live streaming. SVTA has published a number of papers on the subject that were part of that work in the past. And soon we'll have a cloud-based test environment to test real live use cases for low latency streaming and measurement.”
Regarding content quality, Demb says, “So you need to make sure that this video stream gets to your audience at the highest quality possible without any delays or buffering. And that means, first of all, choosing the right format codec to reduce the bitrate as much as possible without losing quality. And for that, you need the monitoring solution in place as well. So, having the right tools with very practical and actionable KPIs is the key. For example, quality degrades. You want a monitoring tool that can tell you exactly what happened, point you in the right direction, and also help understand the root cause of that, not just tell you the quality dropped from five to three. That's not very actionable. So you don't want some actionable KPIs and actionable alarms that will point you to where the problem is.”
Hannent agrees with these points and notes that sports fans are far less forgiving of live-streaming errors than VOD viewers. “There is nothing worse than the anger of a sports fan who isn't able to watch their content,” he says. “Maybe if you fail to publish Game of Thrones properly, there will be an issue, an equivalent perhaps, of passion. I'm not convinced. I think 99% of the time, if you can't watch the piece of VOD you want, you'll find something else. If you have decided the appointment to view is that game, you've got to deliver it. You can get the CDN infrastructure lined up and you can get the video chain sorted, but stampedes on your APIs, stampedes of trying to get CDN links, stampedes of trying to get DRM licenses. If you have got people all clicking on the tile at the same time, or worse still, you have a glitch in the CDN, and people rotate, or your system rotates synchronously…I've seen 40,000 requests in two seconds to the backend and going from a baseline of less than 500. So that kind of ramp up.”
Hannent mentions that autoscaling is not a panacea. “Autoscaling does not work most of the time, and that's a really generic statement, a broad statement, but you have to be really careful about autoscaling,” he says. “But we have multiple regions located around the world, so that load can shift if something happens.”
He speaks of “headends” and how DAZN effectively uses them. “We have two main headends,” he says. “Two secondary and a disaster recovery headend. By headend, I mean the encoder and packager and all that stuff. So that is a very high level of resilience, and all that needs to be monitored as well. Are you monitoring the level to the level that you need on that? And there's another element: Do you want them to be coupled or decoupled? How do you want to ensure that impact on one system doesn't have an impact on another?”
Watch full sessions from Streaming Media Connect November 2023. We'll be back in person for Streaming Media NYC on May 20-22, 2024. More details here.
Related Articles
For high-stakes, large-scale streams, redundancy is critical for ensuring smooth and reliable delivery. This means careful monitoring, sound decision-making, and seamless network switching-also known as hitless failover-and making sure streams are precisely synced so the switch is invisible to the end user. In this clip from Streaming Media Connect 2023, DAZN's Bob Hannent, TAG Video Systems' Michael Demb, and CDN Alliance chair Mark de Jong discuss the challenges of ensuring stream resiliency and strategies for maintaining it.
02 Feb 2024
Taking a multi-CDN approach would seem to be a no-brainer for delivering large-scale streams to global audiences and maximizing uptime in the face of bursts, unexpected regional demand, and other impediments to a smoothly delivered high-stakes stream. But DAZN's Bob Hannent says it's not always so, and a multi-CDN approach can actually introduce inefficiencies, in this discussion with CDN Alliance Chairman Mark de Jong at Streaming Media Connect 2023.
03 Jan 2024
Jef Kethley of LiveSports talks about how his organization develops solid resiliency models in varying production environments through the use of multiple data centers and cost-effective cloud production.
06 Sep 2022
AWS Solutions Marketing Manager Kiran Patel explains the benefits of a cloud native architecture for enhancing stream redundany and resiliency in this clip from Streaming Media West 2019.
03 Apr 2020
Companies and Suppliers Mentioned