How to Monitor and Troubleshoot Your Live Streaming Workflow
What are some of the typical technical issues with workflows for live streaming at scale, and how can they best be handled? Eric Schumacher-Rasmussen, Chair, Streaming Media Conferences, and CMO, id3as, discusses this topic in detail with a panel of experts from Paramount, Amagi, TAG VS, and Nomad Technologies.
Schumacher-Rasmussen begins by asking Peter Wharton, Chief Strategy & Cloud Officer, TAG Video Systems, “Once you've optimized each stage or element of the workflow, how do you ensure the entire holistic package is working as it should be and stays working as it should be?”
Wharton emphasizes that these systems need a certain amount of sophistication to achieve the best technical efficiency and cost-effectiveness. “We are looking for a level of intelligence in these systems,” he says. “We're looking at something that monitors all these points and actually could go deep enough to look at content in those points and not just say, ‘there's data flowing here.’ Because a lot of the problems you might have would be misrouted signals where it's the wrong thing airing in the wrong place because you're building these workflows where you might be routing away for commercial breaks and not coming back or breakaway and not getting the right returns, so, it is complex, but there are the tools out there.”
Corey Smith, Sr. Director, Advanced Production Technology, CBS Sports Digital, Paramount, highlights the importance of partners needing to be just as invested in a seamless live stream as the organization running the workflow. “If they're like, ‘Hey, go ahead and don't worry about this. Just send us your stream, we'll go ahead and distribute it’…that would be my first red flag. If you're not invested in making sure you're sitting down at the table with me, and we're walking hand-in-hand in this, and you have a Tier 1 event you're trying to pull off, and there's an incredible market and PR value to you as an organization…and it appears that nobody cares on the other side, that is a problem.”
Wharton agrees and says, “When you start working with partners and other companies, you need to get those analytics back into your system and be able to watch them too. And sometimes you even put your own monitoring at the edge to provide yourself the trust that they're actually doing what you think they're doing.”
“Trust and verify,” Smith says.
Adam Miller, CEO, Nomad Technologies, notes that it is essential to plan for failure to some extent to ensure a successful outcome. “I think most people don't follow through the scenarios that could actually happen. Like an encoder stops or an internet stops, or the camera stops. They just assume that their backup plan will just be figured out on the fly. In reality, you kind of have to try it. And most people, they're frantic before these events. So the last thing they want to do is unplug things and see if it really works. Most people are afraid to do that, but you have to stress test those systems. Turn on the monitoring, ‘Hey, look, the flag went off! Look, the monitoring said something's wrong. What do we do now?’ And go through that stuff ahead of time.”
Smith says, “Don't be afraid of unplugging. If your encoders are dual path on an ethernet line, pull one of the lines in a test scenario. Your successful failures should happen in a lab. They shouldn't happen live to your customers.”
Schumacher-Rasmussen says, “It would seem like monitoring is an element of the workflow that would lend itself naturally to AI. Any thoughts on AI? Seems like Brian, you might have had some.”
Brian Ring, Senior Director, News & Sports Solutions, Amagi, says, “I guess two things. So, on monitoring things, it's true. We can have automated monitoring, and we can trigger alarms and events and things like that. But for Tier 1, many customers pay us to have eyes on the glass. I think the other point I was going make around AI, it's not quite this kind of reliability thing, it was the broader definition of quality. I grew up in this business. I remember having that first insight. These guys are talking about quality, they were talking about encoding the picture. I was like, what? Even from my early times, I was like, quality in TV means much more than that, right? And that's true today.”
Ring points out the broader benefits of AI, especially regarding FAST linear. “We want to make sure the ad slots are filled, we want to hit all the seamless changes and so forth and so on,” he says. “And then the redundancies to all the platforms. There are just so many different things. One of those things is putting the ad breaks in the right spot so you can use sort of narrow AI for those purposes. There's lots of great stuff in AI that has been happening and will continue to happen, and you can kind of QA those things [like] typos in the subtitles. And so there's lots of great level narrow use cases [to] make the better video experience.”
Learn more about live streaming workflows at Streaming Media East 2023.
Related Articles
Most of the essential elements of streaming workflows and architecture are moving toward the cloud, but the fundamental question driving streaming workflow development should always be not how do we migrate this or that process to the cloud, but what will serve the production and the viewer best, according to CBS Sports Senior Director, Advanced Production Technology Corey Smith, who breaks down these questions in this discussion with Eyevinn Technology's Magnus Svensson from Streaming Media Connect 2024.
12 Apr 2024
What is the essential monitoring checklist for maintaining efficiency, resiliency, and performance for large-scale low-latency streaming using cloud-based workflows? Dolby.io Director of Product Strategy and 2G Digital Post Optimization's Allan McLennan discuss the key components and tools for an effective live stream monitoring strategy in this clip from Streaming Media Connect.
14 Feb 2024
Delivering reliable streams is both more challenging and more critical when streaming at scale. So how do you develop workflows to ensure your large-scale streams won't fail? Experts from Dolby.io, TAG VS, Paramount, and Nomad Technologies weigh in on network stress-testing, monitoring, and more best practices at Streaming Media West 2022.
24 Feb 2023
videoRx's Robert Reinhardt guides viewers through the key enabling technologies of low-latency streaming, including server ingest and client delivery protocols like WebRTC, NDI, RTMP, and HLS in this presentation from Streaming Media West 2022.
20 Feb 2023
How do you meet the challenges of monitoring stream performance and quality for dozens or thousands of live streams at once? Experts from Paramount, Dolby, TAG Video Systems, and Nomad Technologies discuss multistream monitoring tech and strategies in this panel from Streaming Media West 2022.
15 Feb 2023
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi's Brian Ring, Dolby,io's David Hassoun, Nomad Technologies' Adam Miller, Paramount's Corey Smith, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from Streaming Media West 2022.
30 Jan 2023
What are the most dynamic approaches for localizing live event streams? Dan Turow of Evertz and Marisa Elizondo of fuboTV talk about the ways their organizations are working to best integrate local and user-preference-based content and experiences into live event streams.
15 Nov 2022
What are some of the biggest adtech challenges for live streaming at scale? Jarred Wilichinsky of Paramount talks about the ways his team works to mitigate technical issues, such as minimizing latency, load testing, and correcting audio levels, along with ensuring that the ads themselves meet acceptable legal standards and practices.
18 Oct 2022
Companies and Suppliers Mentioned