Location, Location, Location: Cloud vs. On-Prem vs. Hybrid Streaming Workflows
A basic metaphor I've used frequently to describe to my non-technology-steeped friends what I do for a living (or technology as a whole, for that matter) is that of a pendulum. At one extreme, there's a particular worldview ("VHS is better!") that's juxtaposed against an equally hardline opposing worldview ("Betamax for the win!"). As the two worldviews compete for superiority (or market dominance), the two points to which the pendulum swings color how one sees—or even interacts with—a particular technology's place in the world. Marketing these two opposing views will, naturally, lean a prospective user toward one of the two extremes to the exclusion of any value the other holds.
Yet, as we all know, the reality of the "better" technology is often somewhere toward the middle of that pendulum's arc. VHS might not have as good quality as Betamax, I hear my mid-1980s self saying, but it's sure cheaper to buy a VHS copy of a particular movie from Walmart than it is to buy the better-quality Betamax copy from BMG Record Club's monthly "pick hits," which isn't carried in any local store.
So what does the pendulum metaphor have to do with streaming? Quite a bit, actually. Those who have been around the industry for any length of time have been caught up in at least one of the choose-a-camp eras, whether it was the early days of RealPlayer versus QuickTime or, more recently, the HLS versus DASH discussions (which also stand in as a closed system versus open source placeholder for the larger debate around free and open source software, or FOSS).
This brings us around to another pendulum swing: cloud-centric versus on-prem production workflows. Is there a middle ground? That question was front and center for a panel I hosted at Streaming Media Connect in February 2022. The panelists' experience includes traditional broadcast, episodic, and motion picture workflows, and each has a preferred approach. But each has also, at one time in their career, acknowledged and embraced the middle ground of hybrid production workflows.
The term hybrid comes with its own set of distinctions, although they can be a bit muddied. Rather than trying to generate one conclusive definition for hybrid workflows, I'm going to look at four key areas: multi-cloud solutions, reasons to consider the cloud, security, and challenges with cloud-based solutions.
Multi-Cloud or Multi-Region? Varying Cloud Approaches
Bill Thompson, director of Signiant's software-as-a-service (SaaS) platform, kicked off the session by talking about the need to deliver SaaS solutions that are purpose-built for media delivery as well as tweaked for use by multiple cloud providers. "A lot of what you do in one cloud is slightly different from doing it in another cloud," he said. "So the development work, I wouldn't say it's times two, but they're substantially different."
One challenge that Thompson pointed out is the need to test two different variations of services, one for each cloud. That's what a company like Signiant does for its SaaS solutions, but Thompson also noted that there are other ways to approach redundancy in cloud solutions. "I think what most people do is within a single cloud using multi-region implementations," he said. "So that if U.S. East goes down in Amazon Web Services [AWS], as it did in early December, our back end … fails over to U.S. West. AWS doesn't have a lot of outages, so they're pretty rare, but it would be even rarer if the East Coast and the West Coast go down at the same time."
Renard T. Jenkins, SVP of production integration and creative technology services at Warner Bros. Discovery, agreed. "Each one of them [cloud providers] are proprietary," he said. "They are that way for business reasons. I think where we're getting to right now, and I hope that we'll get there, is the ability for us to have a transactional layer and that transactional layer between clouds that allows for standardized messaging where you can actually send and communicate between clouds even though you're not expecting them to act in the same way. That is where I think we need to really put our focus and put our effort and our time."
Space, Speed, and Scale
One issue, especially in corporate environments, is a lack of space to add in new media workflows alongside traditional media, whether that traditional media is print (such as newspaper press rooms) or broadcast (such as a typical multi-person control room at major news production facilities). In keeping with the pendulum metaphor, I asked the panelists how production workflows balance those traditional IT needs (routing everything through a data center that the company owns) with relying on third-party cloud solutions.
"I think that answer is kind of on a per-project basis," said Ben Ratner, live operations manager at CNN, part of Warner Bros. Discovery "Some things are just going to work better in the cloud, and some things aren't, especially older things that have all of the inertia and all of the expertise of on-prem or hybrid." Ratner went on to say that his workspace is "literally in a reclaimed phone booth" at a CNN corporate location that now doubles as his control room. "You just couldn't do it if I needed to have three giant racks in here."
On the delivery end of things, though, there's another concern: the question of scale and how that affects the pendulum swinging from on-prem to cloud to somewhere in the middle.
On the distribution side, of course, there is scaling the delivery to the number of clients who are watching streams, said Loke Dupont, solution architect at TV 2 in Denmark. "We also get a lot of content that's not necessarily produced in-house. Oftentimes, we will get licensed catalogs of content and basically have to ingest 10,000 hours of content that nobody planned for … until somebody decided to license that content." Dupont noted that this frequent problem often directly impacts whether a cloud or on-prem solution is chosen for the time being. "Sometimes, if you need to do that in a very short time frame, it can be challenging with on-prem gear," he said, "since you have to source hardware and get it installed. Especially at the current time we are living in, sourcing hardware and getting it there on time is not necessarily an easy job."
Hybrid workflows come with their own challenges. "I think the industry as a whole has gotten used to the hybrid model," said Jenkins, "as a way to sort of bridge what you're doing today with what you'd like to do tomorrow. The issue that a lot of people have is that they are used to having things near them. They're used to having control over everything within their environment."
Security Concerns
Jenkins noted that, especially if they follow industry-standard practices around security, companies with on-prem data centers "probably do have the most secure place that you're ever going to have your content." But the emphasis on security might just hinder the ability to scale. "If you want to have scalability, with the ability to do it very quickly, you take the cost out of the equation right now," Jenkins said. "If you want something that you're going to be able to spin up quickly, spin down quickly, start a channel, [and] do all these types of things that we would like to do within this space, you need to look at the cloud."
Signiant's Thompson agreed about the need to assess security solutions for cloud-based providers. "We move people's content from their [Amazon] S3 bucket into their file storage or to another cloud," he said, "so it's always starting and ending in their own storage. But all during that time, it's fully encrypted. We can't open the file. So we feel pretty good about providing a solid, secure place where we handle their files." Thompson also pointed out that Signiant isn't just resting on its own security laurels. "We use well-recognized third-party security firms that audit every SaaS release we do," he said. "They open it up and do all sorts of crazy testing on it. You have to have a strong security pedigree. We happen to use the same security firm as the Disney organization does, so we feel pretty good about that. But we spend a lot of money and a lot of effort on security. There's no getting around it."
Challenges for Cloud-Based Production
Having talked about some of the benefits of cloud-based production workflows, what are the potential challenges? One incorrect assumption is that the cloud is designed just for the replacement of on-prem workflows.
"A lot of cases I've seen are people doing what's often called ‘lift and shift,' or basically taking their on-prem service and putting [it] in a cloud environment," said Dupont. "That's not going to give you what you want. That's going to be a very expensive way to run virtual servers. That's not what AWS and Azure and Google can do. You've got to sort of adapt to the facilities each provides."
Dupont added that cloud-based solutions also require a different type of skill set than that of the typical IT or media expert who may already be on staff. "If you have workflows that scale up and down a lot, if you set it up right, when they scale down, you'll pay almost nothing. And then, of course, when you scale up, you'll pay for it," said Dupont. "But, usually, when you need to scale up, it means that you have some revenue at the other end attached to that. So I would, in most circumstances at least, say [it makes sense to hire] the developers to do that instead of just running the same workflow that you do on-prem and [on] virtual machines."
Jenkins agreed. "When you think about getting into this space, you do have to think about the personnel that you have currently in your facility," he said. "There is going to be sort of a ramp-up time. There has to be training for those engineers that are in place today, and then you do have to do exactly what Loke said—you've got to enhance that group with the proper personnel to actually help you bridge that gap that you're asking about."
Another topic that came up was latency in cloud-only workflows, with something as simple as audio issues. "Latency is an issue for a couple of reasons," said Ratner. "At the very core of it, you have people in different locations, and it's that traditional, ‘Hey, it's a delay before I hear you and a delay before you hear me back' that we all deal with [for on-camera talent]. From a pure content level, that's an issue. But the other major, major issue I have with latency is on the control room side, which is intercom. You have not just your talent talking, but also everyone behind the scenes. Just latency of people talking and hearing things at different times on different devices, I think, is going to be one of the long-term problems that is going to be very, very hard to solve. Because ultra-low latency isn't enough. A quarter second is way too much."
As part of my background is in videoconferencing, in which a latency of 250 milliseconds is on the upper end of usable content, I asked Ratner about stories I'd heard of small productions actually using a conference call as an audio bridge to have synchronized, very, very low latency phone conversations among team members. "It's a stop gap, because even with that, first of all, you're physically muting and unmuting, and you don't have all of the things like different party lines and the ability to individually talk to people in the same way," said Ratner, comparing an audio bridge to a robust intercom system. "During COVID, [audio bridges] got a lot done for a lot of people, but the fact of the matter is it takes electrical signals a certain amount of time to get from place to place. WebRTC and any other protocols can't cancel out all of those [control room] speakers and all of those people talking at the same time. Even two people in the same room will be sending signals at slightly different times."
Challenges for the Future
If we go back to the pendulum metaphor from the beginning of this article and think about the middle ground, there seems to be two key takeaways from this panel of experts. First, as Dupont noted, there's a misconception that all cloud services are created equal. "I think it's important to learn how each individual cloud works, because they don't necessarily work the same," he stated. "As Bill [Thompson] said, a lot of these customers are choosing multiple different clouds because some of them are good at one thing, and some of them are good at other things. I can certainly understand that if you haven't read about it or haven't really sort of looked into how they work, it can be intimidating. And it's probably also not a good idea to just go in with that assumption and say, ‘OK. This is just any other data center.' Because in reality, it's probably not."
Second, when considering hybrid approaches, remember that private clouds and public clouds serve specific purposes. In fact, Jenkins advocates tying private clouds to public clouds. "I believe in bridging your private cloud with your public cloud," he said, "and I think that is the direction that will get people to jump on board a lot faster. You have to get comfortable with the fact that there is just a significant security integrated into the major cloud instances and the major SaaS providers. When you're inviting them into your home, you need to make sure that you vet them in the same way that you would vet something that you were building internally," he said, referencing the integration of a public cloud or SaaS solution into an on-prem workflow. "If you do that, then I think there's going to be more of a comfort level from the security standpoint."
Finally, there are always challenges that will arise when production teams aren't face to face with one another. "I envision a world in the cloud, and even if you're hybrid, where you have an on-prem control room but people out in the field," said Ratner. "If so, you're never going to be able to use speakers ever again in a control room if you want people not to hate themselves and want to not work for you because they keep hearing echoes in the headsets. Outside of a string of paper cups, I'm not sure if there's any magical way to get that [audio latency] low enough to be the true real-time [equivalent] of a physical control room with everyone on-prem together. I don't know of any legitimate way to make that work, just because of how physics and internet speeds work."
That's a good challenge for all of us audio-focused technicians who live and work in the hybrid world between cloud-only and on-prem solutions. But we're not going to solve it with the VHS versus Betamax mentality of two technologies that can't coexist with each other.
Related Articles
MovieLabs' CTO Jim Helman discusses his experience creating standards for content classification for the Hollywood studios, as well as the MovieLabs ten-year strategy to bring interoperability to as many vendors as possible. The Entertainment Identifier Registry (EIDR) Helman helped create is a content ID which now holds more than 2.8 million records modernizing many aspects of production and operations. He'll also update us on the industry's response to the 2030 Vision strategy to bring interoperability to media creation technologies, requiring less custom engineering and better communications between software, infrastructure, and services.
15 Dec 2023
Migrating video infrastructure to the cloud is no simple undertaking for government organizations, schools, and other institutional environments, and even though cloud infrastucture is becoming more mainstream, it's not always the way to go. DiscoverVideo CEO Rich Mavrogeanes discusses some examples of government clients and institutions that have gone in different directions on cloud, vs. on-prem, and why they made the choice they did.
24 Jun 2022
CNN's Ben Ratner, LiveU's Mike Savello, and Signiant's Jon Finegold discuss best practices and lessons learned from forays into REMI and cloud production that have yielded successful workflows in this clip from their panel at Streaming Media East 2022.
22 Jun 2022
Megan Wagoner of Norway-based OTT Video CMS provider Vimond discusses the current state of cloud broadcasting and how the pandemic has accelerated migration to the cloud in this interview with Streaming Media's Tim Siglin at Streaming Media East 2022.
07 Jun 2022
Streaming Media Contributing Editor Tim Siglin and CNN Operations Manager discuss the persisting benefits of cloud and hybrid production as practiced at CNN in this interview from Streaming Media East 2022.
02 Jun 2022
Companies and Suppliers Mentioned