Solving the OTT Worfklow Puzzle
The streaming workflow has myriad moving parts: capture, transmission, encoding, transcoding, quality control, video players, packagers, ad insertion, analytics, apps, search, personalization, playout, content management, and the list goes on. With so many pieces, is it even possible to have best practices? The true challenge for streaming technology professionals is not "How do I make it look like Netflix?" but "How many pieces of technology do I have to connect together?"
In June, I moderated a panel at Streaming Media East Connect called "OTT Workflow Integration Best Practices," and what follows is a summary of the high points. More than 800 viewers registered, so clearly it's an area of interest. If you'd like to see the whole hour-long session, you can watch it at go2sm.com/bestpractices.
The panel featured the following speakers:
- Rema Morgan-Aluko, director of software engineering at FandangoNOW, a subscription video-on-demand and transactional service that has more than 80,000 different TV and movies assets
- Michael Bouchard, VP of technology strategy at One Media, a subsidiary of the Sinclair Broadcast Group, which runs 200 TV stations, 20-plus regional sports networks, the Tennis Channel, STIRR,and other properties
- Magnus Svensson, VP of sales and business development at Eyevinn Technology, a development shop that helps customers withintegration, vendor selection, due diligence,and strategic and technical advice
- Geir Magnusson Jr., CTO of fuboTV, aU.S.-based virtual MVPD that offers sports, news, and entertainment via 1000-plus channels of live and video-on-demand (VOD) content
- Darren Lepke, head of video product management at Verizon Media. His focus is on providing a streaming platform that powers many of the largest OTT services
in the market today.
What Are Your Workflows?
Oxford Dictionaries defines workflow as "the sequence of industrial, administrative, or other processes through which a piece of work passes from initiation to completion." While our experts work in similar areas, exactly what's in their workflows varies.
The FuboTV team has built a lot of its own technology. "We do linear and VOD acquisition, transcoding and distribution, server-side ad insertion, and because we are a direct-to-consumer product, we have a bunch of workflows in and around consumer acquisition, content promotion, and recommendation," said Magnusson.
Sinclair Broadcast Group has been a company of acquisitions. "We have typical digital VOD workflows, our digital origination feeds from within our cloud, and [there's] retransmission from companies like Cheddar. We have linear over-the-air feeds, digital studio feeds, where there's a DRM component involved. In all of these workflows, it's really important for us to make it easy for our producers, so we've worked really hard to try to put them in what appears to be a single workflow," said Bouchard.
FandangoNOW has workflows for VOD and pay-per-view content, as well as original content from its Rotten Tomatoes group, which publishes to social media platforms. "We have different workflows that can handle both short and long form at different types of speeds and efficiencies. We have captioning workflows [and] image proxy workflows," said Morgan-Aluko.
More than likely, many parts of each workflow these speakers mentioned have been around for a while. Greenfield development is rare, which brings us to the next topics.
Legacy System Mindset?
Most companies have legacy systems. There's always going to be an interest in fully amortizing a technology purchase—in other words, no one is walking around with an open checkbook. When this topic came up, I intended for the experts to discuss their own systems, but a few people over the course of the conversation talked about customers' use of legacy devices. My goal was to uncover how to make choices around managing legacy technologies within workflows.
"Just because you acquire something doesn't mean you replace all the systems. No one could afford to do that. For us, it's all about coming up with an integration platform to be able to integrate all of these disparate systems so that they can work in the entire ecosystem," said Bouchard. His company has an integration plane that it developed. "It's about building adapters to whatever those systems are to go back and forth to."
When there is legacy technology in place, it's important to make sure there is constant evaluation to find out if it's meeting important needs. There is going to be a cost for continuing to support the legacy devices as things move on, and it starts to become way more expensive than the revenue being generated. "I think all teams, groups, and organizations should monitor and really look at those issues and problems that come up and always ask yourself these questions," said Morgan-Aluko:
- Can you do it faster?
- Do you have the expertise in-house to
make the modifications necessary? - Does it make sense to continue to support these platforms?
- Is there something out there that's better?
- Can I outsource this?
- Can I buy something off the shelf?
- What if you have multiple clients?
A consulting development shop interacts with lots of clients with different workflows, so its experiences are a bit different. "We usually use a blueprint model for the workflows and compare that to what the customer has and where they want to be," said Svensson. "We evaluate the cost, competence, and complexity to replace. Sometimes, it's not just swapping and replacing; [there] needs to be integration in a couple of places." Eyevinn Technology uses this blueprint to explain to customers, colleagues, and stakeholders what needs to be changed and the complexities involved with making those changes.
Eyevinn Technology begins consultations on client workflows with this blueprint, which can be modified or adapted to the specifics of that client's needs.
Upcoming Challenges
Streaming may have taken off in terms of hours viewed by the consumer, but what does this mean for strategic planning? What is top of mind with our experts? It turns out most everyone is thinking about advertising.
Data
"I think the other way to look at integration is, ‘What are the analytics that need to come out of your platform?'" said Lepke. "It's important to have a system that hangs together, that can share data and ultimately integrate all of that data to provide you the insights you need to bring on better ad partners or stream your video in different quality or even start to do things like look at the content itself."
Today, companies have moved past "How do we do this?" and are now trying to understand what is profitable, said Lepke. "Our customers are coming back to us and asking, ‘Which of my ad partners, which of my SSAI ad decisioning partners is performing best? What are the profitability metrics that we can see across all of our content?'"
Growing the Business
"One of the most important [things] to our business, certainly right now, is really optimizing marginal revenue from advertising. Advertising is a very important part of our business," said Magnusson. "Once your avails are gone, they're gone. You'll never see it again."
FuboTV focuses on viewers coming for sports and then staying for entertainment. "The lack of sports has been interesting for us," said Magnusson. The company's consumers have increased news and entertainment consumption to replace the missing content. "As a business, we're constantly focused on making a better consumer experience. We recently became a public company, so that brings a set of challenges. That's going to be our focus: increasing subscriber growth, increasing our margins.
"Because of the growth of OTT in the last couple of months, we're starting to see some of the national distribution infrastructure getting a bit stretched. So as we get even more content with more simultaneous viewership (which is what happens with sports), we're paying very close attention to this," said Magnusson.
Balancing SSAI and CSAI
"Ads are definitely a very important part of our digital revenue," said Bouchard. Sinclair Broadcast Group found it can't only take an SSAI approach. "We actually support both client and server side depending on the situation, and we actually have some streams that do both interchangeably," he said. The issue his company faces is that live news broadcasts aren't scheduled, and breaks don't always fit neatly into the timing of an ad pod. "We thought we could buy it because we thought, ‘Everyone's showing ads,' right? You watch the Super Bowl, you see ads, but it turns out that's not truly live—they understand their pod size. The other thing that surprised me was finding out that for certain advertisers and certain programmatic ads, because of the analytics we get back, they pay a higher CPM for client side."
"I would say that for a server-side ad insertion, preparation is key," said Svensson. "If it's a live event, a lot of viewers will request ads at the same time." For preparation, he recommends the following:
- Prepare the content coming in with markers.
- Have slates and fillers prepared in caseof backup.
- Have properly encoded ads coming in, so you don't have a glitch in quality for the ad itself.
- Be prepared for high peaks when everybody's asking for ads at the same time.
- Use preload and header bidding techniques to flatten the curve.
"When we talk about the CPMs today, SSAI really supports VAST advertising and VAST protocols. Client side still supports VPAID with some of the interactivity there; that's where you see some of the higher CPMs," said Lepke. "I think with some of the advancements and protocols around VAST 4.0, some of the open measurement standards, some of the things that are coming online there, I think we're starting to see a little bit of rebalancing of those CPMs, because you're starting to get more analytics from the server side. You also have to balance inventory in your programmatic or presold ad system. If you only have VPAID ads and not VAST ads, that presents a challenge."
Build or Buy?
The question of whether to build or buy is an age-old debate, and our speakers offered terrific insights. In the hot direct-to-consumer market, the first topic that came up was apps, but advertising soon crept back into the conversation.
Apps
"I think this is mainly about your team and your team's expertise," said Morgan-Aluko. "At FandangoNOW, we have apps on many different platforms, and so we have a reasonably sized group of engineers whose bread and butter is building apps, workflows, DRM solutions, things like that. Those things we wouldn't want to outsource.
"Where's my expertise?" Morgan-Aluko recommends asking. "If something is core to my business, maybe that's something that I want to make sure I keep internal, and if something is not core to my business and there's someone else that can do it better, faster, cheaper, then I'm going to do that."
"Smart TVs are usually very complicated to keep track of," said Svensson. "If you're not strong in device certification development, stay away from smart TV apps and outsource that to someone that can do it at scale and knows proper verification on the old models and operating systems."
Tools
Sinclair Broadcasting Group delivers video to many distribution channels. "There's no one tool that we could buy to handle that. If there was, we would," said Bouchard. When it comes to doing pre-positioning of ads for ATSC 3.0, for instance, the company had to write everything itself.
"All of these systems have to be updated, and we're always trying to make them more integrated," Bouchard said. "So, for instance, if you look at our contracts and rights management, we do a contract with an MGM studio, for instance, and there's an ATSC 1.0 component and an ATSC 3.0 component, there's an OTT component, there's stuff we can show within the DMA [designated market area] where we have a station, there's stuff we can't. There are many rules for all the different distribution channels that we have to keep in place. For things like that, it's really important to have one system where all that information is. If you have multiple systems, hopefully, they can tie in and be something better than a human copying things back and forth."
"Our integrations these days tend to be with new content providers," said Magnusson. "We're getting into new linear channels, and they usually they come with VOD. Those tend to be pretty straightforward, because our workflows over the last 2 years have standardized into things that just tend to work, because there has been a set of general practices in the industry long before we were here. Very often, there was integration for SCTE data. That is fairly straightforward; you're calling somebody's API. [However,] nobody ever responds in exactly the same way. So the teams are pretty good at ensuring that we can integrate scalably and make sure it's not too bespoke.
"We spend a lot of time ensuring that our architecture [works], because we write a lot of it. A lot of the infrastructure, a lot of our distribution system is really custom and bespoke," said Magnusson. "We write our apps; everything between the encoder to the app is ours."
Magnusson said he focuses on "making sure that you have a design and an architecture that's thinking forward at least to some bit. I mean, you can't see too far in the future; then you're just doing speculative engineering. We pay attention to what's going on in the larger universe and make sure that at least we're taking steps forward with new things."
But sometimes, new is too risky. Be careful, surround yourself with friends, and don't be the guinea pig for something new, advised Svensson. "If you want to replace a component, pick another component that's worked before. Try to use open interfaces instead of proprietary interfaces. Use partners and pre-integrations." Going with more mature, standardized protocols and interfaces is a safer bet, he concluded.
"One of the benefits of using a modular architecture is that if there's a certain piece, you can take that out, and you can plug it in with something else," said Morgan-Aluko. Ask whether documentation is available, whether that documentation is easy to understand, and whether or not it's a well-known system. "Make sure that it integrates with the rest of your system. I think integrating is not the easiest thing … but I never want to lock myself into a system that I can't get out of, because things are changing all the time."
TV and Digital Workflow Convergence
Because TV and digital are borrowing from and influencing each other, we are seeing the distinctions blur between the two. For smaller services, the advice was pretty unanimous. "I would say start from where you are with your business. Start small, and expand instead of trying to reach everything from the beginning," said Svensson. "I would say CMAF [Common Media Application Format] is definitely the way to go. Of course, there are devices that won't support it, but maybe you can live without those. If you don't require DRM in the beginning, maybe you start with HLS [HTTP Live Streaming] only and go from there and expand."
On the other end of the spectrum, there's Sinclair Broadcasting Group's environment. "We have a broadcast app that's being transmitted over the air right now that does converge broadcast and digital. Within that broadcast app, the over-the-air signal is … a mixture of MPEG MMT [MPEG Media Transport] with SHVC [HEVC Scalability Extension]; there's some route DASH channels. The digital stuff within Sinclair tends to be HLS because we need it to work on Roku," said Bouchard. "If you're starting out, start out small. We've been working on this for years, and it is complex. We have vendors and TV manufacturers that have trouble keeping up."
"On our sports side, since we just acquired those from Disney, we had to replace all of the infrastructure. Now they're IP-based video origination distribution. We're doing the same thing for all of our 200 TV stations," said Bouchard. Sinclair Broadcasting Group also will be moving its TV stations to ATSC 3.0 over the next 2 years.
It's a Wrap
"How do we best plug into an existing system without causing additional work or additional disruption?" asked Lepke. "It means having both [user interfaces] and [application programming interfaces] that can be automated and done seamlessly in terms of the integration, and ultimately, starting with the customer needs and what they're trying to achieve."
"I think as an engineer, your job is to solve problems no matter what kind of problem comes your way. I think we apply that to workflows. We apply that to technology. We apply that to the platforms that we build in support. So you always have to be flexible," said Morgan-Aluko.
As streaming continues to thrive, the approaches our experts discussed have ensured they will be successful in delivery. But it's difficult to identify best practices that will work across different workflows. Since there's no "easy" button, each of our experts' companies has developed its own approaches.
Related Articles
Verizon's Darren Lepke and Streaming Media's Nadine Krefetz discuss the importance of efficiency, cost, metrics, and ad partnerships in OTT integration in this clip from Streaming Media East Connect 2020.
18 Sep 2020
FandangoNOW's Rema Morgan-Aluko, fuboTV's Geir Magnusson, Sinclair's Michael Bouchard, and Eyevinn's Magnus Svensson discuss best practices for integrating OTT components and platforms in this clip from Streaming Media East Connect 2020.
02 Sep 2020
Verizon's Darren Lepke, fuboTV's Geir Magnusson, and Reality Software's Nadine Krefetz discuss the present and future of OTT in this clip from Streaming Media East Connect 2020.
31 Aug 2020
Bitmovin's Steve Geiger outlines the benefits of per-title encoding, how to use it optimize your delivery, and what the workflow entails in this clip from Streaming Media West 2019.
15 Apr 2020