Best Practices For Premium Video Streaming, Part 5: Performance Testing
OTT distributors are discovering that even seemingly minor glitches in online video quality have the potential to undermine results. Their ability to maintain comprehensive quality control (QC) has become mission-critical in a highly competitive market where viewers have shown little tolerance for poor performance. Knowing how consumers are reacting to their services is essential, and the ability to confirm execution of ad performance is becoming ever more important to bottom lines.
Meeting QC requirements is a tall order, given the range of applications and touchpoints involved in any given streaming scenario. Distributors need visibility into everything impacting the content flow, from initial processing to end-user playback. They must be able to easily assess the effectiveness of playback, including advertising and applications supporting DVR and time shifting.
This piece explores monitoring, analytics, and other best practices that are essential to both achieving desired streaming performance and producing optimal viewing experiences and financial returns.
Comprehensive visibility begins with telemetry to gather useful information from the first mile to client-side user experience. This requires timely coordination of data feeds to speed analysis of performance issues and shorten time to mitigation.
Preparing for Prime Time
Whether preparing for the launch of an OTT video service or a large, live streaming event, the importance of planning and collaboration can’t be overstated. Content providers should consider partnering closely with their video workflow and content delivery network (CDN) providers to ensure there is a common understanding of service goals. All parties should work collaboratively to test integrations, develop and deploy backup plans, and clearly identify and define roles and responsibilities.
In addition, content providers should consider load testing their entire video workflow including all apps, sites, and key partners. This validates that the workflow is ready to handle large-scale audiences and helps ensure viewers are able to find their desired content in any given user interface. This type of testing can identify potential weak points in the architecture and provide the opportunity to mitigate issues before they impact viewers.
Monitoring Content Preparation
Every step in the content preparation process must be monitored to identify issues before they cause disruption. Workloads should be properly apportioned to avoid overloading on transcodersand ensure quality, and latency expectations match the goals that the content provider has set. Expectations can vary depending on whether the use case dictates the need to provide a 1080p or even 4K UHD stream that prioritizes high quality over latency versus a scenario where low latency is paramount and resolution can degrade slightly to meet the latency goal.
Comprehensive monitoring will determine whether transcoders are operating within set quality and latency boundaries, which can be derived from data provided by the transcoders.
Content providers should apply the same visibility into their backup encoders to ensure outputs maintain their latency and quality levels. Monitoring input to all servers is essential to identifying situations that call for switchover to backup.
Monitoring First-Mile Distribution
Maintaining quality control over the first mile requires continuous performance monitoring and analysis. This helps ensure content providers can activate alternative redundant paths between encoders and the ingest network before a disruption occurs on the primary workflow. Visibility into first-mile performance provides insight into the impact of latency and packet loss on video quality (start up time, rebuffering). It also makes it possible to tune mitigation responses based on those levels and determine whether source encoder output quality levels are being achieved at the points of ingest. Any major issues in the first mile can potentially lead to problems during playback, making QC in this area critical for OTT providers.
The best practice is to extract performance and video quality information from the encoders directly as well as the ingest network to ensure access to continuous monitoring and analysis for the first-mile.
Monitoring CDN Performance
The next step is for the CDN provider to maintain quality assurance within the network. This requires centralized access to data from all components and information about specific content streams in near-real time. The goal is to proactively identify network congestion, mitigating issues before they occur.
CDN providers should offer server-side insight into network performance including: availability, performance or throughput from edge server regions, traffic volumes, and HTTP status codes. A CDN should have the ability to use that information to ensure traffic flows are unimpeded. The provider must also share pertinent data with their customers, often in a variety of ways including dashboards and/or portals, logs, and APIs.
Playback Measurement and Analytics
Capturing what transpires at playback on every device is vital not only to determining whether video streams are delivering the anticipated quality of experience (QoE); it’s also essential to assessing the appeal of content and whether ad placements are properly executed. To capitalize on the full potential of client-side data collection at playback, providers must be able to rely on player technology and integrated analytics software that can aggregate and process data - all while adhering to local and regional privacy laws.
Specifically, with a robust, fully scaled playback monitoring and measurement system in place, distributors can:
- Assess quality of user experience from data reflecting play attempts, startup failures, startup time, video availability, bitrate, rebuffering frequency, and durations
- Determine audience engagement and thresholds for tolerance of suboptimal QoE based on viewers, plays, play durations, abandonment rates, completion rates, bitrate, and rebuffering;
- Better understand their viewers by using playback data correlated with account information to develop profiles based on visit and play histories;
- Filter data by geography, devices, connection speed, ISP, video length, and more;
- Support advertising by validating ad placement and proper playback, and apply audience engagement metrics to measure viewership.
In all cases, distributors must be able to integrate the data streams into their analytics workflows. To facilitate these integrations, the compiled data, should be presented with commonly used open interfaces (APIs).
Data Analysis
Distributors’ ability to use data drawn from the delivery chain depends on analytics capabilities that can be optimized to parse and extract relevant data. First and foremost, the tools available to providers must help identify root causes by correlating any problem exposed in playback with data from across the distribution chain.
Monitoring and analytics tools need to be flexible and fast enough to provide reports that quickly identify high-level trends and performance thresholds. They should also have options to drill down in granular detail; for example, examining totals instead of averages, or analyzing data across different customizable dimensions.
When it comes to advertising, reporting needs to verify accuracy of placements, seamlessness of transition to and from ad breaks, and the number of viewers who kept the streams running through the breaks. Playback analysis is also essential to providing content providers an objective check on the accuracy of third-party metrics to validate ad performance.
The stakes in commercial streaming operations are too high to leave to chance. Content distributors need certainty that the content they are delivering will meet their viewers’ expectations. And a critical part to achieving this is establishing strong monitoring and analytics capabilities in order have visibility into content delivery from beginning to end.
[This is a vendor-contributed article from Akamai. Streaming Media accepts articles from vendors based solely on their value to our readers.]
Related Articles
The CEO of RealEyes shares his personal video engineering toolbox, which contains solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video.
03 Jun 2019
Video playback is where the rubber meets the road. Make sure you're using a player that gives you all the functionality you need to successfully deliver your high-quality content to a fragmented device universe.
17 Oct 2018
Steps for ensuring CDN performance meets audience expectations for OTT streaming
22 Aug 2018
The second installment in a series looking at best practices for delivering premium video content, this piece explores preparing OTT video for delivery over content delivery networks.
15 Jun 2018
In this first installment in a series of articles looking at best practices for delivering premium video content, we look at managing the first mile.
02 May 2018