SSIMWave SQM Review: Frustrating Video Quality Measurement
The perfect video analysis tool combines a video quality metric that accurately predicts the subjective ratings of real human eyes with the ability to show these quality differences to the operator. While SSIMWave makes a strong case that its SSIMplus algorithm has those predictive capabilities, the company’s SQM (for SSIMWave Video Quality-of-Experience Monitor) video analysis tool provides only a limited ability to visualize and confirm these differences. If you can get comfortable with the idea that SSIMplus is the best video quality metric available, you’ll find SQM a highly efficient tool for calculating the SSIMplus rating within the UI or batch mode. But I’m from Missouri—the Show-Me state—so when it comes to making compression decisions, I want to see easily the differences that the tool is measuring. If you’re like me, you’ll find the $3,600 tool frustrating in this regard.
Overview
Compressionists need quality testing for purposes that range from choosing a codec to finding the optimal compression parameters. While the gold standard is always triple-blind subjective testing, this approach is time-consuming and expensive. Therefore, we have a number of software-based quality measurement tools and metrics, such as the Moscow University Video Quality Measurement Test we reviewed back in 2014, which can compute multiple quality metrics, including the popular peak signal to noise ratio (PSNR), the structured similarity index (SSIM), and the video quality metric (VQM).
The problem with these metrics is that they don’t relate directly to any meaningful human measure: good, bad, or excellent quality. The scores indicate that one video is of higher quality than another, but provide no clue as to whether either video will look good to a human viewer. Measures such as PSNR, SSIM, and VQM also don’t take into account the viewing platform, specifically that a 640x360 video that looks great on an iPhone might look awful when viewed in full screen on a 31" monitor or 65" TV.
SSIMWave’s SQM tackles both problems, rating all videos on a scale of 1 to 100, with 20-point gaps separating the video into bad, poor, fair, good, or excellent ratings, and the ability to apply the metric to different viewing platforms. You can see this in Figure 1. On the upper left is the test video, on the upper right, the quality map showing how it differs from the source video from which it was encoded.
Figure 1. The main interface of SSIMWave’s SQM
The graph on the lower right shows the ratings for the various devices tested, which includes the SSIMplus Core, a composite metric computed in every test, and device scores for an iPhone 6 Plus, an iPad Air, a Lenovo 30" monitor, and a Dell 27" monitor. Though the video (barely) rates as excellent on the relatively small screen of the iPhone 6 Plus, it drops to mid-good ratings on the larger monitors. The graph on the lower left tracks those scores over the duration of the clip.
You can run the tool in GUI mode (Figure 1), or in batch mode, which I’ll describe later. In both cases you can choose up to 10 devices to score, and you can save groups of devices as profiles to simplify multiple device testing. The program comes with 31 device profiles, and you can request additional profiles after purchase. Another unique feature is the ability to run cross resolution testing, answering questions such as whether an 854x480 video looks better on an iPhone 6 than a 640x360 file does. You’ll see the results of this testing at the end of this review.
Manual Operation
In manual and batch operation, SQM computes the SSIMplus score for a single file as compared to its source file. In both modes, operation is very straightforward, with several wonderful convenience-oriented features. As shown in Figure 2, you can load HEVC files directly, as well as MP4 and VP9, saving the conversion to YUV step required by other tools.
Figure 2. Loading the test file (on the left) and reference file (on the right)
There’s also a Frame offset control for both videos (set to one for the test video in Figure 2), which is an exceptionally simple way to eliminate the extra frames some encoding programs insert at the start of a video. You can also limit the number of frames processed via the Process frames adjustment at the bottom of the upper left panel; between this and the frame offset controls, you can easily test random sections of longer video files.
On the bottom of Figure 2, you can see the device settings available in the base product. According to SSIMWave, these settings take into account the resolution and size of the screen, its luminance, the typical viewing distance, and other factors. The “Expert” in the Sony W8 device name indicates that the quality rating assumes that an expert viewer is watching from very close to the screen, rather than the typical viewing distance. All settings assume that the video is watched at 100 percent screen size, so if you wanted to simulate playback in a video window, you’d have to ask SSIMWave to create a different setting.
After loading the video files and choosing the settings, click Continue on the lower right of Figure 2, which takes you back to the screen shown in Figure 1. To start the analysis, press the Play icon on the bottom of Figure 1, which toggles to a Pause button. During the analysis, another icon appears to the right of the Pause button that lets you toggle the visualization on the upper right to the reference video rather than the quality map shown in Figure 1.
Significantly, there’s no way to rewind or randomly seek through the analyzed file; if you want to see the quality map for frame 18:13, you have to stop the video on that precise frame. Of course, that’s very hard to do, and you wouldn’t know that you wanted to see detail on that frame until you actually ran the test. SSIMWave plans to add simple player controls by the end of 2015, which will be a very welcome addition.
There’s also no way to display the test file over the source file, which is a better way to spot artifacts than side-by-side views, and is a view available in other tools, among them the aforementioned Moscow University VQMT tool and Vanguard Video Visual Comparison Tool. As mentioned, SSIMWave’s SQM analyses a single file only, so there’s no way to visually compare the results of two compression alternatives against a single source, such as VBR vs. CBR, or High vs. Main Profile, or VP9 vs. HEVC. This is an invaluable feature of the VQMT tool and is particularly great for showing consulting clients the quality differences delivered by the various compression alternatives being analyzed.
Related Articles
The tool predicts how a human being would see a piece of video, and consistently gets 90% or higher correlation with the video's mean opinion score.
04 Jun 2019
Netflix announced the open-source availability of the Video Multimethod Assessment Fusion, which it's now using instead of PSNR to analyze the quality of transcodes in its vast catalog
07 Jun 2016
Objective quality benchmarks provide exceptionally useful data for any producer wanting to substitute fact for untested opinions.
01 Apr 2016
How can video compressionists assess the quality of different files? Only by combining objective mathematical comparisons with their own professional judgments.
13 Jul 2015
Which delivers better quality, encoding time, and CPU performance—HEVC or VP9? We put them to the test to decide once and for all.
28 Apr 2015
The delivery optimization company finds that viewers quickly stop watching poor video and often leave the site that's providing it.
26 Mar 2015
Never heard of it? Learn why we call this video encoding analysis tool invaluable, and a must-have for anyone serious about encoding or compression.
30 Dec 2014