Leveraging Amazon’s open source Media Replay Engine (MRE) framework for automating video clipping and replay generation, alongside Amazon Web Services (AWS) infrastructure and solutions, Catch Up With Highlights is an end-to-end solution that can ingest broadcast streams, identify valuable match moments through AI-trained computer vision, and then encode delivery streams with metadata across user devices. With AWS already the foundation of Fox Corporation media workflows, the development team was able to stand up the complex new feature in about eight weeks.
FOX’s Vice President of Data Products and Machine Learning (ML) Strategy, Phil Martin, who led the Catch Up With Highlights development and first deployment, shared, “Over the last five years our team has had tremendous success with unlocking the power of data through the building of advanced and intelligent platforms and products. In addition, we’ve really rallied around the idea of revolutionizing fandom. We believe that sports should be enjoyable and accessible for everyone, which is why we’ve designed our platform to make it easy to engage with and enjoy. Our Catch Up With Highlights feature is a great example of this philosophy. We understand that the level of play at events like the World Cup is exceptional, and we wanted our feature to reflect that standard. To achieve this, we developed a cutting-edge AI technology to create a highlights engine that can dynamically retell the game’s story in real-time, highlighting key moments and events as they unfold. This ensures that our users can stay up-to-date with the action and experience the excitement of the game, no matter where they are. Prior to and through this journey, AWS has been a trusted partner of ours and together we were able to take another step forward in revolutionizing fandom with data all while on the world’s biggest sporting stage.”
Upon launch, the feature ran seamlessly with very little human input. Martin and his team built Catch Up With Highlights to continually optimize the highlight recap video as the game progressed, retelling the game story upwards of 20 times per match with event-driven updates. This approach ensured that the viewers would be able to access a comprehensive recap of the game to date, available via the FOX Sports app on connected TVs and mobile devices. Incremental improvements were rolled out based on performance, such as adding a thumbs-up or down; and event enhancements, including shots saved and penalty kicks. Ultimately, 731 recap videos were published autonomously across 61 matches.
The Catch Up With Highlights workflow includes AWS Lambda event-driven serverless compute, and computer vision with Amazon Rekognition. Sports data is housed in an Amazon DynamoDB NoSQL database, then piped into MRE. Live video is quickly delivered to viewers via AWS Elemental MediaConnect, with videos stored in Amazon Simple Storage Service (Amazon S3), and Amazon ElastiCache for caching live clips.
“Soccer is often referred to as ‘the beautiful game’ because of the flowing and artistic style of play and objectively it’s a simple sport: score more goals than the opposition,” Martin explained. “However, training models and developing logic around a game that lacks set patterns is anything but simple. To further compound the complexities, it was our first time publicly releasing a platform feature that ingests multiple concurrent streams, truncates clips for inference, contextualizes and optimizes each segment, and transcodes and delivers directly into our CMS in a matter of moments of an event occurring.”
Additionally, the team had limited reference information from the event organizers. Instead, they relied on audio cues and camera angles to help train the computer vision model. Clips were auto-assembled with polished elements, like transitions and a fade to black, to ensure a high-caliber viewing experience for fans. Now, with a proven, scalable approach, Martin looks forward to replicating the Catch Up With Highlights playbook for other sports broadcasts.
Martin concluded, “We are extremely bullish on this space and already expanding to other sports properties. Fans can experience the feature today during broadcasts of Major League Baseball (MLB), the United States Football League (USFL) and the FIFA Women’s World Cup™?. Catch Up With Highlights is coming to football broadcasts this fall and more personalized content is on the horizon. The possibilities are endless and we’re just getting warmed up.”
A lot of products tagged as "AI-powered" were doing exactly the same thing before AI. But labeling their familiar features as "powered-by-AI" makes the same tech seem fresh and innovative.
11 Apr 2024