The Hazards of Streaming Live Video: If It Can Fail, It Will Fail

When I’m hired to webcast an event for a client, I like to believe that I’ve planned adequate levels of redundancy for nearly every aspect of the production. Our kits have backups for everything from cameras, lenses, and microphones to video switchers and encoders.

For the encoding pipeline, we standardize on having dual outbound internet connections, using one per encoder. For the government work we do, we usually have access to two dedicated connections: an internal government backbone with more-than-sufficient bandwidth for 720p broadcasts and a local carrier connection such as a cable modem. Each H.264 (or H.265/HEVC) encoder uses a dedicated connection, and the Wowza Streaming Engine tech we’ve built allows an immediate switchover to the secondary encoder stream should the primary stream fail.

Having two connections is likely enough redundancy for just about any live streaming work we do, and I fully appreciate that not all scenarios will allow for having one, let alone two, hardwired internet connections. In the past couple of weeks, though, we’ve had issues with internet stability at venues where previously there were none—and because we were prepared, we had minimal downtime for the webcasts at hand. I thought it would be useful to readers to put some guidelines together for redundancy throughout the entire live streaming pipeline:

1. Prepare multiple networking options in advance of the live webcast.If you’re working with a hotel or conference venue, ask just how dedicated your internet connection will be. Do they have a backup internet provider you could use? If not, look into the cost of hardware and data plans that can use multiple USB 3G/4G/ LTE modems such as those offered by Mushroom Networks or Teradek. I also like to prepare a single hotspot device such as a ZTE Turbo Hub as a third backup. Essentially, any networking appliance that can provide a wired Ethernet connection to an encoder can play a backup role.

2. Prepare more than one origin (or ingest) streaming media server. If you’re using a CDN or hosted streaming environment, make sure you have primary and backup publishing points, one for each encoder. If you like to build your own streaming server instances on the cloud, have two origin servers ready to receive the incoming streams from the server. You may want to have one origin on standby should the primary origin not be reachable (e.g., an Amazon Web Services region becomes unavailable due to a DDoS attack). Your edge servers need to be configured to locate the availability of streams on each of your origin servers.

3. Thoroughly test the throughput of each internet connection prior to the webcast. Run your encoders for at least 30 minutes to ensure that bandwidth is stable. If possible, monitor the incoming stream on your streaming server to see if bandwidth fluctuates wildly during the test. Be sure to run some “active” content during the test—encoders will not necessarily use the full bitrate you’ve specified if you’re simply pushing color bars and test tone. Run live camera feeds of the venue, preferably during a dress rehearsal of the event.

4. Thoroughly test the failover system prior to the webcast. Take each encoder offline to see how effectively your system works. If you have a solid setup, you can bring each component back online without your live stream player displaying an error.

5. Have a disaster recovery plan in place. Write up a specification that anyone monitoring the encoding gear can follow in the event of a network outage. Backups can fail too, and your clients should know in advance how you and your team will respond in the event of one (or more) network outages. You can extend this plan to include anything related to your production setup, including camera or mic failures.

If you have redundancies built into your live streaming checklist, you’ll know exactly how to handle any failure before it turns into a disaster.

[This article appears in the April/May 2018 issue of Streaming Media magazine as "Internet Rules: If It Can Fail, It Will Fail."]