So there I sat, popcorn in lap, waiting for Saturday night’s sixth game in the American League Championship Series best of seven between the Tampa Bay Rays and the Boston Red Sox to go live. And so I sat, and waited, and waited, and waited while watching “The Steve Harvey Show.”
By the time the game did come on, the popcorn was cold and I was hot. Twenty minutes past the time when the broadcast was supposed to air, the game came on in the second inning. By that point, the Rays led 1-0 on a solo home run by outfielder B.J. Upton against Red Sox starter Josh Beckett.
Scheduled to begin a 8:08 p.m., the broadcast didn’t begin until 8:28 p.m. As avid Red Sox fans, my wife and I were a little miffed about missing the first 20 minutes of the game. All right, we were darned hot under the collar. My wife works in television, so she was in shock over the obvious broadcasting debacle.
It took Atlanta-based Turner Broadcasting System, Inc. (TBS) — the network that was broadcasting the post-season games — what seemed like 10 minutes to even get a crawler on the screen, saying: “We are experiencing technical difficulties.” By that time, I’d cycled my 850 cable channels twice searching for the game and contacted friends and a family member to see if the same problem had occurred all around the country. It had.
It wasn’t until this morning that I discovered the cause: Two failed routers.
“Two circuit breakers in our Atlanta transmission operations tripped, causing the master router and its backup — which are necessary to transmit any incoming feed outbound — to shut down,” TBS spokesman Sal Petruzzi said in a statement.
“This impacted our live feed from being distributed to any of the other networks in the Turner portfolio and caused the delay in our coverage,” Petruzzi said. “Both our primary and backup routers were impacted by this problem. We apologize to baseball fans for this mishap that caused a delay in our coverage.”
In other words, TBS’s business continuity plan failed miserably. A major network allowed two routers to hamstring it during one of the most important baseball sporting events of the year. My wife commented that they must have lost millions of dollars in advertising money and may lose out on future sports coverage contracts because of it. (Full disclosure: My wife works for a competitive network.)
But that’s exactly the price that’s paid for not testing a business continuity plan thoroughly: embarrassment and loss of money and reputation.
TBS needed an off-site router to which it could fail over in the case of a main data center emergency. It’s an important lesson that most CIOs probably would agree is universally understood. Yet, the problem persists. Just one more example, I guess.
