Streaming media in the fast lane
Activate and Yahoo Broadcast in photo finish
Streaming media is more than just listening to music, watching a movie trailer or funky animation. It's also an underlying set of technologies that can help your company communicate more effectively, more consistently and more conveniently with employees, partners, resellers, prospects and customers.
When streaming media was new, early adopters installed it in isolated networks and tested internal applications with small numbers of users. In the past four years, streaming media usage has soared. RealNetworks reports that more than 150 million users have registered their client software applications, and thousands of consumer and corporate sites post new content daily.
Warning!
All the charts from this review
Race ground rules
What about Akamai?
More Ground Rules
The joy (and pain) of measuring streaming media
Glossary of terms
NetResults charts
Archive of Network World reviews
Subscribe to the Product Review e-mail newsletter
As the number of users, applications and demands for business-quality streaming experiences increases, companies may want to outsource parts or all of their streaming media to content delivery networks. When to outsource your streaming media application and network management is the first question you should ask. Businesses most likely to benefit from outsourcing are those in which the target audience is physically dispersed and logically separated from the network on which content (live or on demand) originates. Businesses that have periodic live-event broadcasts for large (more than 1,000 desktops) internal audiences, in which the application requirements exceed the data network's capabilities, are also better off outsourcing than attempting to deploy internally.
To determine which streaming media provider is the best depends on your company's size and needs. Assuming that we were working on behalf of a large company with a set of conventional applications, we developed what we believed a service provider would need to meet these demands. The ideal provider has to be successful in four technical arenas: multimedia application services platform; network access and transport; data center facilities and hardware; and intelligent software for streaming media delivery and network management. The company should also stand out in three business areas: customer lifecycle service; pricing; and corporate and product management.
We narrowed the field of vendors from 60 down to nine. From those finalists, Activate, Digital Island, iBeam and Yahoo Broadcast accepted the ground rules of the "race" and made it to the finish line. AT&T, Enron, Globix and Intel declined our invitation because of a shortage of internal resources to dedicate to the project. Akamai started the race, but dropped out.
Activate, Yahoo each win
|
Blue Ribbons
We conducted two studies in tandem. We looked for the best provider of live-event broadcasting services, and the best for on-demand applications (hosting, content delivery and management). Yahoo Broadcast wins the Blue Ribbon Award for live-event broadcasting services, with Activate coming in a close second.Activate wins the Blue Ribbon Award for the on-demand portion of the shootout, but likewise, Yahoo Broadcast was only a split second behind the winner.
We were impressed with Yahoo's account management team, the strength of its application services platform and network performance. On the on-demand side, Activate impressed us with its network performance and a well-polished proposal.
Start your engines:
Live-event performance
We based our final score for the live event services on three sources of data: network performance; analysis of proposals submitted by the participants (see our review TOC page); and the first-hand experience of working with the participants to plan and produce a complete live event.We took the live-event network performance measurements on Sept. 26, from noon to 1 p.m. Eastern Daylight Time. Streaming media network operation centers received live content from the Echostar Dish Network to identically configure receivers, convert the original video to RealNetworks' G2 format and stream video and audio packets to 10 Keynote Systems-owned and managed measurement agents at 15 frame/sec using 100K bit/sec of bandwidth.
Users always seek a responsive network environment, but after the announced start time of a live event they are intensely aware that content is streaming live, whether they're watching it on their screens or not.
Unfortunately, because this was a one-time event, we could only sample one "time to start" for each of the four service providers, so this particular data subset was too small to receive significant weight in our study.
Packet loss and recovery, bandwidth usage, and frame data were considered more important than time to start in our analysis of the live-event performance data. With this said, bandwidth usage is the least important of these three because during a longer stream such as the one-hour broadcast used in this event, three equally good conditions may exist in the network. First, the streamed bandwidth remains high enough that the buffer in the network and client keeps growing until the remaining event sits in buffer until the event plays out, at which point bandwidth drops to zero. Alternately, the server may alternate between rapid buffer filling and quiescence. The third scenario is one in which the bandwidth usage is constant over the length of the event. Because these three scenarios are compatible with successful live streaming, bandwidth usage or bandwidth variation fail to be good indicators of performance.
Our live-event performance then rests most heavily on the packets lost and frames received per second (for video) data. Chart 2 shows audio and video packets lost expressed as percentages of the total packets sent, averaged over all 10 measurement locations.
In the case of iBeam, the agent in Philadelphia caused significant anomalies in the data. When we removed the Philadelphia agent's data from iBeam's data set, its performance matched Activate. To put the video data in context, remember that roughly 20 packet/sec of video are shown, so that even at the highest average of video packet loss, only about one packet per 16 seconds is lost on average. One missing packet might noticeably damage several video frames, but on the whole, the average frame was quite unlikely to be damaged.
Video frame rate is key
The video frame rate displayed by the player was the most useful predictor of perceived quality of video. While each vendor encoded its stream at 15 frame/sec as agreed, the average frame rates experienced by all agents was higher than 15 frame/sec for all networks and differed substantially from one provider to the next. This is in part a matter of the players interpolating new frames between frames received under high-quality conditions to produce the best possible experience. The agents also experienced different rates of frames dropped. Unfortunately, it isn't possible in this data set to determine whether the network or the client was at fault for the frames dropped.Performance data is challenging to collect and analyze. We were confident that the methodology used in this test produced the most level playing field possible and that the measurement agents captured performance metrics accurately. We found that the Activate and Yahoo Broadcast networks were good and performed significantly better than the other networks studied. We gave Yahoo Broadcast the highest score (8.5 out of 10) because the network introduced the lowest errors and the fastest initial buffering on the stream (6.7 seconds). Overall, we found Activate and Yahoo Broadcast networks more than adequate, but believe there remains room for improvement.
Lap 50: RFP responses
A customer's first impression of a service provider often comes through contact with a salesperson. Following the introductions, a full-length request for proposal (RFP) that describes the company's goals for streaming media should be submitted to at least three service providers. Responses to RFPs should be prompt, complete and emphasize how a company's strengths match your company's needs.We submitted two separate scenarios and asked the participants to submit RFPs. Each proposal was evaluated on completeness in addressing key areas of concern, such as the application services platform, network infrastructure/architecture, streaming media management technologies, and business issues such as pricing and service-level agreements (SLA).
|
The results of this study reveal only the conditions experienced by the
study team, and the networks' conditions in 10 metropolitan areas as of
September and October 2000. Although massive improvements in a provider's
customer service centers, pricing, and corporate and product management are
unlikely in the course of a few months, the technical capabilities (access
to high-bandwidth, low-latency networks, data centers and sophisticated
streaming media network management software) can advance rapidly. We
believe our findings are accurate, but encourage our readers to repeat the
study for themselves before signing a multiyear commitment with any
streaming media network service provider. In other words, your results may
vary depending on the applications you have and the network conditions
within the company, the service provider infrastructure and your target
audience.
Activate's proposal was well-written, well-formatted and gracefully highlighted the company's strengths, such as its international reach and number of strategic alliances. Partners include backbone providers and complementary technology providers, such as Senada.com for attendee invitations, and Engage for advertisement insertion capabilities. Although network-monitoring practices were well explained, the proposal did not have a sample SLA as requested.
Digital Island's proposal was informally written, and while it covered the network architecture in detail, there was an assumption that the customer would need numerous supporting documents that further described the network's performance.
One of the important supporting files we found above average was Digital Island's sample SLA. It contained extensive detail and specifics about the consequences of the service provider's failure to meet service-level standards.
The last section on pricing offered us too much depth. Fortunately, the company's response included a separate pricing summary, but we don't believe the price differed sufficiently from competitors to overcome the weaknesses of its proposal.
IBeam's response was well-written and well-formatted, and covered many subjects in appropriate detail. The comprehensive explanations of the network architecture were helpful in understanding the company's ability to circumvent the Internet for high-quality content delivery.
We were also impressed with the capacity of the company's infrastructure (It has the ability to serve 500,000 simultaneous users) and complete application services platform. The pricing tool was complicated, but well-summarized in the proposal.
Yahoo Broadcast's proposal was pleasing to the eye and easy to read. It responded to all topics of interest comprehensively and added value in many places. In particular, this proposal went above and beyond simply responding to the RFP's questions it demonstrated that the sales team could use many of the parent company's assets, such as special interest mailing lists, to expand audiences. According to the company's proposal, its current capacity exceeds one million simultaneous viewers, making it the largest of the networks tested.
Reporting was explained in depth, while pricing followed a simple formula. Having seen complex and simple price quotes, we believed the proposal should give a simple formula and it is the customer's responsibility to ask for details if the proposal does not sufficiently explain pricing.
Lap 200: First-hand experiences
The most demanding element of the live-event broadcasting was first-hand assessments of each service provider's customer service staff, including event producers, content acquisition engineers and the application services platforms. We made detailed observations on these points during the event preparation phase the day of the event and in the hours and days following the broadcast.
Activate and Yahoo Broadcast received nearly the same final score in this portion. The two demonstrated mastery of the pre-event phase and performed well during the events, but excelled in different aspects. Both companies had well-prepared and proven approaches (for example, an interview with the customer/presenter using a scripted work order input form) to understanding the specific event requirements. The invitations to the event were sent in an orderly fashion in both cases. While we found the Senada.com system for audience management attractive at first blush, it required an extra step before registration with the Activate application server (actually the Mshow.com engine) could begin. The Yahoo Broadcast guest registration and audience capture was methodical and clear. The Yahoo team had the registration and pre-event site mocked up using the Perey.com look and feel, and approved more quickly and professionally than any other provider through this stage. This indicated a tighter integration between Yahoo's design team and event management.
Activate's and Yahoo's producers were relaxed and made the arrangements seem commonplace. During the event, everything went as planned. On-screen interactivity was supported and responsive in both cases. After the event, Yahoo Broadcast sent a thank you e-mail to the participants, as requested. Both these application services platforms had the most robust reporting tools of the five we saw, for real-time view during the event, as well as historical data.
Yahoo Broadcast finished with a narrow lead over Activate because it was evident that all of the components of the applications service platform, including audience acquisition, management, slide synchronization and reporting tools, were in-house, tightly integrated and well-managed.
Digital Island performed well in the pre-event phase, demonstrating a high level of customer contact during technology testing, requesting adjustments to lights and other local environmental factors to improve the quality of the signal acquisition. In general, the team appeared less likely to direct and was more likely to follow the customer's lead. The design of the interface seemed less familiar to the production team, perhaps indicating a need for more in-house competencies in this area.
While Digital Island's inclination to follow more than lead works for an experienced presenter, it could reduce a novice presenter's sense of confidence. The run of show developed prior to the event and the producer's persistent contact with the presenter's location via telephone ensured that once live, everything went smoothly. The post-event report was limited compared to other platforms, and did not match the ones offered in the written proposal.
IBeam's event producers seemed comfortable with every stage of the production. The pre-event orientation was casual, but unfortunately, the guest registration page was repeatedly found to have broken code. There was never a request for pre-event audio testing, although we did test connectivity and video signal acquisition 24 hours prior to the live event. For reasons that remain unclear, the presenter is required to run ICQ client software. We were told that this was the best way to obtain the results of polls in real time. We repeatedly asked for a tour of its "Dashboard" application, and were deflected each time. Finally, during the post-event debriefing, it was made clear that the application used by producers is highly customizable and tailored for repeat customers on a case-by-case basis. For reports, we were disappointed to learn that the reporting system was being overhauled and was not available for viewing. Shortly after the event, we received an Excel spreadsheet with a limited data set.
Live-event finish line
Having three components to measure let us assess the technical and business qualities of these vendors. All were certainly prepared to host and stream our live performance content, and had ample expertise and tools to produce routine live events (live event first hand). As the number and complexity of events increases in the near future, the bets are on the top two contenders because they have standardized their procedures, and automated processes where possible. In our experience, Yahoo Broadcast is the furthest in this process and earned the score of 8.2. And although it is integrating the technologies of third parties in its service, Activate is not far behind and received a 7.8.
Next race
On-demand services
Our on-demand testing score was based equally on the performance metrics and our analysis of participant proposals. For the performance measurement portion, we solicited the assistance of two companies: WatchIT.com produced original content, and Streamriver Networks, a division of Media100, encoded the stream in two file formats (RealNetworks and Windows Media Technologies) at 40K bit/sec, 100K bit/sec and 300K bit/sec. The participants were asked to host six identical files in their networks and provide us with the URLs. Keynote Systems measurement computers requested the files from the URLs in each network and streamed them while sampling data one time per hour for 14 days, for a total sample of 336 datastreams per network.
Revving up:
On-demand performance
We analyzed the data collected and based our scores on four parameters: the start-up time (seconds); the percentage of clips that had rebuffering delays; the average length of rebuffering delay per clip (seconds); and the consistency of packet delivery (packet performance).Surprisingly, the time to start did not vary significantly for higher bandwidth data. In contrast, we were not surprised that, in general, the times were lower (faster) than for the live event broadcast because in on-demand scenarios, the server has the ability to customize the buffering conditions it sends to a particular client. In this data set, we found that iBeam consistently had the lowest start-up time.
During a file's playback, if the network introduces a significant period of time when the player does not detect data, the server and client negotiate to refill the local computer's frame buffer. The effect on the user's experience is detrimental because there is a pause of variable length during which there is a blank window or the last full frame received is frozen in the window. The percentage of clips that had rebuffering delays hurt the performance of Digital Island and iBeam.
Next, we observed the average length of rebuffering delay per clip. These delays are averaged over those clips that had delays. So if 50% of clips had a rebuffering delay of 10 seconds, we would report 10s, not the average rebuffering delay of 5 seconds for all clips (see Chart 6, below). We found that clips on the Digital Island network took substantially longer to recover from a buffering problem, while the Activate network had the shortest average rebuffering time at two of the three data rates.
Our fourth and final assessment of the network performance was based on packet performance. This is the number of video packets received by the client per second on average at a given data rate.
At all data rates, higher is more likely to produce a better audio and video experience. We found that Digital Island and iBeam streamed fewer video packets on average than Activate and Yahoo Broadcast.
This is not because the stream was encoded using fewer packets (all vendors had the same files loaded), but rather because of inconsistent packet delivery, especially as measured by certain agents. This packet "thinning" was observed most dramatically in video on the Digital Island and, to a lesser degree, in the iBeam network.
These data and analyses led us to conclude that of the four participants, the Activate and Yahoo Broadcast networks would produce superior viewer experiences in on-demand applications. Activate finished only 0.5 points ahead of Yahoo Broadcast, while we gave iBeam a score of 6 and Digital Island a score of 5.
Lap 450: RFP responses
The proposals we received for the on-demand RFP were similar in style and content to the live-event proposals. The only place we found significant differences was in the pricing. In this respect, Digital Island was the most confusing, and perhaps the most economical service, because it charged on a usage basis, not a flat fee. Digital Island also mirrored its live event response in that valuable supporting information was sent in separate files. In contrast, Yahoo Broadcast's quote came in using plain English, listed all the options included and did not appear to be priced on a per-viewer or total-traffic-delivered basis, although this may have been explained in the fine print.Activate's response for on-demand services was well-prepared, and the pricing was well-explained. It was particularly strong in explaining the value-added services, such as the numerous signal acquisition technologies for receiving original content from customers and the ad insertion system the company offers.
IBeam's response to the RFP reflected well on the company's sales team, including a complex pricing tool but a succinct summary of the fee that the customer would pay for the proposed services.
On-demand finish line
We believe that on-demand applications are popular to reach global audiences with a consistent yet compelling and memorable message. Based on our measurements and the RFPs, customers who used the Activate and Yahoo Broadcast networks in fall 2000 probably had significantly superior experiences than those who were using the Digital Island and iBeam networks.Our studies revealed that Activate and Yahoo Broadcast are offering well-managed streaming media networks, have high event and viewer capacity, and finely tuned business processes. Whether deploying live or on-demand applications, corporations are likely to have good results with these vendors. If you are planning to do on-demand and live applications, we recommend that you solicit proposals from these companies, as well as one or two other providers, and choose only one content-delivery network for your streaming media hosting and delivery services.
RELATED LINKS
