The beta benefit

Testing buggy products can give your company a competitive edge.

Current Job Listings

In 15 years at Case Western Reserve University in Cleveland, systems engineer Jim Nauer has been involved in six major beta tests. As Nauer has found, being a beta tester has numerous benefits: The most important is the ability to get your hands on cutting-edge technology to gain a competitive advantage.

"It meant guaranteed hardware compatibility and plug-and-play device drivers, and in those days, there was quite a bit of voodoo involved in getting Ethernet and SCSI cards configured correctly," he says. "You didn't just pull [the network operating system] out of the box and click through a wizard to install it."

The beta product worked beautifully, he says. Unfortunately, Apple decided not to develop it commercially. "It would have been really neat if they'd finished debugging it and it solved some of our issues," he says. "But it didn't come to pass."

As Nauer has found, during the Apple beta and his other more successful tests, there are numerous benefits to being a beta tester. The most important is the ability to get your hands on cutting-edge technology to gain a competitive advantage.

However, be prepared to deal with the frustrations, the resource demands and the possible disappointments that accompany the experience. Not only does beta testing stretch the resources of IT staff members, but it also can try their patience, as they are, after all, working with buggy products. Vendor approaches to beta testing can range from very organized to haphazard, and your own testing and bug-reporting process has to be rigorous.

As Nauer found with the Apple beta, there's no guarantee the product will ever hit the market. Still, Nauer says Case Western came out a winner. "It enabled us to get our hands on NetWare 4.1 code before we had the PC native version," he says. "We got knowledge and experience we were able to apply elsewhere."

In fact, other beta testers report that even when things go wrong, they still feel the pros outweigh the cons. Reasons include early access to new technology that can solve long-standing problems, the opportunity to influence product development and a direct line to code engineers.

"You're doing it because you have a vision; you want to beat the guy down the street and improve your revenues quarter over quarter,'' says Louis Barton, executive vice president at Cullen/Frost Bankers in San Antonio.

Secrets to successful beta testing

The secrets? Don't expect beta testing to be a free ride, and be prepared to jump into it with both feet, says Mark Moroses, senior director of technical services at Maimonides Medical Center in New York.

Advice for beta testers

Don't underestimate the time commitment

Beta tests are not something to enter into lightly. You may need to put lower-priority projects on hold or delay other projects by a few days or even weeks. "It's a strain because you still have to get the other work out, so you're working lunches and overtimes," Barton says. So if beta test is low-reward or other high-visibility projects are going on or you don't have a test environment prepared, he says, don't do it.

Negotiate everything you can get upfront

If you want the names of support people or other terms of agreement, make sure you do it upfront and in writing. "You can't just pretend your previous agreements don't exist anymore,." Barton says. "It has to be done legally and in an organized manner." And while financial incentives can be issued, be sure you get a commitment before you go into testing, when you still have leverage, he says. l

Apply yourself

When you're applying to be a beta tester, fill out the form completely, says Michael Fine, director of client services for CenterCode. "We get forms where they don't give data on what equipment they have to test with," he says. "And people who cant spell are less inclined to be chosen." Abide by the confidentiality rules, he says, and don't expect a miracle. "Don't set overly high expectations on the product before you get it," he says. "This is beta; it's being tested. Go in with an open mind."

Moroses has been involved with several beta tests, and while he says his team is very accomplished at the process, it's not something he'd turn into a hobby. "We only go down the [beta test] road if we believe the technology is superior," he says. "If it's a tie, we prefer to work with an established platform with less bugs and support issues."

Moroses counts just one time that a beta test came close to being a failure. He was asked to test a thin-client version of a product he was already using in-house. Unfortunately, the beta version wasn't ready for prime time.

"As our debugging process went on, it became apparent that their approach needed to be completely redone," he says. His team pulled the plug on the beta effort and submitted feedback to the vendor. But Moroses still contends the experience wasn't a waste of time because it gave the team deeper insight into the technology's capabilities.

To survive a beta test without the negatives outweighing the positives, Moroses says, your IT organization has to start with several characteristics, including a strong problem-solving and partnership mentality. In fact, his group is often approached by vendors to do beta tests because of those traits, he says.

"They're looking for certain types of folks," he says. "Our engineers and people in the IT department don't say, 'I have to call the vendor' when something doesn't work - we spend time finding the core of the problem."

But if you don't manage that effort, you can drown in it, Moroses says. "You have to stay cognizant of where your entire portfolio of projects and upgrades is and what people's workloads are," he says. One management tip that he offers is to conduct just one beta test at a time. "If we're at the end of one, we might start up another, but we wouldn't do three or four at a time," he says.

It's also important to establish upfront what you expect to gain. For instance, Moroses insists on getting the names of the software developers he'll be working with, not just a support phone number. He demands 24-hour turnaround time on issues that arise when the tested product goes into production, as well as senior support personnel on-site.

"You have to be careful whom you partner with," he says. "Not all vendors will voluntarily suggest bypassing the normal support center."

Moroses also negotiates a price break on the product or better software maintenance terms. "We tie it to an economic benefit," he says. "We end up making the product better anyway, so we might as well get a financial reward for it."

Beta testing can be a drain

The resource demands of beta testing made Barton think twice before agreeing to test the Cognos 8 business-intelligence system. "It was a balance between the commitment of resources we'd have to apply and what we'd get back," he says. "If you're going to do it right, you have to really do it and put your best people on the project."

Barton is not alone. "Resource constraints are the No. 1 challenge for our customers," says Faiyaz Shahpurwala, vice president of the data-center practices group at Cisco, who says tests can average out to be a three- to four-month effort. "There are times when we're not able to complete a test plan on time, and we have to keep pushing on our side."

However, because Barton's $10 billion financial-services conglomerate was making full use of other Cognos products, and he planned to eventually move to the new business-intelligence system, he decided beta testing would be a good way to prepare for the implementation.

Barton says Cognos was an "extremely organized and thorough" partner in the beta test, which is not always the case. "It wasn't, 'Here's some software you can install so we can get some feedback,'" he says. "There was a plan, a structure, a schedule, names of people to support us, good communication and objectives, which was a new experience for us."

Preparing for the beta test required several phone conferences, Web conferences and e-mail communications, after which a few Cognos engineers came on-site at Frost for three weeks. The entire project lasted three months and required the full-time effort of two to three IT staff, plus some support staff and business users.

But no matter how organized Cognos was, Barton says the effort would not have been successful if his own team didn't have a structured approach to testing, complete with test-control forms that recorded the objective and expectations of each test and documented what happened during testing. "When you get an error, you have to make sure you can document it and communicate about it thoroughly enough to the vendor so they can fix it," he says.

While Barton says the beta test prepared his team to implement Cognos 8, there were some downsides to the experience. "Software always has bugs, but beta software will have more bugs," he says. The time involved in reporting on those bugs, waiting for a fix and then testing it again can be a source of frustration. "It's a drain on resources, especially when everyone already has a full-time schedule and full workloads," he says.

The testing period is intense. "Testing is an art," he says. "Many conversions fail because of insufficient testing or the wrong types of tests, which takes planning and discipline and procedures and documentation."

But the benefits outweigh the risks

If you plan right, Barton says, the downsides are offset by the rewards, one of which is a head start into new technology, which can mean a competitive advantage. "The hands-on personal training reduces the learning curve for support staff, business users and everyone else involved," he says. "It's hard to quantify, but they're real when you start using the product."

John Denardo, CTO for Eagle County, Colo., agrees. As an early adopter of storage technology from LeftHand Networks, Denardo has beta-tested new products and configurations for the vendor. "I'd be sending my guys out to train otherwise anyway, and this way, we get more detail," he says. "Having access to the engineers working on the code is important."

Nauer says beta tests give the university a big leg up in terms of its deployment plans. For instance, it has been able to make an early move into storage virtualization by beta-testing Cisco's MDS Series switch in 2002. The university already had a small storage-area network, but it wanted to migrate to iSCSI technology, which MDS enables. "Cisco brought to the table things we didn't even know existed," Nauer says, such as intelligent applications at the switch level. "We don't need it today, but it's absolutely strategic for us to move in that direction."

Beta tests can give companies a chance to influence product development. "You're providing feedback that will help them make money, so they'll listen and respect your input vs. if you went blindly through your sales rep," Barton says. This depends, of course, on the tester's ability to truly enter into a partnership with the vendor.

Getting leading technology on-site is also a boon for recruitment and retention efforts. "If your tech guys can put on their résumé that they've worked with cutting-edge technology, they're less likely to seek other employment," Moroses says.

Don't forget the financial incentives. A couple of years ago, Michael Fine, director of client services for Center Code, helped beta-test a high-end security product, which ended up providing the user with $20,000 worth of product advice for free.

But while lower software costs can be a benefit, experienced users warn that it can't be a driving factor. "You can't go into beta testing because it's a good way to get some freebies - that's the wrong motivation," Nauer says. "If you do, one or both sides won't be happy."

It's better to approach it as business engagement. "If you treat a beta like it's outside of your core mission, you'll be disappointed. But if you treat it like any other business project, you'll get a good result," Barton says.

Brandel is a freelance writer in Newton, Mass. She can be reached at

Learn more about this topic

Microsoft debuts new Explorer beta


Corporations finally getting hands on Microsoft's Vista


Google News emerges from beta with new features


Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT