By Lorinda Brandon, Director of Strategy at Smartbear Software
The demands on quality are greater than ever, with so many software applications passing data through shared networks and cloud environments. With more critical systems being developed for healthcare, finance, government and defense, companies are rightly concerned with security and vulnerability issues. They are also faced with the difficulties of uncovering complex issues that would be expensive to set up and test in-house.
So what to do?
One avenue many organizations are pursuing is crowd-sourced bug detection. This is more than just complaining on Twitter or reporting issues to support. There are actual organized, and sometimes highly paid, methodologies behind this. Here are a few of the strategies companies are employing in order to tap into the larger, more varied world that real users represent.
Yawn. Not so new – we’ve been around the beta block a few thousand times as an industry. From managed betas to private betas to public betas… we’ve adopted them all. The benefit of a beta test is that it is typically time-bound and overseen by someone in the company. The downside is that is it time-bound and overseen by someone in the company. What do I mean by that? Well, we all bring our own biases to the table whether we mean to or not. When companies select beta customers based on some criteria and/or assign them tasks to perform, they often miss crucial issues because they limit customer actions and environments.
This is slowly becoming a lucrative business because many companies see the value of “in the wild” testing, especially in the mobile space. You can only create and re-create so many user scenarios in a lab environment unless you have unlimited time, money and resources. Using a crowd-testing service can provide you with much-needed insight into your application quality and uncover some difficult-to-find bugs in the process. But beware – if you want clean defect reports with steps to reproduce and analysis of user impact, this is not where you’ll get that. I prefer to think of crowd testing as a great ongoing feedback loop outside the normal rigor of your in-house testing.
"Find that potential bug and bring it in – we’ll pay you a bounty." That’s the premise behind bug bounties, which are primarily aimed at researchers rather than testers. Having a security breach can be very costly, as any of those companies who have been hacked can tell you. But it’s very difficult to unearth security flaws despite code reviews and security testing – so what companies are focusing on are the potential problems in their applications, otherwise known as vulnerabilities. It’s a win-win for both the researcher, who has the potential to make a lot of guaranteed money from finding a security flaw, to the company, which would otherwise face enormous payouts to consumers and government for security breaches. Hiring the right people full-time to look for these flaws is also a costly proposition without a long-term value prop.
I love bug bashes. Sounds like a party, doesn’t it? And in a way, that’s the idea. A bug bash is a time-boxed testing frenzy where testers are often rewarded with prizes and recognition. Some companies still keep their bug bashes to themselves, turning the whole company into an extended testing organization for a few days in an effort to flush out any bugs that evaded the test team. Others get more courageous and open the bash to their external users, simulating the power of crowd testing but under more controlled circumstances by setting specific challenge guidelines and timeframes. The idea, of course, is to get as much brain power and ‘randomness’ into the mix as possible but within constraints and under the oversight of your own testing organization.
What does this mean to the future of testing?
I don’t see any inherent danger to the future of software testing from these types of strategies – in fact, I think it’s refreshing to see the renewed vigor behind software quality that is prompting people to find more ways of finding and wrangling bugs than they have traditionally had at their disposal. In the testing world, we’ve always known there is a limit to the types and quantity of bugs we can find within a project’s scope – often we’re constrained by the availability of test environments and the limitations of project schedules. Being able to tap into the larger user pool and their real-world environments can only be a boon to the software industry, and helps to alleviate the strain on most testing organizations that are fully aware that they can’t get to every bug.
About Lorinda Brandon, Director of Solutions Strategy at SmartBear
For more than 25 years, Lorinda Brandonhas worked in various management roles in the high-tech industry, including customer service, quality assurance and engineering. She is currently Director of Solutions Strategy at SmartBear Software, a leading supplier of software quality tools. She has built and led numerous successful technical teams at various companies, including RR Donnelley, EMC, Kayak Software, Exit41 and Intuit, among others. She specializes in rejuvenating product management, quality assurance and engineering teams by re-organizing and expanding staff and refining processes used within organizations. She has a bachelor’s degree in art history from Arizona State University. Follow her on Twitter @lindybrandon.