Challenges of software quality assurance

The long view of security strategies for your network.

In this, the third part of an interview about automated software quality assurance (ASQA), Andy Chou, chief scientist & co-founder of Coverity, finishes with some interesting case studies about real-world application of ASQA.

6) Have you ever seen an absolute disaster in the implementation of ASQA? Can you tell the readers what happened (without giving embarrassing identifying details) and the lessons you and your colleagues learned from the experience?

None of the challenges to automated software integrity analysis are fundamental or insurmountable, but sometimes people don't understand the technology, what it can do for them or how to deploy it. This might mean that they make poor decisions when implementing it or helping developers understand how to use it.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

In this, the third part of an interview about automated software quality assurance (ASQA), Andy Chou, chief scientist & co-founder of Coverity, finishes with some interesting case studies about real-world application of ASQA.

6) Have you ever seen an absolute disaster in the implementation of ASQA? Can you tell the readers what happened (without giving embarrassing identifying details) and the lessons you and your colleagues learned from the experience?

None of the challenges to automated software integrity analysis are fundamental or insurmountable, but sometimes people don't understand the technology, what it can do for them or how to deploy it. This might mean that they make poor decisions when implementing it or helping developers understand how to use it.

We recently wrote a paper for the Communications of the ACM 53(2):66-75 on the challenges we faced bringing Coverity and automated integrity testing to market, "A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World." We write about these challenges and the mind-set change it takes to be successful.

Some of the key findings we discussed in that paper about the differences between a research lab and the real world include the following:

• The volume of tests in real-world applications of ASQA is orders of magnitude greater than in the product-development lab. The number of language dialects, programming styles, bugs and false positives all go up when testing real-world software.
• Users don't necessarily have the same perspective as ASQA tool builders; they may interpret error messages very differently from the developers' intentions. The product must provide rapid processing and clear results that can be understood with minimal training.
• Sometimes programmers or system managers actually stop the ASQA tool from testing specific parts of their code, making the results untrustworthy and incomplete.
• Differences in the technical platforms used to generate compiled code may not be compatible with the ASQA tool, including even radical differences in the interface used by the programming teams (such as graphical user interfaces vs command-line interfaces).
• Company policies may forbid even harmless changes to a production sequence, making a specific ASQA tool unusable because of minor incompatibilities with the operating environment.
• Compilers that don't reject illegal source-code constructs as defined by language standards can produce unparsable code that the ASQA tool cannot test – and cause conflict with the programmers who define their code as, say, C++ if it is accepted by their C++ compiler no matter how illegal their constructs are.
• Many sites involving safety-critical systems are stuck with ancient compilers because it's too expensive to recertify the safety-critical software every time a compiler changes version. These old compilers – sometimes decades old – cannot be purchased at all by the ASQA maker and so are very difficult to include in ASQA-tool testing.
• Programmers may insist that the bugs found in their code are not bugs – including for example treating buffer overflows as normal!
• The organizational culture may cause people to dismiss many bugs as unimportant because they don't have any direct effect on themselves.

M. E. Kabay, PhD, CISSP-ISSMP, specializes in security and operations management consulting services and teaching. He is Chief Technical Officer of Adaptive Cyber Security Instruments, Inc. and Associate Professor of Information Assurance in the School of Business and Management at Norwich University. Visit his Web site for white papers and course materials.