The importance of vulnerability research

Experts debate the hows and whys of security testing

WASHINGTON – Testing in-house and vendor-built software for security holes should be an enterprise priority, said a group of vulnerability research experts speaking on a panel at the Gartner IT Security Summit held here this week. But Rich Mogull, the Gartner analyst who hosted the panel, questioned how practical it would be for companies to dedicate the dollars and resources required for this testing.

Thomas Ptacek, founder of application security consulting firm Matasano Security, defined vulnerability research as analyzing software for holes that attackers could take advantage of before the product is deployed, using techniques such as reverse engineering and source-code auditing. Software vendors and many enterprises have teams of engineers in house to perform this testing, or rely on third parties such as the panelists’ companies that specialize in finding vulnerabilities.

The benefit of this testing is being able to avoid the damage an attacker could cause by fixing software problems before implementation.

“If you don’t find the problems, someone [else] will find the problems,” said Chris Wysopal, co-founder of Veracode. “If you leave crumbs on the floor the ants are going to show up. That’s a huge liability … for your company.”

For software built in-house, vulnerability testing should be part of the software development life cycle, not an afterthought, Wysopal said.

“Threat modeling to find out what are the weakest parts and easiest attack vectors [of an application] is what people should do when designing software; you find the weak points through threat modeling then start reverse engineering,” he said.

Simply using tools that scan for vulnerabilities is not enough, the panelists agreed.

“Scanning tools can reduce the amount of time you spend [analyzing the code] manually, but if you care about the security of the application … you need to go deep and augment the scanner,” says Ptacek. “The place of the scanner is to accelerate testing, but you can’t rely on them.”

Gartner’s Mogull did an electronic poll of the roughly 1,000 conference attendees, asking what level of vulnerability testing they performed at their organizations. The majority said their testing was limited to using commercial scanning tools.

One reason enterprises may not be doing more intense vulnerability testing is because the necessary skills are rare, Mogull suggested.

“It’s a huge skills issue,” conceded Wysopal. “It would be best to have an expert researcher looking at every piece of code out there, but you just can’t find them.”

Ptacek disagreed, saying services such as Web application penetration testing are readily available.

Another panelist, Errata Security co-founder David Maynor, added that any steps an organization can take to find vulnerabilities in software are worth it.

“You’re not wasting your money just because you don’t find bugs,” Maynor said.

Yet the process is still an expensive one, Mogull said, and enterprises can’t be expected to dedicate such time and money to extensively testing every application.

“You have to have the appropriate level of testing [correlate to] the risk of the application,” said Wysopal. “Not every application requires hired experts coming in, or training up a large team. Some applications are not as risky as others.”

Ptacek again disagreed, saying that every application offers entry into an organization.

“Even innocuous applications deployed inside an organization can be lethal” if exploited, since just about every application contains or connects to sensitive data. The one example the panel came up with of a program that wouldn’t pose a threat if compromised was an employee lunch-ordering application.

“Every piece of software is not a Boeing flight-control system where everyone will die if there’s a bug,” countered Wysopal.

Mogull asked what percentage of the application development life cycle budget should be devoted to vulnerability research. Wysopal answered at least 5%, Ptacek said between 5% and 10% - adding that the cost for such testing should actually come out of the quality and assurance budget -- but Maynor said closer to 25%.

“It’s easier to allocate dollars before deployment than fix [a vulnerability] after it’s been deployed,” he said.

Turning to commercial software, Mogull asked the audience for a show of hands if they reverse engineer products they buy from vendors to look for vulnerabilities. Outside of the panelists, few hands went up.

“It would be ridiculous to test [to that level] every single product you bring into your organization. What level of testing is OK?” Mogull asked the panel.

“Put one to two person weeks on any new product and you’ll find stuff,” said Ptacek. “Use a sniffer and look at packets on the wire. You’ll find vulnerabilities, and you’ll gain control over how they’re going to be fixed.”

Learn more about this topic

From Whac-A-Mole to chess: The maturing of security

06/04/07

Editors' Picks
Join the discussion
Be the first to comment on this article. Our Commenting Policies