Flaws put open source on hot seat

The sendmail and Snort security bugs exposed last week brought front and center the unique challenges inherent in producing and applying patches to open source software.

The bottom line, experts say, is that corporate users should be aware that open source patches can be produced quickly but won’t necessarily come from a trusted source. Also, it is difficult to track software that might need a patch.

“With open source you really have a double-edged sword,” says Dan Ingevaldson, the team leader of X-Force Research and Development at Internet Security Systems, which discovered the sendmail bug. “It’s very open but there is no single point of contact where there is a list of enterprise customers using the code.”

That could foster a disconnect between code developers and users not plugged into mailing lists.

The issue was raised last week with Sendmail, Inc., and SourceFire, which employ creators of popular open source software but also sell commercial versions of the code.

In the sendmail case, code creator Eric Allman was notified of the bug and then he informed companies such as HP, IBM and Sun that he knew had the 15-year-old open source code in their commercial products. Those vendors then developed patches for their own customers.

But Allman says, “on the open source side, you don’t always know who picked up the software. You can get the big companies, but for all the others you just announce the problem in the appropriate places.”

The open source world includes mailing lists and Web sites such as sendmail.org and SecurityFocus Bugtraq.

On the closed source side, a central point of contact, say Microsoft, becomes the flash point for OEMs and other known licensees of products. But the criticism against closed source vendors is that often they don’t respond quickly or at all until hackers release exploit code.

On the commercial side, Sendmail talked to every customer via direct contact or e-mail, Allman says.

The same scenario was true for SourceFire CTO Martin Roesch, the creator of Snort, who pulled no punches in laying out the severity of the security flaw in his code last week.

But Roesch says on the open source side at times it can be difficult to know who is running your code and where.

“It’s an interesting perspective,” Roesch says. “It reminds me of the SNMP bug last year that had people scrambling to find out what systems had the software and were vulnerable.”

In that February 2002 incident, a bug some called the biggest security threat in the history of the Internet was found in SNMP, which runs on everything from switches and routers to workstations and servers.

Historically, the open source community has been quick to respond to problems. Which was evident with open source sendmail, as a patch took less than 24 hours to develop. The commercial version took a week because it had to be tested on every platform, Allman says.

But was everyone using open source sendmail, or products that incorporate the software, aware of the issue? In this case, the answer is likely yes, given that sendmail moves about 75% of the e-mail on the Internet.

But what about lesser-known software and code?

“There is tons of open source code being embedded in all sorts of places, and awareness is an issue,” says Roesch, who adds that closed source and open source patching have their differences.

“We have 8,500 downloads a week of the open source code and Snort is all over the place. It’s in places that I never imagined.”

Roesch says he found out the FBI was using Snort while watching a television news program that showed an FBI computer running an application. Roesch recognized it as Snort.

Experts say it’s imperative that open source software users keep up with community development efforts or with their vendors that supply open source software.

“That is why most commercial customers will get open source from a known vendor that supports the software,” says Dan Frye, the director of IBM’s Linux Technology Center and a former member of President Clinton’s technology advisory committee on open source. “That’s why most enterprise shops don’t develop their own Linux distributions or modify the source code.”

Companies that change the binary files of source code and recompile it change the “signature” of a file and can make patching a nightmare. Once source code is modified or extended, the software becomes a unique application that must be maintained by its creator, which could be a single developer.

“Our tools check software based on a binary signature,” says Mark Shavlik, CEO of Shavlik Technologies, which develops patch management tools for Microsoft products and is working on a Linux version. “If someone extends an open source product and recompiles the code with a few tweaks it becomes almost impossible to track.”

It’s a key issue, experts say, because patching software is becoming a process that requires automated tools and explicit corporate policy.

In the sendmail case, the patch updated three files, but if any of those files had been previously modified in a particular user’s version of the software, the patch might not have worked right, Sendmail’s Allman says.

For instance, Sun has extensions to the sendmail code that are specific to its implementation.

“None of the extensions touched the files that were affected, but they did have to look at this a bit more carefully,” he says.

The issue is not lost on corporate users who have adopted open source software.

The CTO of a well-known online e-commerce site says his site rarely makes any modification to open source code beyond configuration files.

“If we do make changes we do it for the long term and we don’t touch that code again,” says the CTO. The company uses two open source tools to help with changes. One is called Patch, which helps incorporate source code changes into patches; the other is CVF, which provides a revision history. “But we try to stay away from modifications and extensions because the support of those can become a real headache, a real nightmare.”

Despite the differences, the real corporate problems today with patching vulnerable software is that it often doesn’t get done.

That was evidenced in the recent Slammer attack on a hole in Microsoft’s SQL Server for which there was a known patch.

The open source world isn’t any better – a study last year by consulting firm RTFM after the OpenSSL bug hit showed that more than three weeks after the vulnerability was discovered 60% of servers in a test group probed randomly on the Internet still had not been patched. In addition, three weeks after the Slapper worm exploited the OpenSSL remote buffer overflow the pattern of unpatched servers was similar.

“It really comes down to your processes [for remediation],” says Dan Agronow, vice president of technology for Weather.com, which has converted to nearly a complete open source infrastructure over the past two years.

“If you have the processes in place to get patches and get them installed you can be successful in both environments, he says.”