Analyzing software bugs to death

Professor's project relies on users: 'There's always more bugs than engineers.'

Ben Liblit wants to get the bugs out of your software, but he really could use your help.

His Cooperative Bug Isolation (CBI) project, with the aid of a few thousand users who run "instrumented" versions of e-mail clients, music players and other software, is designed to help developers pinpoint the causes of bugs, often down to a specific line of code.

Cooperation is key because "there's always more bugs than engineers," says the University of Wisconsin assistant professor.

The CBI instrumentation technology records bits of data about a program as it runs and keeps track of whether it ran successfully or crashed. Once the program has ended, this information is sent to a server for analysis. Liblit has configured several freely available versions of programs that send data to his server, and they are listed as bug isolation downloads on the project's Web site. Developers can instrument other software and have the data sent to their own servers to be analyzed.

Passwords and other specific information are not captured, but Liblit acknowledges that by the nature of the software "there is some information leakage. . . . It's very small, but nevertheless it is nonzero." Although no security flaws have been found, an instrumented program theoretically could send sensitive data such as encryption keys to Liblit's server. The program would have to be structured and instrumented in a very specific way to do this, and a developer who encountered this issue could avoid it by not instrumenting encryption routines. Still, IT managers may be leery of a program that phones home with even a little of their data.

Liblit says running a program a few thousand times gives him and other researchers enough data to compile, statistically analyze, and compensate for false positives and negatives; and to correlate its bugs to certain pieces of code. Users haven't supplied enough data for him to use - the University of Wisconsin pegs the total number of runs at about 3,000 per month - so he finds bugs by running instrumented programs in the lab.

In one paper, Liblit and other researchers explain how they use this technique to solve a crashing bug in a Linux-based calculator called bc. This bug occurs sporadically, which makes it "inherently difficult to fix," the paper says. After running the program 3,051 times and analyzing the data that Liblit's instrumentation provided, researchers pinpointed the issue to one offending line of code. Says music player Rhythmbox developer Colin Walters on a Red Hat mailing list, "His work has helped me a lot with Rhythmbox - certainly he's found some important bugs. . . . It's a really well done project - they have it so users can easily opt out . . . and even include a tray icon [to show that the instrumentation is collecting data]." (See this discussion)

Unlike Microsoft's somewhat similar Dr. Watson utility or the TalkBack software used to document crashes in Mozilla Firefox, Liblit's software records data from both successful and failed runs. In addition, it collects data while the program runs, not only when it crashes. According to a 2004 post on a developer mailing list, this method lets the software catch bugs that software like TalkBack might have difficulty analyzing. (For example, if memory is corrupted in the crash, after-the-fact analysis of that data might not be very helpful.) "There are cases where the crash is so severe that it's impossible for [TalkBack] to kind of reconstruct exactly what's going on" says Chris Hoffman, director of special projects for Mozilla, "[but] generally, those have been very few." Hoffman adds that TalkBack's method of capturing data makes it "lightweight and nonintrusive."

Although Liblit's instrumentation is used only to detect bugs that cause a program to crash, he says it could be used to detect any issue that could be labeled good or bad by a computer. Liblit adds that the developers of the open source graphics program GNU Image Manipulation Program (GIMP) have suggested a "smack the GIMP" button to let users indicate when they encounter a bug.

Although the program can't otherwise detect that there is a problem, pressing the button would register a bad run with the system in the same way that a crash would. Liblit is confident that with enough data, his analysis software could find a correlation between these indicators and the data samples that his instrumentation collects.

Users of the theoretical "smack the GIMP" feature might cause problems when they encounter bugs but don't click the button, and when they "click it because they're in a bad mood," Liblit says. Nevertheless, he's confident his analysis can compensate for such false positives and negatives. "In a mathematically well-defined way, you can actually build a model of uncertainty into the system," he says.

With a few other researchers, Liblit began writing papers on statistical debugging as a graduate student at the University of California at Berkeley in 2003. Liblit has also met with Mozilla to discuss statistical analysis of Talkback data. In addition to his academic work, Liblit mentions on his Web site (which features a "goofy self-portrait") that he spent four years as "an actual grown-up with a real job as a software engineer," experience that he says helped him understand the needs of programmers who use his system.

He realizes most programmers don't have time to implement a complicated bug isolation system, so he has made his software easy to use: Programmers take their source code (currently, only C under Linux is supported) and put it through a modified version of the popular, open source gcc compiler. This compiler adds the instrumentation necessary to capture data and send it for analysis. Liblit says he hopes programmers think "Hey if I use this tool, it makes me look good." "Once you've done that, you've got them hooked," he says.

Liblit provides instrumented versions of popular open source programs such as Novell Evolution (an e-mail client and contact manager similar to Outlook), Gaim (an instant messaging program), Rhythmbox and Spim (a CPU emulator used by computer science students). All his software is licensed under the open source Berkeley Software Distribution license, which lets users and developers modify and redistribute it freely. "I have gotten tremendous mileage out of the [open source software] community" he says. "It would be incredibly hypocritical for me not to release my own stuff into the open as well."

According to Liblit, the performance hit caused by the instrumentation varies depending on the program, but he works to keep programs running smoothly while still providing useful information. Word processors and similar programs "spend most of their time waiting for the user to find the 'J' key," he adds, so some performance degradation may not have a noticeable effect.

Versions for interpreted and compiled Java are in the works, as are other additions that Liblit says he'd rather not talk about yet. Also, he has spoken with several companies that have considered using his technology to test their software internally. Although he isn't allowed to name any of them, a University of Wisconsin page mentions that Microsoft and IBM have shown interest.

Liblit says most of the people who use his instrumented builds do so for "geek appeal" or because they believe in open source software. He says it hasn't been as widely adopted as he had hoped, which is an issue because he needs thousands of data submissions to find bugs in a program.

Still, Liblit continues to improve the software, working on better tools for analysis and improved instrumentation, and plugging away toward his dream of "instrumenting the world." In 2005, the Association for Computing Machinery presented Liblit with its "Doctoral Dissertation Award" for his paper on the topic.

Even if you don't use Liblit's software, you may well end up using a program that he and his team have helped fix. The project provides hope that one day, a user of a buggy but promising open source program will reach what some might call the Holy Grail of user-friendly statistical debugging: smacking the GIMP.

Got an idea for A Wider Net story? An offbeat technology industry-related topic? A fascinating personality we should profile? Contact Executive News Editor Bob Brown at bbrown@nww.com

Learn more about this topic

A Wider Net archives

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Now read: Getting grounded in IoT