Big-name companies like America Online (AOL) and Adobe could do a better job of writing secure software, according to a recent report by two Princeton University researchers.

The researchers took a look at a number of popular applications, including AOL Instant Messenger and Photoshop, and determined that many of them made changes to the operating system that could allow attackers to bypass some Windows security mechanisms. (Read the report - PDF.)

The Princeton team focused on the Windows access control system, which determines what types of things users and applications can do on any given PC. Their conclusion: Many programs ask for too many privileges, opening the door for potential attackers.

"Vendors are making mistakes when they write programs for Windows," said Sudhakar Govindavajhala, a Princeton Ph.D. student, and one of the authors of the paper. "It's worrying that your computer can become insecure on installation of new programs."

An attacker would first need to gain access to a local account on a computer to take advantage of the problems described in the paper, Govindavajhala said. "These attacks are not exploitable over the Internet, but if someone can get a handle of your machine, then one can do interesting things," he said.

After years of focusing on Windows, attackers are increasingly targeting the software that is running on top of the operating system, according to the SANS Institute, a training organization for computer security professionals. SANS lists instant messaging applications, media players and backup software among the most critical areas for new security vulnerabilities.

Another Princeton computer scientist who is familiar with the paper said that the research shows just how widespread these "privilege escalation" problems really are. "For the average user, it's a reminder that software applications can open security holes and that application vendors do make mistakes that can cause risks for users," said Ed Felten, a professor of computer science and public affairs. "No application should be considered completely safe."

The MediaMax copy protection software used by Sony BMG Music Entertainment was recently discovered to have this kind of privilege escalation flaw, according to Felten. MediaMax's producer, SunnComm, has since patched the problem, he said.

The IDG News Service is a Network World affiliate.