When the sensors feeding into Internet Security Systems' Global Threat Operations Center detected a new XML HTTP vulnerability early last November, attacks exploiting that vulnerability already were occurring across the Internet.
Level zeroes are just one of many new threats pushing vulnerability research labs such as X-Force to their limits, says Gunter Ollmann, X-Force director. For example, using polymorphism techniques, attacks now can change on command to defeat signature detections. Also on the rise is fuzzing, which tests data fields for vulnerabilities by generating and filling them in with a barrage of bizarre text strings until they break.
These new threats are enough to keep members of the X-Force up nights -- literally.
"It's always about 2 a.m., when the bad guys come knocking at your door," says Chris Schueler, who hails from a security operations center background in the Army and who directs ISS' global Security Operations Center (SOC). "The number of calls that bump up in priority that are more severe are increasing so much that I'm worried we might be getting these types of calls all the time."
Schueler was one of the first people to jump into action when the HTML exploit was captured by X-Force sensors. Those sensors were running a combination of protocol analyses and assessment and virus-protection engines, and feeding the information into the X-Force Global Threat Operations Center (GTOC).
A meeting in the war room
It was after dinnertime when pagers went off summoning key X-Force directors and managers from their TVs, dates, friends and pets to meet in a cramped corner conference room barely big enough for six. This is the "war room," down the hallway from the third-floor elevator and past the black X-Force banner on a cubicle wall. Here, Ollmann, who formerly was a penetration tester in the United Kingdom and New Zealand and who has degrees in science and physics, held a meeting to plan strategy.
When a level zero hits, everyone has a job to do. The X-Force database library of 30,000 vetted vulnerabilities is consulted and updated. The R&D team reverse-engineers samples of the malware and builds signatures for the company's Proventia detection engine. Testers run the exploit against potential targets in the lab, which is packed full of vulnerable machines. And the GTOC and SOC share information continuously with R&D and vice versa.
Here, on that cold night last November, a girlfriend did her homework in a fourth-floor office waiting for her boyfriend, who had gathered with several other researchers in what's called the bullpen or the pit. It's a central, room-sized cubicle on the fifth floor where researchers use Linux- and Unix-based systems to run tests against the exploit's binary code and its behaviors and actions. They use this information to update the database and to make signatures against the specific type of attack.
The R&D team, which totals about 60, mostly young, male, über-geek types, keeps canned beer and bottled water in a small refrigerator at the entrance to the cubicle for nights like this. Nearby restaurants know them, Bailey says. "When we come in for dinner, they'll ask us, 'How many this time? How bad is it?'"
Despite the increasingly brutal schedule, Bailey says he loves his work because it's never boring. "You never know what you're going to do when you go to work," he adds.
Neel Mehta, team lead of X-Force Advanced Research and Development, says what keeps him interested in vulnerability research is that he gets to be the first to know about vulnerabilities.
"We get to discover vulnerabilities in software, hardware and embedded devices before the bad guys can take advantage," says Mehta, who studied biochemistry and digital copyright before being hired right out of college five years ago, after he was recommended for the job by a friend who is an X-Force researcher in Australia.
Now the trend is away from worms and toward the increased exploitation of browsers to install botware for the purpose of making money, Mehta says. "We're still finding new wormable vulnerabilities, but we're not seeing the spread, because most of the activity is in ones."
Blackmail also is a common purpose of botnets today, Mehta says. He has had clients, big financial organizations, who've paid money to Internet extortionists to stave off denial-of-service attacks that botnet operators have threatened to unleash, he says.
As for future threats, Mehta says he's seeing more activity in the mobile threat landscape, particularly on the Symbian platform. He expects to see financially motivated attacks against the platform and proximity-based services, such as Bluetooth, during 2007.
To report or not to report?
Vulnerability-information management is full of pitfalls. The X-Force and other vulnerability researchers have been accused of releasing too much information, and at the same time, not enough, says Pete Lindstrom, senior analyst with the Burton Group.
Company policy says that all medium-to-critical-impact vulnerabilities that X-Force members discover are reported directly to the vendors, says Gunter Ollmann, director of X-Force. It would help if vendors, particularly small and midsize ones, had an intake point and methodology, which most of them don't, he says.
Typically, the X-Force releases public alerts only for critical vulnerabilities and only if there's a patch or fix available. On the other hand, Ollmann says, there are times when vulnerabilities found at customer locations are not reported at the request of the customer.
The system isn't perfect, however. For example, Cisco and ISS were involved in a controversy in 2005, when ISS researcher Michael Lynn was slated to make a presentation at the Black Hat conference about a vulnerability in Cisco routers that if exploited, could have taken down most of the Internet.
At the last minute, ISS ordered Lynn to scrap the presentation, but he quit his job and delivered the presentation anyway, causing a major brouhaha.
The vulnerability lab is segmented from the rest of the network, just as the SOC has a completely separate network that's locked behind a biometrics-protected door in the basement. And for good reason: In addition to the Unix- and Linux-based machines used for testing in the pit and in individual researchers' offices, the team has at its disposal 1,000 real and 600 virtual machines of all makes and models -- even printers and IP-enabled cameras -- running complete with their native vulnerabilities.
It is on these machines that researchers study the way a given attack will behave, and test for new ways to break these systems. On any given day, code for an exploit scooped from a malware bulletin board might be scrolling up the screen of a Linux-based testing system inside the pit.
By studying the source code and how it acts in the vulnerable devices in the lab, X-Force vulnerability researchers averaged about 200 new finds a month last year, accounting for 2,400 of the 7,000 vulnerabilities entered into the X-Force database in 2006.
Self-discovering vulnerabilities might be viewed as a conflict of interest for companies in the business of vulnerability management, says Pete Lindstrom, senior security analyst for the Burton Group. Ollmann argues the reverse is true.
"We're stress-testing hundreds of protocols through our parsing technology to find entire classes of weaknesses that we can protect against," Ollmann says. "Naturally, some research will lead to new exploits."
Even though preemptive protection had been available in ISS desktop products for two years, X-Force developers still built signatures for the exact exploit as backup in case, say, someone hadn't updated, or otherwise had misconfigured or removed the signatures from their scanning engines. But ultimately, Ollmann says, the entire industry needs to move away from exploit-signature cycles and toward more proactive measures.
"Take Zotob. Before it broke out [in 2005], Snort, the top engine at the time, had released 318 signatures for the individual variants of an attack against Universal Plug and Play that Snort researchers could envision," Ollmann says. "Then Zotob broke out [which took advantage of a programming error in the Plug and Play service used by Windows], and bypassed them all."
Vulnerability discoveries in the lab and through intake points at the worldwide protocol sensors at ISS account for about half of the new vulnerability information posted to the X-Force database each year. The rest is culled from bugtraqs and mailing lists, exploit sites such as MetaSploit and PacketStorm; government reporting agencies, including CERT and the French Security Incident Response Team; hackers' Internet Relay Chat, software manufacturers, and occasionally, the Common Vulnerabilities and Exposures (CVE) dictionary from the nonprofit Mitre Group.
In all, three X-Force analysts monitor 44 sources of information on the World Wide Web looking for new vulnerabilities. They use an automated scooping program to grab information from these locations, sort it against the database and assign categories. Once entries are assigned, they're given 40 fields to populate, such as detailed description, consequence rating (can they gain access or just port-scan?), affected operating system and remedy.
"With vulnerability-information management as a discipline, you're taking this mountain of vulnerability data and trying to make it relevant to everyday IT and the security consumer," says Steven Christey, editor of the CVE.
The CVE has placed itself in the middle of the for-profit vendors vying for the latest vulnerability information by creating common identifiers for vulnerabilities instead of relying on vendors to do this uniformly.
When it began in 1999, CVE sought do some of what ISS is doing, primarily to review and vet new vulnerabilities through an editorial board before assigning CVE numbers. That quickly turned out to be too much work, and CVE abandoned any ideas beyond assigning CVE numbers as identifiers for new vulnerabilities so researchers could have a common point of reference at the onset of new vulnerability events.
"Given the volume of information and the incredible number of vulnerabilities -- 100 to 150 each week -- verification, it turns out, is extremely resource-intensive," Christey says. "Sometimes we may have to go back and change an entry, but that's a far cry easier than trying to do what vendors like ISS are doing."
The deep resources and detection outreach at ISS are what make it possible for the X-Force to stay abreast, and even a step ahead, of online criminals, Christey says.
Michael Rossman, director of global IT services and information security for spice-maker McCormick & Co. in Hunt Valley, Md., agrees.
"When we did our contract update for managed security services in '03, then again in '06, X-Force was a critical factor in evaluation," Rossman says. "The horsepower of the X-Force provides the analytical intelligence, which is the underpinning of making events on the Internet nonevents for protected customers."
ISS manages 10 firewalls for McCormick, which has a footprint in 100 countries. It also manages seven Proventia intrusion detection/prevention systems at key, international ingress and egress points (China, Australia, Singapore, United Kingdom, France, Canada and the United States). And like other enterprises, McCormick runs several non-ISS security devices, most of which are managed through the ISS SOC.
Security begins at home
X-Force networks are under the same kinds of attacks they're trying to protect their customers from -- more, actually, given their brand name.
"We have the normal risk areas as every other global company: network security, compliance, privacy," CIO Helen Berg says. "And, with 500,000 significant intrusion attempts per week against our portals, we're definitely on someone's radar screens."
X-Force networks are protected by their own SOC that manages security for 2,000 enterprise and midsize organizations. Berg's team manages its own security through ISS' managed security portal with Proventia running at key global points of intake.
In addition to its own products and services, ISS, like all other enterprises, runs a variety of security devices and brands across the organization. Information taken off these devices by ISS sensors and translated through the customer portal puts at Berg's fingertips a dashboard ranking of events of significance, which is how she knows of the half million attacks per week.
Berg's policy for events of significance is to disable them automatically or otherwise shut them down, which is carried out at the SOC. "We err on the side of caution, a deny-all kind of policy based on our criteria," she says.
As a result of using its own products and services to protect its network, ISS performs another function: It operates as a living test bed of how ISS products and services work in a live production environment, Berg adds.