- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
Network World - The engineers at vulnerability testing tool vendor nCircle spend $100 per month at the coffee shop in the lobby of their office building in downtown San Francisco. But there is one day each month when a trip to the cafe is more urgent than at any other time: Patch Tuesday.
As anyone involved in running a Windows network knows, this falls on the second Tuesday of the month and is the day when Microsoft announces product vulnerabilities and releases patches for them. For the nCircle engineers, it's the start of a very long day that's filled with coffee, work and more work.
"We know we won't get any sleep when it's Patch Tuesday," says Michael Murray, director of nCircle's Vulnerability & Exposure Research Team (VERT), whose engineers rarely go home before 2 a.m. on the big day.
Microsoft last year released 45 security bulletins and patches; 18 of them deemed "critical" (the highest rating) and 19 "important" (the second-highest). October was the busiest and most crucial month, as Microsoft hit customers with 10 patches, seven rated "critical."
As soon as Microsoft releases its patches, nCircle's programmers in San Francisco and Toronto begin unraveling the fixes. It's a race between them and hackers who might be doing the same. The engineers have 24 hours to meet service-level agreements with their customers to determine what has changed in the software and to deliver tests that the customers can use to decide whether their systems need to be patched.
Microsoft doesn't publicize the changes, so that exploits for the vulnerabilities can't be created - not immediately anyway.
Knowing Microsoft's patch schedule lets the programmers mentally prepare for Patch Day. Murray is based at nCircle's Toronto office, but for Patch Tuesday on Dec. 14, 2004, he's at the San Francisco facility.
The day begins on a promising note: Microsoft is 5 minutes early, issuing its patches at 10:05 a.m. Four VERT engineers in San Francisco gather at the company's "war room," a tiny office that's barely large enough to house a table that sits eight people. Nine engineers in Toronto wait for Murray to open the phone bridge, which will stay open all day.
Microsoft has issued five patches, each rated "important." Murray, his PowerBook perched on the table amid cables, two digital projectors and the conference call unit, clicks between the pages Microsoft has posted about the vulnerabilities and patches. He scribbles on a whiteboard the vulnerabilities and their Microsoft-assigned numbers, "041 Wordpad, 042 DHCP, 043 Hyperterminal, 044 Kernel/LSSAS, 045 WINs." He puts a cross next to each of the five to denote that they are all "locals," meaning that they could be exploited via e-mail or by people on the system. Next to the Dynamic Host Configuration Protocol (DHCP) and WINs vulnerabilities are two more crosses, denoting that they are also "remotes" - services that are accessed remotely.
"I'll buy lunch for anyone who writes an exploit for Wordpad," quips Christian Hunt, manager of VERT in San Francisco. Murray assigns teams; all the "locals" will go to the Toronto engineers, and San Francisco will take the "remotes." The teams are scheduled to meet every three hours.
Back at their cubicles just outside the war room, the engineers download the patches onto an instance of the affected operating system that they've created using VMware's emulation software. They grab the binaries from the patches and load them into the IDA Pro disassembler tool from DataRescue. IDA reveals the lines of code written by Microsoft programmers, and nCircle engineers pore over the evidence to pick out the differences in the patched and unpatched software.
The engineers jump back and forth from their cubes to the war room, where Murray and Jeremy Cooper, a senior software engineer, are sitting, swapping notes and advice. Sometimes the engineers will communicate via Internet Relay Chat or cell phones - even though they are in the same building. Other times they un-mute the bridge with Toronto (this is most often done by Hunt, who mercilessly teases his Canadian colleagues). By 11:45 a.m., the tests for 041 and 043 are ready for quality assurance, and Murray orders pizza.