Microsoft patches, calendaring seem to be biggest issues
Scott Metzger wasn’t taking any chances when it came to the earlier than usual daylight-saving shift that happened on Sunday, but even with extra staff on hand and system changes completed a week ago, he’s not resting easy until he sees how things play out over the next few days.
“It is too early to tell if there are … issues on the horizon, especially since we send and get data to and from a large number of third party sources,” says Metzger, CTO at consumer credit management firm TrueCredit in San Luis Obispo, Calif. “While we have been told they are all on schedule with their updates, only time will tell.”
A survey of a handful of IT managers on Sunday showed that most have feelings similar to Metzger’s. While changes generally went smoothly, IT executives say they are remaining vigilant over the next few days as they watch for issues related to the daylight-saving shift.
That’s because most systems aren’t automatically programmed to move ahead an hour until the first Sunday of April, which has been the first day of daylight-saving time for more than 20 years. Starting this year, as a result of the Energy Policy Act of 2005, the daylight-saving schedule is lengthened, with clocks jumping ahead three weeks earlier than normal and falling back a week later than normal. Legislators supporting the change say it will save about 100,000 barrels of oil a day.
But the change also could still wreak havoc on IT systems not patched or programmed to handle the new daylight-saving schedule. The possible implications go beyond missed meetings and botched schedules to serious glitches within time-reliant applications that are critical to a company’s business. As a result, most IT professionals have been taking a close look at their systems to determine where fixes needed to be made.
Metzger, for example, began preparing for the daylight saving shift in October when he and his team began updating dozens of Java virtual machines, a task that was particularly tricky because of the wide variety of Java Runtime Environments.
“We had our big push last weekend, which required the restart of every production server, database and application to apply all the patches for the new [daylight-saving time] rules, with double the staff we allocate for typical maintenance events,” Metzger says. “I also had additional management staff on hand to make sure all went smoothly.”
The only hiccup Metzger saw was a snafu in Microsoft Outlook Calendar appointments that were off by an hour. That can be fixed through a workaround that users download and install, he says.
While he’s still waiting to see how things shake out this week, Metzger says he’s happy with the way things have gone so far. If he had to do it over again, he says he would likely do things a little more piecemeal.
“I would do the [operating system] patches on a separate weekend from the database and applications,” Metzger says. “It wasn’t a problem for us to do everything at once, but it would be a better way to mitigate risks.”
At Floral Supply Syndicate, a wholesale floral supplier in Camarillo, Calif., system administrator John Balster was able to update most of his Red Hat servers, but found no workarounds for two of the servers running an older version of the operating system. “Looks like we’ll have to live with their clocks being off for three weeks of the year,” he says.
Balster says time on those servers isn’t critical and he’s hopeful his other systems will handle the shift without a problem. But he, too, says time will tell.
“Monday will be too soon to tell what the effects are,” Balster says.
The wholesaler is not open Sunday and it’s the time-dependent processes that occur each evening – tasks such as transfer of daily sales information from stores to the main location, the distribution of the updated customer database back to stores and the backup of the central system – that could reveal problems.
“It’s the Monday evening [time-based] tasks which might have time synchronization issues,” he says. “The results of those won’t be reviewed before Tuesday a.m.”
CareGroup Health System in Boston began reviewing its applications in January and patching Windows XP, Outlook and Exchange in accordance with Microsoft’s recommendations. It patched about 500 BlackBerries wirelessly.
“All has gone well today,” said John Halamka, CIO at CareGroup, late Sunday. “Our 146 mission critical applications [billing, customer relationship management, clinician systems, e-mail and so on] are up without any server issues.”
The only issues Halamka saw were a few cases in which people had not yet loaded laptops with the Outlook update, causing their calendars to be off because the Outlook client, as well as the Exchange server, need patches.
“We pushed the Outlook patch to all our e-mail users on the corporate network, but folks with laptops unconnected to the network for the past few weeks may have not received the patch,” he says.
At Kenexa in Waltham, Mass., it seems the patches and updates the company started rolling out in early February have worked. Kamal Jain, director of ASP operations there, says while the impact of preparing for the change has been “significant and disruptive” more from an administrative aspect than a technical one, the efforts are delivering the desired results.
“The folks who were up watching and running audit/comparison scripts in the middle of the night reported absolutely no issues,” Jain says. “As we hoped and expected, there was no impact to our customer-facing environment.”
That’s not to say that Jain and his company didn’t feel any impact from the change.
“Outlook definitely had issues with messed up appointments. Whoever’s bright idea this [daylight-saving time] change was … it turned into a triple-whammy for important but non-mission-critical things like calendaring,” he says.
The results are the same for Todd Wilson, Operations Manager of Enterprise Services at Johns Hopkins University Bloomberg School of Public Health in Baltimore, who reported Sunday that “everything so far is solid.”
Wilson’s team in November started researching necessary patches, distributing educational materials to the user community and making update information for personal machines available via the corporate intranet. With vendors frequently changing their stance on the needed patches, his IT staff waited until March 1 to begin updating the systems. Wilson says the advance work is making for a smoother transition today.
“We started patching servers and support software based on the best practices the vendors had provided at that time. The issue was the vendors kept changing that method slightly or kept updating the patches they were providing,” Wilson says. “So far [the work was] well worth the effort. I am sure we have not seen all the client issues. I think tomorrow will be the real tell of the tail. Systems with users hitting them.”
In any case, IT managers can use the lessons learned from the spring time change in November, when many will again be addressing the daylight-saving issue.
“We should be good in November. We will have to do a review at the end of the month to see what new patches come from the vendors because of what they have learned and fixed,” Wilson says. “We know where we stopped and will review all new patches that came out from that point.”
Like Wilson, Jain says efforts at his company will largely depend on the patches vendors update, but the groundwork has been laid for a smoother transition come November. IT managers will once again be waiting on the vendors to ensure the patches they distribute deliver the best results.
“Any systems and equipment which were patched for the ‘spring ahead’ will have the new/current time zone definitions so we should be good to go unless the laws change again,” Jain says. “We will need to ensure the patches are uniformly applied to all systems, and until such time as a standard patch revision from Microsoft, Cisco, F5 and all our other equipment vendors is certified and baked into our build procedures, we’ll need to re-run the audit of all systems which are of concern before the November ‘fall back’.”
When it comes to vendor-supplied patches, Rich DeBrino, CIO for Everett, Wash.-based Advances in Technology, which handles IT for a variety of healthcare organizations including parent company Compass Health, says his team will be a bit more wary next time.
“The problems we did have were related mostly to the lack of 100% effectiveness of the vendor-supplied patches in some instances, which forced us to either adopt workaround solutions or take several ‘runs’ at the problems until they were resolved, which they were … but not without losing sleep in the process due to the length of time it took to test each possible solution along the way,” he says.
DeBrino had each of his engineers assigned to specific servers that they were responsible for patching, updating and testing with a deadline to have all servers completed before the weekend. Those who had problems getting their systems to sync with the new time had to work through the weekend to ensure they were brought in line with the new time by the start of business Monday at 6 a.m. PST.
“Next time we would set the expectations of our engineering staff to not be quite so trusting of vendor-supplied fixes and allow for more time for actual problem resolutions – and assume the worst,” he says. “That way we wouldn’t have been quite so surprised at some of the issues we had to resolve along the way.”
“The transition was a lot like a duck swimming: graceful to see from outside the water, but down below paddling like crazy at times,” DeBrino says.
