A delectable storage plan

By employing new data-center technologies to address a storage binge, Krispy Kreme baked up a new back-up scheme.

Krispy Kreme. The words have become a nationwide Pavlovian bell that stimulates cravings for fresh, sticky doughnuts. Although founded in 1937, the company waited until 2000 to go public, after which its confections obtained near cult status and its growth exploded. From 140 stores in 27 states at its IPO, the donut maker now boasts 343 stores in 42 states and four countries, plus other venues such as grocery stores. All this while sales at individual stores have achieved years of double-digit growth.

Such success left Krispy Kreme CIO Frank Hood with a problem: out-of-control storage growth.

Between company-owned and franchise stores, Krispy Kreme’s 30-person IT department supports about 1,000 users, many through its extranet for franchisees – Mykrispykreme.com – which gives access to real-time applications such as sales reports and training videos. Some data generated by these 1,000 folks follows easily predictable growth patterns and some doesn’t.

For instance, Hood could accurately predict the storage needs for the three data warehouses used to track sales. He simply used data warehouse tools to estimate how many transactions each store was likely to have (the data warehouses store every sales transaction from about the past eight years). But storage for other applications was leaping wildly and unpredictably.

Hood had thrown hardware at the problem, but in fewer than four months, a new 1G-byte disk array had been filled. Clearly, he needed a far better method than server-attached storage to predict and manage the hodgepodge storage arrays Krispy Kreme had begun to accumulate. He needed new data-center storage technologies without spending a fortune.

After examining his options, Hood selected network-attached storage (NAS). He says NAS was a proven technology with advanced management functions, such as self-healing and speedy backup/restore options. Plus, NAS gave him a fast route to virtualized, pooled storage while using his already-paid-for disk arrays.

Specifically, Krispy Kreme deployed Network Appliance’s NetApp F825 appliance, which gives the company 2.4T bytes of storage capacity. NetApp SnapShot and SnapRestore software provide data recovery, and Virtual Local Disk software performs virtualization tasks. Rather than Fibre Channel, the company uses the less expensive and easily deployable iSCSI Ethernet-based storage protocol to link e-mail servers and the main SQL datastores to the F825, Hood says. Ethernet connections, from a Nortel Passport 8300 Layer 3 core switch, operate at 1G and 10/100M bit/sec. Users reside on one virtual LAN, taking one of four 1G-bit/sec connections for directory access.

“We took the opportunity to take all this disparate storage that was decentralized – a little bit of storage here and a little bit of storage there – and we took the low-hanging fruit first. We consolidated user directories and major data-stores that were in common areas to the [NAS], then we looked at higher-user systems – like e-mail,” he says.

Servers that access the NAS include Microsoft SQL Server, IBM Lotus Domino Web servers and those for other, specialized applications. These servers, a mix from HP and IBM, aren’t self-healing, but are self-managing. “What we’ve got is a box that says, ‘I’m sick, come and fix me’ before our data center folks would know it’s broken,” he says. Hood organized the application servers onto a VLAN and created a 3G-bit/sec pipe to them using the remaining Gigabit links.

Company-owned stores tie into the network over private lines operating at 64K- to 128K-bit/sec. (Franchisees are asked to supply their own Internet access, high speed if possible, to gain access to Mykrispykreme.com. For those that must dial up, Krispy Kreme maintains a Citrix server.)

Improved backup was an immediate benefit of the new storage design. With NetApp’s SnapShot software, Krispy Kreme conducts incremental backups of its most precious 1T byte of sales data 12 times daily. Should a crash occur, “I can get you back up with your data from two hours before the event, and a re-install takes seconds,” Hood says.

Centralized storage over an Ethernet core also led Krispy Kreme to link corporate, manufacturing and distributing facilities near its Winston-Salem, N.C., headquarters over an outsourced metropolitan-area network (MAN) operating at 100M bit/sec. With the MAN and NAS in place, Hood is contemplating plans to ditch Krispy Kreme’s expensive outsourced disaster-recovery provider in favor of placing synchronized, mirrored NAS systems at the facilities it owns.

“Instead of going to Chicago for recovery, we can go to Greensborough, just 30 minutes down the road,” he says.

The only reservation Hood had about his new data center storage was moving to a single point of failure. But that has proven to be a non-issue, he says. Plus, better storage management, including automatic failover of a troubled disk, has freed his staff from many mundane tasks. “You have to force yourself to notice [the NAS] because it is monitoring itself.”