The ins and outs of interconnects

It used to be, if you wanted a storage-area network, you turned to Fibre Channel, the high-speed transport technology that moves SCSI traffic from servers to disk arrays and tape drives.

Now you have to sift through a veritable alphabet soup of storage interconnects to determine which best suits your SAN needs. You have iSCSI, Fibre Channel over IP (FCIP) and Internet Fibre Channel Protocol (iFCP) for pushing storage data over IP, and InfiniBand, a high-speed I/O switching fabric.

"The lines are blurring between what used to be the separate universes of storage, networking, clustering and I/O technology," says Karl Walker, a vice president of server strategy at Hewlett-Packard. "All of that stuff is really starting to come together in interesting ways."

Interesting is one way to put it. Complex is another. In a recent survey of 96 network professionals who attended Network World's summer seminar tour, "Storage Town Meeting: Ensuring Business Continuity," 21% said they could see an immediate need for new storage interconnects but weren't sure what was available. Another 49% said they could see a future need but needed to investigate options (see graphics).

Users should start their research with iSCSI, FCIP and iFCP. Storage vendors concocted these interconnect protocols in response to shortcomings of Fibre Channel. Fibre Channel networks require expensive dedicated fiber lines and specialized expertise and present a distance limitation that could be troublesome. Fibre Channel deployments cap off at about 10 km, a real problem when trying to set up geographically distributed storage for disaster recovery.

Vendors are embracing iSCSI, FCIP and iFCP in varying degrees, but haven't rallied behind one protocol yet. Some are hedging their bets with storage switches and other products that support the incumbent Fibre Channel and IP-based storage. "The industry has invested a lot of money in building Fibre Channel SANs. Fibre Channel is clearly going to remain the dominant force for the next half decade," says Jamie Gruener, a senior analyst with The Yankee Group.

On the plus side, this dual support lets users get a feel for new IP SANs without trashing existing Fibre Channel SANs.

ISCSI, FCIP and iFCP await standards ratification, as does InfiniBand. Until then, most enterprise users are watching, but not deploying, the protocols.

"We'll continue implementing in our existing manner, using our existing strategies: Fibre Channel, direct-attached storage. But we are looking at iSCSI and InfiniBand to figure out where they're going to make sense," says John Blackman, systems architect in the emerging technologies and consulting group at Wells Fargo in Minneapolis. "I can't say where they would actually be deployed because I don't think the industry knows yet."

Sorting things out

To understand where each interconnect could be used, start by examining their differences.

ISCSI is a native IP interconnect that wraps SCSI data and commands in TCP/IP packets. This approach gives users full access to IP management and routing.

But latency, introduced when putting storage on the same route as other network traffic, and security have been big concerns. And iSCSI hasn't made the inroads expected when it was introduced by Cisco and IBM in February 2000. Cisco and Nishan Systems are among only a handful of vendors offering iSCSI switches. Still missing are iSCSI storage arrays from vendors such as EMC and HP. IBM put a damper on iSCSI when it halted development of the 200i iSCSI target storage appliance it introduced in early 2001. Users wanted more power than provided by this "kick the tires" version, says Clod Barrera, technical strategy director for IBM Storage Systems Group.

Users may get that oomph from TCP off-load engines (TOE), which off-load CPU-draining processes from servers. Companies such as Adaptec, Alacritech and Intel already offer TOEs for iSCSI, and others are expected to follow.

George Nathanson, IT director at Wireless Retail, sees iSCSI as an affordable SAN option.

"Without TOEs on the host adapter side and on the storage side, I don't see iSCSI happening, because the performance would be too poor," says Arun Taneja, senior analyst at Enterprise Storage Group.

Wireless Retail, a specialty provider of wireless services and products, ran into such performance problems with its iSCSI deployment. Like most small companies, Wireless Retail had been using direct-attached storage. Last fall, it was ready to move into more distributed, scalable storage. It chose iSCSI to avoid the typically big investment required for Fibre Channel.

"I wanted to utilize the storage for everything that it was worth, and that meant getting as many servers as possible connected and doing it as economically as possible," says George Nathanson, IT director at Wireless Retail, in Scottsdale, Ariz.

At the same time, Nathanson concedes that performance is one nagging concern of iSCSI. To address this, and failover, he opted for Fibre Channel to connect his SQL databases, which contain some of the more critical data in his storage setup.

Freeing Fibre Channel

FCIP and iFCP may be the answers for companies that want to extend their Fibre Channel storage farther than 10 km. Each uses gateways to encapsulate Fibre Channel commands into IP packets.

FCIP moves the encapsulated Fibre Channel data through a "dumb" tunnel, essentially creating an extended routing system of Fibre Channel switches. This protocol is best used in point-to-point connections between SANs because it cannot take advantage of routing or other IP management features. And because FCIP creates a single fabric, traffic flows could be disrupted should a storage switch go down.

IFCP, on the other hand, lets users maintain the Fibre Channel architecture while gaining the benefits of IP networks. IFCP wraps Fibre Channel data in IP packets but maps IP addresses to individual Fibre Channel devices. That provides more stability than a tunneled FCIP deployment, some vendors say.

When iFCP creates the IP packets, it inserts information that is readable by network devices and routable within the IP network. "IFCP gives you visibility into the talking pairs between servers and storage, and gives you a way to monitor discrete conversations between storage initiators and targets," says Tom Clark, director of technical marketing at Nishan, which supports iFCP, FCIP and iSCSI in its storage products.

Because the packets contain IP addresses, customers can use IP network management tools such as HP's OpenView to manage the flow of Fibre Channel data using iFCP.

Marketing, travel and hospitality conglomerate Carlson Companies, in Minneapolis, opted to use iFCP to extend the reach of its Fibre Channel direct-attached storage. It plans to eventually link the IP SAN to thousands of remote offices in more than 140 countries. It uses Nishan multiprotocol switches to convert data between Fibre Channel and IP. IFCP routes the Fibre Channel commands across the company's Ethernet LAN.

Gary Johnson, architectural consultant for Carlson Shared Services, says iFCP will let Carlson quickly extend its SAN globally because it uses native IP management. He says companies should be aware that latency could be an issue once storage traffic moves outside a dedicated link.

"As long as you engineer and design your network to support a storage application, rather than a messaging application, it's going to work just fine for you," he says. "As soon as we leave [our dedicated storage] network to go to a remote location, we certainly expect that we'll be using [quality of service] to manage that."

The biggest downside with Fibre Channel-based IP deployments is that they still require the expensive Fibre Channel back end. Vendors such as CNT, EMC, Pirus, SANcastle and SAN Valley Systems support FCIP. Nishan is the primary vendor supporting iFCP.

To InfiniBand and beyond

InfiniBand differs, at least for now, by being aimed at server processing. Portrayed as the fix for server I/O bottlenecks, InfiniBand connects server, storage and network devices at 2.5G to 30G bit/sec.

Current bus technology supports speeds up to 1G byte/sec.

But analysts don't see InfiniBand as an immediate SAN choice. "Over time, you may find that you can design a single InfiniBand fabric within a data center and basically run storage traffic on it as well," says Enterprise Storage Group's Taneja. "Conceptually, you could build a SAN with InfiniBand rather than Fibre Channel. But that's definitely a ways out."

Most storage vendors are holding off on InfiniBand plans until the market settles. Storage software maker Veritas Software, for example, went into a wait-and-see mode after Intel's late May announcement that it would not ship InfiniBand chips as planned in 2003, says Ruth Colombo, product manager for network storage at Veritas.

So it seems that Fibre Channel will not be quickly unseated. Rather, a number of protocols will each have a place in the storage architecture.

Apply for your free subscription to Network World. Click here. Or get Network World delivered in PDF each week.

The ins and outs of interconnects

Related Links