SONET provides high-performance SAN extension

Until now, storage-area networks have mostly been local networks, providing high bandwidth, low latency interconnect for the high-speed movement of block data from server to storage.

But, if all the data that needs protection is under one roof, it is vulnerable to natural disasters such as fires, floods and earthquakes. What is needed is a method of interconnecting multiple SAN islands over distance.

SAN extensions must provide high performance and work within today's datacom/telecom infrastructure. Fibre Channel over SONET provides such a solution. The protocol and gateways introduce low latency into the system.

Imagine a primary data center and a mirrored site each with hard disks operating at 15,000 rpm and 2 msec seek time. If data is written to both disks, we will first get a response from the local disk based on the 2 msec average seek time.

However, the response from the mirrored site will be delayed by the round-trip time to send the data. This round-trip time has two components: the length of the fiber between the two installations and the latency of the network equipment used to connect the Fibre Channel SAN to the fiber.

The latency of an optical signal traveling in a fiber-optic cable is 5 microsec per kilometer. If we locate our mirror site 200 kilometers away, the fiber alone will add 2 msec of delay to the response of the remote disk.

Now, the average latency for the remote SAN has been increased to 4 msec. Because the remote SAN is a mirror to the primary data center, the overall latency of the system has been doubled. We can think of our state-of-the-art 15,000 rpm, 2 msec seek-time disks as now being 10,000 rpm, 4 msec seek time drives. All the bandwidth in the world won't make this go away. Latency is a separate phenomenon from bandwidth. Both affect the throughput of a system but in different ways.

Unfortunately, the fiber-optic cable is not the only component of latency. The Fibre Channel gateway will introduce latency as Fibre Channel is encapsulated for transport over IP, ATM, SONET or dense wavelength division multiplexing (DWDM).

Depending on the architecture, a gateway may have multiple internal stages. Each stage introduces queuing delays and other forms of latency. A large, multiprotocol gateway might introduce hundreds of microseconds or even milliseconds of latency, whereas a highly integrated chip-based solution might introduce only 100 microsec or less of latency.

What matters is the total latency from end to end. Fiber length and gateway equipment contribute to that latency. Both should be evaluated as the storage network system is planned.

Why SONET?

Of the methods mentioned, Fibre Channel over IP provides scalability and convenience while Fibre Channel over ATM can be readily extended over distance (albeit with reduced data rates of 155M bit/sec or 622M bit/sec). Unfortunately, both techniques can introduce substantial latency (up to tens of milliseconds).

On the other hand, SONET and DWDM can provide low latency, high-bandwidth transportation of Fibre Channel traffic. The problem with DWDM is that the equipment is expensive and requires dedicated fiber to interconnect the sites. Dedicated fiber that runs 200 km is prohibitively expensive.

Fibre Channel over SONET emerges as an excellent choice for carrying Fibre Channel data. It is readily available from many carriers at rates up to 2.488G bit/sec and is already part of their existing infrastructure. Fibre Channel over SONET gateways can be built to introduce only minor amounts of latency (approximately 100 msec or less).

Because Fibre Channel over SONET does not use routers that drop packets under congestion, TCP and its attendant retransmission delays are eliminated. Also, Fibre Channel over SONET can be readily extended hundreds or even thousands of kilometers without installing dark fiber.

Helland is the director of product management at LightSand Communications. He is also one of the authors of the Internet Engineering Task Force standard for carrying Fibre Channel over IP. He can be reached at andyh@lightsand.com.