The data center fabric checklist you should know

Users share experiences in converging operations with three different implementations

Like anything in IT, there are several considerations to mull over before employing fabric technology to converge data center operations.

At its purest form, a switching fabric is a network topology where nodes are connected to each other through switches employing multiple active links. This is in contrast to a broadcast medium like traditional Ethernet, where only one path is active – but Ethernet is evolving through standards bodies like IEEE and IETF to incorporate multiple active paths and a link-state routing protocol to replace Spanning Tree for data center fabric deployments.

But there are several checklist items to comb through before deciding if a fabric, and which fabric technology, is right for your environment:

  • Are you a greenfield or brownfield shop?
  • Which fabric – Ethernet? Infiniband? Fibre Channel?
  • What do you want it to do?
  • How will you design your fabric architecture?
  • Should your environment be purely one or the other, or a hybrid?
  • Should you route or switch between the different fabrics in a hybrid environment?
  • What are the factors to consider when converging data and storage?

[CHOOSING THE RIGHT ONE: What to look for in network fabrics]

MORE: Epic Interconnect Clash! InfiniBand vs. Gigabit Ethernet]

PayPal is a Mellanox Infiniband shop. It has more than 300 servers and about 12 storage arrays across three Infiniband hypercube clusters, with converged storage and network transport. The fabric has been in place since 2008, and PayPal is migrating from a 16Gbps Double Data Rate (DDR) environment to a 56Gbps Fourteen Data Rate (FDR) infrastructure.

PayPal looked at 10G Ethernet but “we knew we would (overrun) that with storage,” says Ryan Quick, principal architect. Infiniband provides better bandwidth and latency than both Ethernet and Fibre Channel, Quick found.

“IB brings a lot to the table, especially for storage,” Quick says. It has a 64K packet size vs. 9K; wire-speeds are much higher; there are lots of different learning and path calculation capabilities, including dynamic routing at the fabric level; and multipathing works “right out of the box.”

“It had one big negative,” Quick says of the Infiniband fabric. “No one’s using it in the enterprise yet. But it had an awful lot of positives.”

PayPal was a greenfield fabric deployment in 2008 with convergence as its target. The company has a hybrid Infiniband/Ethernet environment with an internally-developed router connecting the two.

“It’s easier to inspect packets at Layer 3,” Quick says. “But none of the vendors are offering a Layer 3 router for IB-to- something else. We had to build our own.”

The router has two to four 10G network interface cards (NIC) for gateways, and a pair of Infiniband Quad Data Rate – 8/32Gbps -- NICs on the other side. Hypervisors are configured to create a virtual switched network with “pseudo Ethernet,” Quick says.

“The guests think they’re using Ethernet but it’s really [Infiniband] on the other side,” he says.  

Storage is directly cabled into the hypercube via the SCSI RDMA Protocol with “dual rails” configured for failover, Quick says.

The only “gotchas” during implementation were again, the lack of practical experience or ‘best practices’ with Infiniband in the enterprise.

“Reference topologies tend to reference everything in the world except Infiniband,” Quick says. “There’s never a place where it’s the only thing around.”

References such as white papers and vendor professional services consultants were not very convincing for Eze Software Group, a Juniper QFabric M shop with data centers that support a SaaS offering for financial traders. Eze was sold on QFabric and its single-tier/one hop networking proposition after extensive testing.

“You read the white paper, professional services come in here and you’re still skeptical,” says Bill Taney, vice president of network engineering and operations for Eze Software. “And when you test the implementation like we did and see it failover seamlessly, it’s pretty impressive.”

Eze deployed QFabric M nodes, interconnects and directors in March in two primary data centers that were relocated from previous facilities and redesigned from the ground up – greenfields. Four QFabric M interconnects form the 150G backbone core.

The fabric supports 400 mostly physical servers and a storage server. Servers are connected to multiple QFabric nodes over Link Aggregation Groups.

Eze engaged Juniper professional services and performed extensive failover testing to ensure the QFabric single-tier/one hop topology worked as advertised. It did, Taney reports.

“It rolled out a lot easier than I thought it would,” he says. “It was a fairly straightforward implementation. It just failed over very, very well.”  

Eze is now waiting for Juniper to add Virtual Chassis capabilities to its QFabric QFX top-of-rack switches, which would essentially allow users to configure a fabric with the QFabric interconnects.

“If you can get QFabric in a smaller deployment, without the interconnect… that would certainly be a solution I’d consider for our small data centers,” he says.

Another fabric deployment for a SaaS offering belongs to NPD Group, a Long Island-based market research firm. In January, NPD implemented HP’s Intelligent Resilient Framework (IRF) technology across a 40 switch environment in a greenfield data center anchored by HP 5900 switches in the 10G core.

IRF was deployed more to converge management than it was to converge storage. NPD is running data and storage on two separate Ethernet and Fibre Channel networks, though it is currently evaluating HP’s Fibre Channel-over-Ethernet offerings.

But NPD’s big challenge now is handing big data.

“(IRF is) really for ease of management, lower total cost of ownership,” says Gabe Abreu, senior vice president, Global Enterprise Systems at NPD. “Having the capability to manage connections virtually rather than physically making connections is just more appealing. Our biggest issue is getting the most throughput and processing power.”

NPD’s new data center, located about 80 miles from New York City, connects up to 5,000 devices, including servers, storage and networking nodes. While IRF virtualizes connection management, HP’s IMC management system provides a single-pane-of-glass monitoring capability for the IRF fabric, Abreu says.

NPD has another legacy data center that runs OSPF as its unifying “fabric.” The company plans to transition that to IRF on 2015, Abreu says.

Jim Duffy has been covering technology for over 25 years, 21 at Network World. He also writes The Cisco Connection blog and can be reached on Twitter @Jim_Duffy.

Editors' Picks
Join the discussion
Be the first to comment on this article. Our Commenting Policies