One of the hardest-hit industries in the recession was financial services, the largest consumer of corporate servers, representing nearly 25% of the market. And while Wall Street is starting to get back on its feet, the financial-services companies are undoubtedly going to be far more frugal with their capital investments.
One of the hardest-hit industries in the recession was financial services, the largest consumer of corporate servers, representing nearly 25% of the market. And while Wall Street is starting to get back on its feet, the financial-services companies are undoubtedly going to be far more frugal with their capital investments.
Recognizing this, server designers are looking for ways to reduce costs while still being able to meet the aggressive demands of robust financial service applications. Consequently, components such as PCI Express (PCIe) switches and other highly integrated devices have become more attractive because they allow designers to limit the chips and power required to build systems.
Form factors help determine server costs. The traditional rack-mount server is simply a stack of independent boxes connecting to a switch that could, for example, be sitting at the top of the rack. The switch is the traffic cop, routing traffic among the servers, storage systems and the network. Usually there are two switches in a rack to fan-out to the standard 1U and 4U modules. The most common racks are 42U and can support a maximum of 40 servers.
As of late, blade-servers have been growing in popularity due to their compact form factor, low power consumption and efficient cooling mechanisms. Blades remove much of the system overhead by doing away with many of the redundant components used in rack-mount servers.
In bladed systems, for example, multiple blades share a single power supply and network connection as opposed to having those sources dedicated to each module. Each blade plugs into a common mid-plane that provides connections to the switch fabrics, which in turn allow the blades to communicate with external systems or modules via Ethernet, Fibre Channel or InfiniBand. A blade server chassis supports 10 to 16 blades and a 42U rack can support up to 100 blades.
Applications
In either form factor, each module is responsible for a specific task. For example, certain blades may be used for computing/processing, while others may be used for I/O purposes, such as storage controllers or customized components. Each blade plugs into the mid-plane, which provides connection to the switch fabric at the back of the chassis that provides connection between all the blades as well as the outside world.
Financial-services applications tend to be heavily focused on computing and storage. A massive amount of computing power is needed for extensive number-crunching, financial modeling, simulations and analysis conducted by investment banks, government agencies and financial consulting firms. In such applications, high-availability computing clusters are created to ensure high performance as well as to minimize the risk of system downtime caused by a single failing module. A large number of processing blades are used in such applications.
Over the past decade, the growing trend of online banking, shopping and stock trading has fueled the increasing storage demands of the financial-services industry. From transaction history, account balances and online payments to real-time stock quotes, 24-hour customer service and around-the-clock trading, all this data needs to be available instantly to both institutional users and consumers alike.
Losing the data is not an option. Any system downtime is considered to be a huge inconvenience to consumers and potentially damaging to businesses. A large portion of the servers deployed in the financial sector is used to store and back up (often several times) newly created/updated financial data. Furthermore, to accommodate the rapidly growing storage needs, institutions employ expansive server farms.
The need for fan-out
Enter PCIe, which has established itself as the serial-interconnect protocol of choice to link the host (processor/chipset) to a wide variety of motherboard-mounted peripherals. These peripherals can be I/O components such as graphics processors, network controllers, storage controllers or just empty PCIe slots for customizable I/O expansion by the user. An extension of the conventional PCI protocol used since the early ’90s, PCIe provides not just high-speed interconnect but also Quality of Service (QoS), data integrity, as well as an expansive ecosystem.
Due to the power requirements and cost-sensitive nature of chipsets, vendors such as Intel and AMD only integrate a limited number of PCIe lanes and ports on the chipset. This means that only a limited amount of endpoints (i.e., devices or modules) can be connected to the switch fabric.
At the same time, the need for system I/O continues to grow. In an effort to differentiate their products and create specialized systems targeting niche market segments (e.g. super-computing or high-end storage), system designers often pump their systems full of endpoints, designing them to support numerous interfaces.
Typically, the number of PCIe lanes/ports supported by the chipset is not sufficient for the needs of application-specific servers. Hence the need for a PCIe switch with a large number of lanes and ports to provide connectivity to all the other blades/modules on the system.
The common theme among computing and storage applications, whether in rack or blade form, is that a large number of modules are interconnected with each other.
The graphic below shows a bladed computing cluster where the switch fabric fans out to a number of blade-servers via a PCIe switch. Typically there are four switch fabrics in each blade server — two for networking (e.g., Ethernet) and two for storage (e.g., Fibre Channel).
There are two switch fabrics per interface to provide redundancy in case one switch fabric fails. Each switch fabric is connected to each blade-server via the mid-plane. As shown in Figure 1, each blade-server often houses a smaller 24-lane switch, which then fans out to the various endpoints (memory, ASICs and I/Os) on the blade. If needed, PCIe switches may also be added to the storage switch fabrics to fan out to a large number of Fibre Channel controllers.
Many of today’s PCIe switches support an integrated non-transparent (NT) port, which allows for host isolation in failover systems; it keeps the CPUs from trying to take ownership of each other’s devices. So, instead of having to use a separate device to serve as an NT buffer for host isolation, the integrated NT port enables designers to implement a one-chip solution, thus saving board space, power and cost.
Highly integrated solutions lower overall system cost
With the advent of PCIe as a protocol that fills next-generation-system designers’ interconnect needs, worldwide demand for PCIe switches has experienced significant growth. As a result, PCIe switch vendors now provide an expansive portfolio suitable for a variety of applications. Furthermore, these vendors are integrating more and more features and functionality that designers, in turn, are leveraging into versatile, highly efficient systems; designers can now accomplish with one switch what they had to use multiple devices for in the past.
For example, several PCIe switches today feature an integrated direct memory access (DMA) engine. In the past, designers had to rely on the system processor to provide DMA functionality. This meant they had a limited number of processors to choose from and ended up paying a premium for these processors. But with switches now featuring integrated DMA engines, designers can choose from a larger selection of lower-cost processors that won’t be called upon for the DMA function.
Similarly, next-generation servers benefit from multi-root (or multi-host) support found in PCIe switches. Large switches with ample lanes and ports, multi-root switches can be divided into smaller virtual switches, each with its own host. Hence, redundant computing clusters can share one multi-root switch in the backplane as opposed to having a separate PCIe switch on each switch fabric.
As illustrated in the graphic below, for example, Blades 7 through 12 are redundant with Blades 1 through 6. However, instead of each redundant cluster requiring its own PCIe switch, the multi-root feature allows one large 96-lane switch to service both clusters simultaneously.
To simplify the diagram, only one switch fabric is shown here. However, a redundant switch fabric would be utilized as a backup in case the primary switch fabric ever failed. Furthermore, the switches on the blades support an integrated DMA engine, enhancing system performance in compute-heavy applications.
Another way this new generation of servers is benefiting from PCIe technology is through a feature called multicast, also being integrated in PCIe switches. Multicast allows for a packet coming into the switch to be simultaneously forwarded to multiple devices connected to the switch. This helps offload the processor from having to do redundant writes.
For example, in a redundant system, the processor may have to send the same data to as many as four systems to ensure the data is properly backed up. Without multicast, the processor would have to execute four separate write transactions to complete this task, slowing down the overall system performance. With multicast, the processor only has to execute one write and the PCIe switch will then automatically create redundant copies of the same packet and send them to the appropriate locations. This frees up the processor to focus on other tasks and helps increase overall system performance.
With financial-services sector putting much more pressure on server makers to provide dramatically better performance, a wider range of capabilities and more bang for the buck, PCIe switching is now playing a crucial roll in server design. IT professionals can be assured that servers that incorporate the technology will be the most efficient available.
Touseef Bhatti (tbhatti@plxtech.com) is senior product marketing engineer at PLX Technology. He holds a BSCE from the University of Michigan.
