As more and more servers are virtualized, connections between them are increasingly handled by virtual switches running on the same servers, begging the question, does the top of rack data center network switch ultimately get subsumed into the server?
Advocates say yes, especially given that servers today are packed with multicore processors, additional Layer 2 intelligence and dense optical connectors. Upstream core connectivity could then be provided by optical cross connects that just move traffic based on directional guidance from the server.
Pessimists say no, or not right away. Servers will continue to assume more switching duties between virtual machines, but the ToR will live on for some time to come.
“The short answer is no,” says Alan Weckel, switching analyst at Dell’Oro Group, when asked if servers will eventually replace ToR switches. “At the end of the day, it will be rack servers connected to top of rack switches. That’s 80% of the market now. That ToR isn’t going anywhere.”
Alan Weckel, switching analyst at Dell’Oro Group
Fiber Mountain is one company that disagrees. The startup makes software-controlled optical cross connects designed to avoid as much packet processing as possible by establishing what amounts to directly attached, point-to-point fiber links between data center server ports.
+ MORE ON NETWORK WORLD: Juniper unbundles switch hardware, software +
“We’re getting rid of layers: layers of switches, layers of links between switches,” says MH Raza, Fiber Mountain founder and CEO. “Switching as a function moves from being inside a box called a switch to a function that co-resides inside a box we call a server. If we put the switching function inside a server, it’s the same logic as a rack front-ending a number of servers; it’s the housing of a server with a switch in it front-ending a bunch of VMs. Why can’t that decision be made at the server? It can be made at the server.”
Raza says he knows of a vendor – whom he wouldn’t name – offering an Intel multicore server motherboard with a Broadcom Trident II switch chip and a high capacity fiber connector. The 1U device has a fiber port that can support up to 64 25Gbps lanes at 800G to 1.6Tbps of capacity – which is similar in capacity to the Intel and Corning MXC connector. With the MXC and similar silicon photonics, servers can communicate directly without any switch between them, Raza says.
“The decision could be made by the server,” he says. “I can assign packets going out the right lane. How many places does it need to go? Ten, 12, 40? Not a problem. When you have an MXC connector you could take them to 32 different destinations.”
Raza says this is possible now but no one is talking about it due to its disruptive potential. We are still wearing the blinders of traditional network thinking. “Nobody is talking about this because it is based on how fast silicon photonics will get adopted,” Raza says. “But it can be done now. The timing depends on investments and shifts” in technology and markets.
Given VMware’s NSX product is designed to handle virtual switching in VMware virtual server environments, you might think the company would be a big proponent of the idea that servers will eventually subsume switches. But even though the server-as-a-ToR switch architectural model is being proposed for hyperscale environments, Guido Appenzeller, chief technology strategy officer in VMware’s Network Security Business Unit, has never seen it used.
“If you want to get rid of ToR altogether, you need new silicon in servers,” like packet sorting engines, Appenzeller says. “You probably need a mini switch in the server. It doesn’t work with today’s architecture.”
That mini switch would be an Ethernet device enabling a direct server-to-server fabric. Another option would be a Layer 1 cross connect and multiplexer on the server motherboard, Appenzeller says.
Appenzeller gives the nod to the Ethernet mini switch implementation due to its familiarity in the server world, and its ability to do virtual LAN separation, something optical cross connects cannot do, he says. “I’ve never seen either deployed,” Appenzeller says. And both may be impractical given the steadily dropping price of ToR switch ports. “ToR prices are coming down quickly.”
Dell’Oro Group group agrees. The company reports that the average selling price of a 10G Ethernet port will drop from $715 to $212 between 2011 and 2016.
And the price/performance of network silicon from suppliers like Broadcom and Mellanox is outpacing that of general purpose CPUs, according to JR Rivers, CEO and cofounder of Cumulus Networks, a maker of network operating system software for bare metal switches. Also, bogging down the central CPU with networking features would sap its value. “When you start to put really beefy stuff in the middle of your CPU silicon, you diminish returns,” Rivers says.