Fabric Extender - Extended Virtual Chassis ... VERY FEXY !!!

Continuing further on our End-to-End Nexus DC design and Nexus 7K vPC blogs … One of the other prominent CPOC test components was FEX (Fabric Extender) module. Best way to describe FEX would be a (remote) line card hanging off a Cisco Nexus 5000 switch. Just like CAT6500 has Supervisor module, line cards and a Chassis … Similarly imagine Nexus 5000 as the supervisor module (with lots of 10gig ports) and FEX as a remote line card connecting to one or more of those 10 gig ports … forming an “extended virtual chassis”. As many as 12 FEXes can be connected to the Nexus 5K switch and this entire infrastructure can physically be separated by racks/rows (limited by 10 gig SFP+ optic range … which can be in KMs) … but still act like a single management entity (switch). FEX Module (as any other line card) to be operational require connectivity to a 5K and has no entity on its own. FEX modules belong to Cisco Nexus 2000 series and first member in this family (one we tested @CPOC) is Nexus 2148T. Nexus 2148T provides 48x1GE CU server ports for host connectivity and 4x10Ge SFP+ uplink ports for Nexus 5K connectivity. Depending on customer over-subscription requirements one or more 10Gig ports on FEX can be leveraged for uplink (5K) connectivity. Although we tested it with an EFT (early field trial) code version… the actual CCO build version (4.0(1a)N2(1)) was released by Cisco last week and it is possible now to deploy FEXes. Great work from Server Access & Virtualization BU wrapping this up. A new switchport mode type is introduced which differentiates Nexus 5K ports connecting to a FEX module - “switchport mode fex-fabric”. SDP (Satellite Discover Protocol) is enabled with this command which in turns discovers any connected FEX on that port. Very neat concept providing best of both Access layer approaches (end-of-row and Top-of –rack). With 5K deployed as end-of-row switch and FEX deployed as ToR. This approach works out great for all aspects of Data Center planning E.g. space, power, cooling and consolidation. For more information on FEX please refer Cisco Nexus 2000 CCO link Below is a high-level diagram of our Test-bed setup ... [IMG]http://i706.photobucket.com/albums/ww66/kamalvyas/FEXY.jpg[/IMG] POD servers needing 10Gig (orange links) connectivity are hooked directly to Nexus 5000 where as all 1Gig copper links (blue link) are terminated on any of the attached FEX module. Once the FEX module looses its link to uplink Nexus 5K ... it shuts down all its down-link (server) ports ... forcing Severs to failover to Switch-B (FEX) In our testing we were able to test and satisfactorily demonstrate most of the access level (FEX appropriate) features. Below is a list of some of the prominent test cases we ran;

  1. Static and Port-Channel Pinning Modes
  2. Unicast and Mulicast traffic forwarding
  3. IGMP snooping
  4. PACL and VACL on FEX server Interface
  5. Port-fast and BPDU guard
  6. VLAN and 802.1q trunks
  7. Support for Jumbo Frames and HW queues
  8. Disconnecting FEX from 5K and ensuring all FEX ports are shutdown once disconnected

Surprises (luckily not major show stoppers for us) we found during our testing …. No Support for Port-Channel on FEX Server (down-link) Ports So if you have a server which requires multiple1Gig port channel connectivity FEX won’t be an option. Fortunately the Nexus 5K code version which supports FEX modules also allows administrator to configure the first 16 ports on Nexus 5K as 1 Gig ports rather than 10 Gig. So it can be a possible option if you have one odd case of a server requiring port-channel connectivity. I personally will not prefer to eat up my 10 gig ports to support 1Gig port channels. No Support for 10/100 speed settings on the FEX module Who needs 10/100 inside of a DC … right? Well I though same way. But if you have not invested in a out-of-band management network and have devices like KVM, Terminal Servers, or servers which have RLM ports … you need to have 10/100 support. No Distributed forwarding on the FEX module All traffic even if destined to a port on same FEX … is forwarded upto Nexus 5K switch and returned. It would be nice if in future HW versions Cisco builds DFC like features on FEX modules. Overall we were quite impressed with the product and plan to leverage it for our upcoming DC project. We will be testing and playing with a lot over next few weeks. Please feel free to share any comments/questions/suggestions you might have regarding this topic. More to come on our end-to-end Nexus Design … Stay tuned. Thanks for Reading …!!!!

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2009 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)