About a week ago I was talking with Jeff Hardee and he brought to my attention the idea of using long Spanning Tree Protocol (STP) path length. He asked me if other enterprises were using the new long STP path cost method due to the increase in link-speeds within data centers. I had never given this much thought so I did some research and this is what I found.
Jeff Hardee introduced me to the idea of using longer Spanning Tree Protocol (STP) path length values for high-speed links. I thought this was an intriguing idea and so I started to do some informal research. I asked other CCIEs at GTRI if they had encountered this at other clients. They confirmed that they hadn't configured this or seen it configured at clients. Regardless, even though this practice is not common it could still very well be a good idea.
Link speeds are increasing within data centers. 10 Gigabit Ethernet (10GE) links are becoming more popular within data centers. Many data centers still rely on layer-2 flat network topologies in data centers to allow for easy movement of virtualized servers.
As 10GE becomes more prevalent along with the use of Rapid Spanning Tree Protocol (RSTP), the Spanning Tree path cost values continue to shrink. The path cost is the metric STP uses to calculate the shortest path to the elected root bridge. The path cost is based on the speed of the bridge port interface. Back when Radia Perlman developed the Spanning Tree Protocol, 10 GE links were not even considered within the realm of possibility. Just like we have outgrown 32-bit IPv4 addresses, the 16-bit path cost value hasn't kept up with the pace of the networking industry.
I did some checking on cisco.com and found that the command for setting this feature on a Cisco 6500 is "spanning-tree pathcost method long". This command changes the path cost to increase it from a 16-bit value to a 32-bit value. More bits in the path cost value increases the range of possible link speeds. You can confirm the path cost method being used on your Cisco switch with "show spanning-tree summary" command.
By default Cisco switches use the original spanning tree "short mode" path costs using a 16-bit value. However, as interface bandwidth has increased the 16-bit value does not provide room for future high-speed interfaces. Using the newer spanning tree "long mode" path cost using a 32-bit value provides more granularity in data centers that use extremely high-speed interfaces. Following is a table of links speeds and the old and new values for comparison.
|Bandwidth||Old STP value||New Long STP value|
|N X 1 Gbps||3||10,000|
The primary consideration is that if you change your network to using the "long mode" then it must be changed universally within the LAN-switched environment. All of your switches should agree upon the method of spanning tree, the timers, and the path cost metrics. If not, then you could have inconsistencies that could cause your spanning tree not to converge properly as you would expect. That is important because once your layer-2 topology converges on top of the physical network then the layer-3 routing protocols will converge. If your layer-2 topology is not set as you expect then you could be surprised at how IP packets are routed through your network in non-optimal paths.
In my research I found that this "long mode" command is supported on data-center LAN switches like the Cisco 6500s/4500s/4900s and NX-OS. However, I was talking with Tim Clegg at GTRI, and although he hadn't seen this command used, he confirmed through his research that it isn't supported on 3560/3750 or 3560E/3750E. It is conceivable that some data center environments may be using these switches in the Layer-2 environment as top-of-rack switches so this may be a problem.
The spanning tree path cost values have been adjusted in newer Cisco IOS versions using the short-method. I recommend that you check the configuration guides and command references for the Cisco IOS versions you are using. It is conceivable that you may be using older IOS versions that have use the short-mode but use a cost of 1 for a 1Gbps link while newer IOS switches using short-mode may use a cost of 2 for a 10Gbps link. Therefore, if we are using a mix of old IOS and newer IOS we may have issues with older devices thinking the path cost is lower than it should be and that would cause problems. Therefore, we would need to manually adjust the path cost on those older devices on a per-interface basis - YUCK!
Two good books that cover the topics of Spanning Tree Protocol and the long-mode path cost are: Cisco LAN Switching Fundamentals, David Barnes; Basir Sakandar, Cisco Press, July 15, 2004, ISBN-13: 978-1-58705-089-3. Cisco LAN Switching Configuration Handbook, Second Edition, Steve McQuerry - CCIE No. 6108; David Jansen - CCIE No. 5952; David Hucaby - CCIE No. 4594, Cisco Press, June 19, 2009, ISBN-13: 978-1-58705-610-9.
This issue may also come into play when you are using a mix of vendor's LAN switching equipment. It is easy to see how one could make the assumption that spanning-tree is universal across multiple vendor's LAN switches and forget about the path cost implications. As we have seen, some manufacturers like Cisco have a mix of possibilities based on the IOS version and hardware model. Some manufacturers have been using the STP long-mode for years. For example, Juniper switches use the long-mode by default. Therefore, you will need to use the "spanning-tree pathcost method long" command on Cisco switches to make them interoperate with Juniper switches.
If you are using a Juniper LAN switch then the following JUNOS command shows each interface's port-cost, state, and role.
Scott@Switch# run show spanning-tree interface Scott@Switch# run show spanning-tree interface ge-0/0/0 detail
Scott@Switch# show spanning-tree bridge detail
A good reference for this information on STP configuration on Juniper switches can be found in this book. JUNOS Enterprise Switching, 1st Edition, Harry Reynolds; Doug Marschke, O'Reilly Media, Inc., July 21, 2009, ISBN-13: 978-0-596-15397-7.
Therefore, I think that most organizations might want to hold-off on using this command until the organization can validate that this command is universally supported on all their LAN switches. If you are in a multi-vendor LAN switched environment then you should definitely take inventory on which switches are using the 16-bit values or 32-bit values. All switches within a layer-2 domain/network would need to have full agreement on the path costs in order to prevent problems. Therefore, if you plan on migrating to long-mode then you should do it on all switches within your layer-2 LAN switch domain at the same time during a carefully orchestrated change window.
I am curious if any of you are using the longer STP path costs and what other issues should we consider as data center link speeds increase.