Ethernet on a Ring

Methods of using Ethernet in a ring topology

If you are considering a network design that has a ring topology and you desire to use Ethernet then you should first be aware of the issues of using Ethernet with a ring architecture. Ethernet is traditionally thought of a as a bus or star topology. With the pervasiveness of Ethernet technology you may find yourself contemplating using Ethernet with a ring architecture. These new extensions to spanning tree allow for fast convergence of Ethernet ring topologies.

If you have ever thought about building a ring topology using Ethernet links you quickly come to realize the limitations of traditional spanning tree. We are all familiar with the limitations of traditional IEEE 802.1D spanning tree.

The first issue you run into with spanning tree is the convergence times. The traditional spanning tree timers (15 seconds for listening, 15 seconds for learning, and 20 seconds max-age timeout) don’t allow for fast convergence times that measure up to today’s standards for converged network traffic. When a service provider tries delivering converged triple-play services the 30 seconds to 50 seconds of convergence is unacceptable. Service providers would prefer optical convergence times on the order of 50ms which is what they expect from their DWDM rings.

The second issue that you encounter is the 7-hop maximum dimension of spanning tree. The default STP timers are set with the expectation that the diameter is no more than 7 hops. This 7-hop Spanning Tree limit is not to be confused with the 5-4-3 rule of thumb for 10Mbps Ethernet (5 total segments, 4 repeater hops, and 3 user-populated segments).

Below is a diagram of a ring Ethernet network. If one switch is the root of the spanning tree then another switch somewhere on the opposite site of the ring will have a port leading to the root in blocking mode. Therefore, the topology under normal circumstances would be a long line of Ethernet switches. If a failure were to occur somewhere on the ring then the spanning tree protocol will need to converge the topology which can take longer then desired. The second byproduct is that the interfaces on the STP root will be oversubscribed and probably higher utilized with the cumulative traffic volumes traversing the ring.

Ethernet Ring

So you may be wondering why even try to design a network with this arrangement. Service providers would consider this type of Ethernet ring topology if they are constructing a Metropolitan Area Network (MAN) or Metro-Ethernet service. A carrier may be trying to develop a service in a region where it is difficult to create point-to-point fiber links from their central Point of Presence (PoP) to customer sites or buildings. In this situation they may chose to use a ring if the rights-of-ways-and-easements dictate that topology is easier to construct. Enterprise organizations who may consider using a ring may be a school district that is going to use municipal fiber pathways to construct a ring that bypasses all their schools. Municipalities or counties may have access to fiber networks that have been laid in their geography and rather than deploy expensive carrier-grade optical switching devices they want to use simple Ethernet interfaces that they are familiar with managing.

There are solutions available to help you construct Ethernet ring topologies and overcome these limitations of traditional spanning tree. These Ethernet ring protocols do not replace spanning tree but rather augment it by limiting the STP domain to increase STP convergence and also help STP become aware of topology changes. These solutions typically have a master bridge on the ring that is involved in keeping the topology intact. These master nodes on the ring send messages to other devices on the ring to prevent a loop from forming. The master node also responds to breaks in the ring (fiber cut) and rapidly changes the node on the ring that is blocking.

Cisco Resilient Ethernet Protocol (REP):

Cisco offers a solution for service providers building these types of Metropolitan Area Networks (MANs). Cisco’s Resilient Ethernet Protocol (REP) is not a replacement for spanning tree but provides a way to limit spanning tree and optimize it for Metro Ethernet ring topologies. REP is designed to give 50 ms. convergence times and worst case times around 250 ms.

In REP, the master node controls the ring status. Instead of a ring, REP creates a chain of nodes along ring which is called a REP segment which is bordered by two edge ports. In the diagram below, edge ports (E1 and E2) between master nodes provide the upstream bandwidth off the ring. It is possible to have a single master node with both edge ports on it. When the ring is fully operational then one portion of the ring between the edge ports will be in blocking mode but if the fails then that point will go to an unblocking state to help the ring heal it quickly. REP is used to detect Loss of Signal (LOS) and detect neighbor adjacency failures on the segments that are created out of the connections between bridges. When a failure is detected then REP sends a failure notification and all switches flush their MAC entries on REP ports and the alternate ports transition to unblocking state to help repair the ring. REP also helps optimize the ring bandwidth by providing load balancing on a VLAN basis. The common link between the two edge switches can run REP, spanning tree, or can be an EtherChannel.

REP Ring

REP is one of Cisco’s Cisco Carrier Ethernet Solutions. REP is supported in Cisco’s Carrier Ethernet switches and intelligent edge router products. Cisco’s Metro Ethernet line of Ethernet switch products (ME 2400, ME 3400, ME 3750, ME 4900, 4500, ME 6500, and 6500) all support REP. The reason that the hardware is important is that REP Adjacency Protocol messages use a Cisco reserved MAC (0100.0CCC.CCCE) address to identify them and when they are received by a REP-capable switch those REP messages are fast-tracked to the processor.

Other Vendor’s Ethernet Ring Solutions:

There are other Ethernet ring solutions that exist. Configuration of each of these techniques is not very complicated but it can get complicated depending on the topology you want to create. Each vendor has their own terminology and a slightly different way of approaching the Ethernet ring topology problem.

Extreme has their Ethernet Automatic Protection Switching (EAPS) which can provide an Ethernet ring capability. EAPS uses domains to define a ring or multiple rings and a control VLAN is creates to send EAPS messages between the nodes. Nodes on the ring can have the role of master node, transit node, edge node, or assistant edge node depending on where they are along the ring. Each node has a primary and secondary EAPS ring port that leads toward the master node. The secondary port on the master node is used to block traffic and prevent a loop from forming. If the master node senses a ring link failure then it will bring its secondary port into forwarding state to repair the ring. Extreme has been working within the IETF to create an EAPS RFC 3619. However, only a few other vendors have adopted the standard. 3Com (Huawei/H3C) has developed the Rapid Ring Protection Protocol (RRPP) which follows the EAPS standard. Allied Telesis also has an EAPS protocol they call Ethernet Protection Switching Ring (EPSR).

Brocade offers their Metro Ring Protocol (MRP). In this solution Ring Hello Packets (RHPs) are sent between Master switches around the ring. It works similar to the other techniques where during normal conditions one point along the ring is in blocking mode to prevent a loop. If a failure is detected then that port is put into pre-forwarding state to listen for RHPs and then the port transitions to forwarding state to restore the ring quickly.

Although there is little publicly available information on it Force10 Redundant Ring Protocol (FRRP) is a similar extension to spanning tree that can achieve the same results of these other approaches.

While not exactly a household name N-Tron also has their N-Ring technology that is integrated into their 700, 7000, and 9000 series Ethernet switches products. It is basically the same idea just they call the master nodes N-Ring Managers.

Industry Standard Approaches:

It should also be mentioned that the current set of deployable Ethernet ring protocols are proprietary. Each of these vendor’s solutions will not interoperate with each other and if you chose one you will be locked into that vendor. I never advocate that customers jump in with both feet into a proprietary solution and I always favor industry standard solutions when available. In the future these different techniques may converge into a standard but for now these solutions are not interoperable and thus require a single vendor’s equipment throughout the network to make an Ethernet ring operate. There are some standards being developed that solve the Ethernet ring problem in a similar way.

The International Telecommunications Union (ITU) Telecommunication Standardization Sector (ITU-T) Study Group 15 – Transport and Access Q9 has been working on Ethernet Rings Protection Switching (ERPS) (ITU G.8032). In this solution a Ring Protection Link (RPL) is created along the ring’s path and a single RPL owner node blocks the ring from forming a loop. Control messages called Ring Automatic Protection Switching (R-APS) traverse the ring to detect failures and repair the ring. When an RPL node detects a failure then R-APS signal fail messages are sent and the RPL owner node unblocks the RPL port to restore the ring to operational status. One vendor that has adopted this industry standard approach of ERPS is Juniper Networks. The downside to ERPS is that it only works on rings and it won’t work in a mesh environment, it doesn’t support overlapping rings that share interfaces, and it requires specialized hardware.

The IEEE’s view is that Ethernet can simply be used on top of their IEEE 802.17 Resilient Packet Ring (RPR) protocol. There is an extension to this standard called Enhanced bridging (IEEE 802.17b) that defines the use of Ethernet over RPR networks. RPR is a MAC-based protocol and Ethernet can use RPR over either Ethernet PHY or SONET/SDH PHY. One advantage of RPR is that is uses the Special Reuse Protocol (SRP) to create efficient traffic paths with either aggressive or conservative fairness algorithms, rate shaping and service classification. A disadvantage of using RPR for Ethernet ring connectivity is that is requires specialized hardware that can converge the ring quickly.

Final Thoughts:

If you are building an Ethernet-based network that uses rings then you should consider the limitations of spanning tree and use a technology that will give you the performance you desire. You should look at these Ethernet ring solutions and determine if these will give you the convergence times you require. Again, if you want to avoid getting locked into a vendor proprietary solution then you will have to wait for the standards to congeal and for products to support those industry standard protocols.


Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2009 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)