Why carriers aren’t ready for SD-WAN services

Throughout my early years as a consultant, when asynchronous transfer mode (ATM) was the rage and multiprotocol label switching (MPLS) was still at the outset, I handled numerous roles as a network architect alongside various carriers. During that period, I experienced first-hand problems that the new technologies posed to them.

The lack of true end-to-end automation made our daily tasks run into the night. Bespoke network designs due to the shortfall of appropriate documentation resulted in one that person knows all. The provisioning teams never fully understood the design. The copy-and-paste implementation approach is error-prone, leaving teams blindfolded when something went wrong.

Designs were stitched together and with so much variation, that limited troubleshooting to a personalized approach. That previous experience surfaced in mind when I heard about carriers delivering SD-WAN services. I started to question if they could have made the adequate changes to provide such an agile service.

As with most network services, carriers such as Verizon are integrating SD-WAN and security devices together to form SD-WAN services. This is very different from new SD-WAN providers, such as Cato Networks, who wrote their own networking and security code together. Other providers, such as Silver Peak, Viptela (now Cisco) and Velcloud (now VMWare) offer an SD-WAN appliance model approach.

After extensive study, I have come to the conclusion that carriers are simply structured incorrectly to deliver new technologies like SD-WAN. There are five major flaws responsible for this disorganization.

1. Lack of true end-to-end automation

From the business and technology standpoint, carrier-managed SD-WAN providers lack true end-to-end automation, hindering the operational efficiency of daily activities and information management. The management of information and internal change process for customer changes is often carried out by the manual legacy approach, which is both time-consuming and error-prone.

Customers want control and visibility over their service. However, challenges surface in traditional infrastructure environments when the carrier combines different vendor products to form the service. Multiple box-by-box designs compiled of different vendor equipment increases the likelihood of loss of service, not to mention, the associated security risks when the customer wants to make changes. Carriers offer very little flexibility and the short-term contracts are impossible to be negotiated. It is as if you are locked into an inadequate service where you are unable to move at will.

Have you ever tried to open a ticket? I remember demonstrating a well-designed front end graphical user interface (GUI) to a customer. However, it was used in conjunction with the manual approach in the back end; an approach that has drawbacks in today’s fast-paced world.

2. Network cowboy

Within a provider that lacks true end-to-end automation, there is often one team member who is allowed to act as the “network cowboy.” He or she knows all the reasoning behind the bespoke customer designs and is the first point of contact when something goes wrong. Everybody looks up to this person who is sadly missed when not present.

However, the precious customer knowledge is in the mind of the network cowboy, not in a centralized database that can be updated and accessed at will. What happens to the valuable information when the network cowboy leaves the organization? It goes with them and if not documented correctly so does the reasons for the tailored customer designs.

3. Team mix: one architect, provisioner and troubleshooter

There are a number of team members involved in the design, implementation and troubleshooting of the customer networks. A single architect will design the network, a provisioning engineer will implement the configuration and a separate operational team will control the troubleshooting.

As a result, we have the scope for three different team members acting alone, potentially without any structure or guidance. The network will be designed based on personal preference of the designer, which the provisioner may not fully understand. Furthermore, troubleshooting approach will be according to the personal preference based on individual skill and experience.

4. Bespoke network designs

Carrier-managed SDN-WAN often uses a mix of network kludges to pull together the customer network designs. Service chaining and service insertion introduce new layers of complexity that do not follow the traditional forwarding and monitoring rules. Service chaining is difficult to troubleshoot, as it does not adhere to the standard forwarding paradigms.

Third party network services are implemented in a variety of ways, resulting in complex designs. This leads to the box-by-box architectures with many internal and external touch points. Network and security devices are not the easiest to develop communication since there are different ways to connect them. Some support link aggregation control protocol (LACP) and virtual routing and forwarding (VRF), while others don’t. Based on the experience, the supported configuration will be vendor-dependent.

Throughout my working history, I have had many battles with sales teams throwing in network changes. They don’t care about the support, they just want the sale. Sales personnel will go to great lengths to close a deal and will step out of the structured design categories, pushing the problem to the technical teams to be solved with tailored configurations.

Providers often start with good intentions offering gold, silver and bronze design categories but over time stray from the structured templates. This destroys any design structure and reduces velocity, manageability, stability and overall efficiency.

The appliances supplied by vendors act as building blocks for engineers to formulate a design. Vendors offer validated designs; however, these are merely guidelines, nothing is set in stone. Essentially, it is up to the individual engineer as to how he or she designs the network.

There are many ways to design a service. The manual approach guided by the mind of the engineer results in a huge variety of designs and configurations. You will never see two identical networks, even if providing the same service resulting in bespoke network designs.

5. Lack of science to troubleshooting

Factually, there is lack of science in the domain of troubleshooting. Although it has been tailored over the years and molded into an individual’s mind, however, it is personalized and has developed over time depending on that person’s job history and activities. As a result, troubleshooting behavior has become unpredictable. The science of troubleshooting and knowledge gained from former engineers should be centralized and updated with the lessons learned but it never is.

Depending on the experience of the engineer, they may opt for a top-down or bottom-up troubleshooting approach. Personally, I have experienced so many problems when I usually aim for gold straight away and troubleshoot the layer that I feel is impacted. However, the time it takes to fix a problem depends on the individual experience of the engineer.

These challenges leave the door wide open for the loss of service. If human behavior is unpredictable in these areas, can anyone confidently rely on carrier-managed SD-WAN?

Summary

Every organization has different wide area network (WAN) requirements and SD-WAN comes in a variety of forms in an attempt to fit those requirements. There is a lot of noise in this area and one should take a holistic approach and examine how both network and security are affected. My personal opinion is that cloud-based services that integrate both network and security are the correct path to take. When you are in a room with a carrier providing SD-WAN services, please keep in mind the mentioned concerns.