• United States

IPSec – A swiss army knife of kludges

May 10, 20186 mins
Network SecurityNetworkingSD-WAN

One shoe does not fit all when it comes to SD-WAN providers. There are a variety of ways to tunnel user traffic. Stay clear of anyone who hasn't made the right extensions to IPSec.

man in network room data center tech troubleshooting
Credit: Getty Images

When I started my journey in the technology sector back in the early 2000’s, the world of networking comprised of simple structures. I remember configuring several standard branch sites that would connect to a central headquarters. There was only a handful of remote warriors who were assigned, and usually just a few high-ranking officials.

As the dependence on networking increased, so did the complexity of network designs. The standard single site became dual-based with redundant connectivity to different providers, advanced failover techniques, and high-availability designs became the norm. The number of remote workers increased, and eventually, security holes began to open in my network design.

Unfortunately, the advances in network connectivity were not in conjunction with appropriate advances in security, forcing everyone back to the drawing board. Without adequate security, the network connectivity that is left to defaults, is completely insecure and is unable to validate the source or secure individual packets. If you can’t trust the network, you have to somehow secure it. We secured connections over unsecured mediums, which led to the implementation of IPSec-based VPNs along with all their complex baggage. 

There are a variety of SD-WAN vendors offering traffic segmentation as the core service. However, this can be implemented in a variety of ways. Some implement with the basics of IPSec, others move to datagram transport layer security (DTLS) and companies like Lavelle Networks’ feature set uses a proprietary tunneling mechanism.

From my past experiences, introduction of complexity is the number one enemy of security. All the configuration parameters of an IPSec-based VPN remind me of a Swiss Army knife. It can do many things but no one thing very well. 

Similarly, IPSec falls short in many ways. There are far too many moving parts between control and data planes. It has a long history of interoperability problems where challenges surface with advanced network designs. Consequently, the lack of performance metrics leaves me to rate IPSec as a poor security framework. Let us review the major shortcomings of IPSec that makes IPSec- a Swiss Army knife of kludges.

Many moving parts leads to compromise  

IPSec consists of a collection of protocols that initiate a control channel and then a data channel. Both channels must first be synchronized. As a result, there are many moving parts before the data is securely exchanged.

The internet key exchange (IKE) control plane uses user datagram protocol (UDP) for transport. Like all other control planes, this needs to be established and maintained, creating state maintenance, which itself can be the victim of an attack. 

I remember the first time a network I was involved in was hacked. It was due to the lazy defaults of an unskilled administrator who was configuring a partner’s IPSec-based VPN. Sloppy IKE configurations are heaven for a bad actor, leaving the doors wide open. Once a skilled bad actor compromises the network, the bad actor can easily circumvent the standard defense mechanisms while moving laterally throughout the network. This results in data breaches and exfiltration, which often goes unnoticed for months.

Security professionals must take appropriate steps to ensure the protection of the IKE process. On a network or security appliance, this may include a number of different configuration parameters such as infrastructure ACL or control plane policing (CoPP). 

IKEv1 has been around for a long time and hackers have developed tools to attack its negotiation. IKEv2 is more secure than IKEv1 but it’s not backward compatible. Implementations of IKEv2 are also very recent, so we have few lessons learned from the protocol and the compatibility issues it will have.


Had there been one vendor supplying all appliance and endpoint equipment; IPSec would not have been as problematic. Have you ever tried to create an IPSec-based VPN between different vendors? For example, a Cisco IOS router and a Juniper NetScreen or Apple MAC to a Microsoft Operating System?

From memory, creating a site-to-site VPN between a Cisco IOS router and a Juniper NetScreen appliance stretched my work into the weekends. It created far too much noise and finally, once I had all the ducks in a row, it was flaky and hard to troubleshoot. There are IPSec standards that vendors must follow but they can be implemented in various ways. Many vendors add additional functionality to meet specific requirements, thereby, adding to the complexity. 

I have shot myself in the foot so many times. Certificate fields can be interpreted differently, the mismatch of default parameters, the supporting of different encryption and compression algorithms resulted in headaches and long nights. The more interoperability you have with other systems, the more complicated the setup will be. The complexity can itself become a security risk.

Challenges with advanced designs

IPSec is not good enough when it comes to complicated multi-site interconnectivity and high availability designs. By default, IPSec exhibits static designs, for advanced agile architectures, it needs to be incorporated with other higher-level protocols. The additional kludges required to make IPSec work, further, add to the network complexity, which is a killer for security.

IPSec is point-to-point architecture, not site-to-site. This leads to complexity when you have a dual connected site, with a number of uplinks to different service providers. IPSec does not perform well when you have redundant uplinks. If you require IPSec failover, you need to add in a routing protocol or use some other add-on mechanism. However, these must be integrated properly and therefore, troubleshooting becomes complicated.

There is no doubt that Network Address Translation (NAT) will be somewhere on the network path. NAT changes things like IP address, header checksum, port number, TCP sequence number, breaking the end-to-end connectivity. Traditional IPSec cannot be forwarded through a NAT device and the loss of full end-to-end connectivity can cause interoperability with NAT configurations. There are workarounds but at the same time, they may not be implemented by all the vendors.

I have come across a number of vendor appliances not supporting multicast traffic within the IPSec tunnel. This forced me down the road of using Generic Routing Encapsulation (GRE). GRE adds another layer of complexity with the potential of recursive routing, thereby, causing GRE flaps. Essentially, I would have to configure IPSec for small size networks and GRE over IPSec for medium to large networks.

Lack of performance metric

If you have configured redundant IPSec tunnels, the IPSec tunnel selection is not based on an intelligent metric. However, today’s applications warrant intelligent metrics by default. IPSec does not take into consideration, for example, the quality of the link, latency or packet drops. As long as the tunnel is responding to a simple keep-alive heartbeat, it can only think it’s up and working, hence, missing the key link performance metrics.

Old frameworks like IPSec are not good enough to support today’s agile requirements. Agile networks require a new approach to security and IPSec has too many layers that need to be fully understood. Essentially, instead of hardening the network gates, it potentially opens them due to complex and varied configurations. One shoe certainly does not fit all when it comes to IPSec configuration.


Matt Conran has more than 19 years of networking industry with entrepreneurial start-ups, government organizations and others. He is a lead Architect and successfully delivered major global greenfield service provider and data center networks. Core skill set includes advanced data center, service provider, security and virtualization technologies. He loves to travel and has a passion for landscape photography.

The opinions expressed in this blog are those of Matt Conran and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.