The Trouble with IPsec VPNs, Part#1

Depending on their size and configuration, IPsec VPNs can be relatively easy to design and deploy, even if you are not all that knowledgeable about how IPsec actually works. But, if you don't understand how IPsec works and you don't apply a good troubleshooting methodology, then when your IPsec VPN breaks or doesn't work in the first place, you'll probably have to resort to what I call ‘stab-in-the-dark' (SITD) troubleshooting.

SITD troubleshooting is random, time consuming, and will often not resolve your problem at all- it might even exacerbate it. So, in order to help you save some time and frustration, I thought I'd write a short series of blog posts that will help you fix broken IPsec VPNs in an expedited manner using a step-by-step, easy to understand process.

Before actually jumping into a description of the IPsec troubleshooting process, it is (as previously mentioned) very important to understand how IPsec VPNs work. In this blog post I quickly review some of the major components, before moving on to the actual troubleshooting methodology next time. I'll start with site-to-site IPsec VPNs, and then discuss remote access IPsec VPNs in a later blog post.

In a site-to-site VPN, IPsec tunnels are built between an organization's sites, and all traffic is authenticated and/or encrypted as it passes over the intervening network.

IPsec consists of a number of elements, including the following:

• Cryptographic algorithms: these include MD5-HMAC-96, SHA-HMAC-96, DES, and AES.
• Security protocols: IPsec uses two security protocols, the Authentication Header (AH) and the Encapsulating Security Payload (ESP).

AH is a packet header that provides connectionless integrity, data origin authentication, and optional replay protection. ESP is a packet header that provides connectionless integrity, data origin authentication, optional replay protection, data confidentiality, and limited traffic flow confidentiality (available only in tunnel mode).

• Security associations (SAs): An IPsec SA is unidirectional in nature and defines how traffic for a particular traffic flow is to be protected by IPsec. An IPsec SA is identified by a Security Parameter Index (SPI), and includes information such as security protocol, security protocol mode, cryptographic algorithms, and SA lifetime.
• IPsec databases: IPsec defines three databases to ensure that IP traffic is correctly processed (with regard to IPsec). These are the Security Policy Database (SPD), the Security Association Database (SAD or SADB), and the Peer Authorization Database (PAD).

The SPD specifies traffic that should be protected by IPsec and traffic that should bypass IPsec. The SPD is consulted for all inbound and outbound traffic.

The SAD or SADB contains an entry containing information related to each IPsec SA and interfaces to the SPD to ensure correct IPsec packet processing.

The PAD provides a link between the Internet Key Exchange (IKE) protocol and the SPD. The PAD specifies the range of identities (for example, IP addresses) for which the IPsec device is authorized to negotiate IPsec SAs with a peer; it also specifies how to authenticate a peer

• SA and key management techniques: IPsec allows two methods for the management of IPsec SAs and keys: manual SA and key management and automated SA and key management through the IKE protocol.

One way of managing IPsec SAs and keys is to manually configure SAs and keying material on IPsec peers. Manual configuration of IPsec SAs and keying material is analogous to the configuration of static routes (though much more involved), and just like the configuration of static routes, the manual configuration of IPsec SAs and keying material is not scalable

The IKE protocol allows IPsec peers to dynamically authenticate each other, generate keying material, and negotiate IPsec SAs.

There are two versions of IKE: IKE Version 1 (IKEv1) and IKE Version 2 (IKEv2). IKEv1 is made up of elements of a number of protocols including SKEME, the Oakley Key Determination Protocoll, and the Internet Security Association and Key Management Protocol (ISAKMP).

IKEv1 negotiation is divided into two phases and three modes. In phase 1, IPsec peers establish an IKE SA. This IKE SA is used to protect phase 2 negotiations, which are then used to negotiate IPsec SAs.

IKEv1 phase 1 can be negotiated using main mode (typical for site-to-site VPNs) or aggressive mode. IKEv1 phase 2, on the other hand, is negotiated using quick mode. It is worth noting that the IKE SA negotiated during phase 1 is bidirectional, but IPsec SAs negotiated during phase 2 are unidirectional.

During IKE phase 1 main mode negotiation, the two IPsec peers exchange three pairs of messages, giving a total of six messages. The function of these messages is as follows:

• First pair of messages (messages 1 and 2)- These are used to negotiate IKE policy parameters, such as the hash algorithm, encryption algorithm, and method of authentication. These parameters are specified using the crypto isakmp policy priority command.
• Second pair of messages (messages 3 and 4)- These are used to exchange Diffie-Hellman public values and nonces (random numbers).

The Diffie-Hellman exchange allows the IPsec peers to agree a shared secret key. The nonce values are used as keying material in the calculation of session keys on the IPsec peers.

The IPsec peers now generate the first of four session keys called SKEYID. A further three session keys (SKEYID_d, SKEYID_a, and SKEYID_e) are then calculated using SKEYID. IKE phase 2 keys are derived from SKEYID_d. IPsec peers authenticate and encrypt remaining IKE phase 1 and phase 2 messages that they send to each other using SKEYID_a and SKEYID_e.

• Third pair of messages (messages 5 and 6)- These messages are used to exchange identities and authenticate the IPsec peers to each other.

Phase 1 is now complete, and an IKE SA has been established between the IPsec peers.

The figure below illustrates IKE phase 1 (main mode) negotiation:

When IKE phase 1 negotiation is complete, phase 2 can begin. The purpose of IKE phase 2 negotiation is to established IPsec SAs. These IPsec SAs are then used to protect user traffic as it transits the intervening network between the IPsec peers.

IKE phase 2 negotiation consists of three messages:

• Message 1- This message is sent by the initiator and contains IPsec SA proposals such as encryption algorithm, hashing algorithm, and IPsec lifetime. IPsec proposals (transforms) are configured using the crypto ipsec transform-set command.
• Message 2- This message serves to accept one of the IPsec proposals sent in message 1.
• Message 3- This message serves as an acknowledgment of message 2.

Once phase 2 is complete, your IPsec VPN should be able to transport traffic between your sites. That's assuming everything goes to plan!

The next figure shows IKE Phase 2 (Quick Mode) negotiation:

Next time, I'll describe a methodology for troubleshooting IPsec VPNs in a fast and efficient manner.

Mark.

• 3des
• AES
• DES
• ESP
• ike
• IPSec
• security
• VPN