How TLS protocol works

Part 3 of a six-part article: The RFC 2246 document states the following: “The cryptographic parameters of the session state are produced by the TLS Handshake Protocol, which operates on top of the TLS Record Layer. When a TLS client and server first start communicating, they agree on a protocol version, select cryptographic algorithms, optionally authenticate each other, and use public-key encryption techniques to generate shared secrets.

Part 3 of a six-part article:

  1. Configuration and Troubleshooting TLS in Exchange Server
  2. What is Transport Layer Security protocol?
  3. How the TLS Protocol Works
  4. How to configure TLS encryption on Microsoft Exchange 2003 server
  5. Testing and Debugging TLS protocol on Microsoft Exchange 2003 server
  6. What do I do if there is no TLS handshake?

The RFC 2246 document states the following: “The cryptographic parameters of the session state are produced by the TLS Handshake Protocol, which operates on top of the TLS Record Layer. When a TLS client and server first start communicating, they agree on a protocol version, select cryptographic algorithms, optionally authenticate each other, and use public-key encryption techniques to generate shared secrets.

The TLS Handshake Protocol involves the following steps:

• Exchange hello messages to agree on algorithms, exchange random values, and check for session resumption.

• Exchange the necessary cryptographic parameters to allow the client and server to agree on a premaster secret.

• Exchange certificates and cryptographic information to allow the client and server to authenticate themselves. Generate a master secret from the premaster secret and exchanged random values.

• Provide security parameters to the record layer.

• Allow the client and server to verify that their peer has calculated the same security parameters and that the handshake occurred without tampering by an attacker.

Note that higher layers should not be overly reliant on TLS always negotiating the strongest possible connection between two peers. There are a number of ways a man in the middle attacker can attempt to make two entities drop down to the least secure method they support. The protocol has been designed to minimize this risk, but there are still attacks available: for example, an attacker could block access to the port a secure service runs on, or attempt to get the peers to negotiate an unauthenticated connection. The fundamental rule is that higher levels must be cognizant of what their security requirements are and never transmit information over a channel less secure than what they require. The TLS protocol is secure, in that any cipher suite offers its promised level of security: if you negotiate 3DES with a 1024 bit RSA key exchange with a host whose certificate you have verified, you can expect to be that secure.”

TLS protocol has been designed with several security measures. It numbers all the records and uses sequence numbers in the Message Authentication Code (MAC). TLS protocol uses message digest with a key and only with this key can you check the MAC. As mentioned, TLS also protects against several attacks such as “man in the middle” or those which involved downgrade of the protocol to older less secure versions or a weaker cipher.

The message that ends the handshake sends a hash of all the exchanged data seen by both parties. The pseudo random function splits the input data in two halves and processes them with different hashing algorithms (MD5 and SHA), then XORs them together. This way it protects itself in the event that one of these algorithms is found vulnerable.

The Windows Server 2003 operating system can use three related security protocols to provide authentication and secure communications over the Internet:

• Transport Layer Security Version 1.0 (TLS v1.0)

• Secure Socket Layer Version 3.0 (SSL 3.0)

• Secure Socket Layer Versions 2.0 (SSL 2.0)

You can find more in depth information on how TLS protocol functions in Windows 2003 server environment in the following Microsoft TechNet article.

< Previous story: What is Transport Layer Security protocol?How to configure TLS encryption on Microsoft Exchange 2003 server >

> Next story:

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Now read: Getting grounded in IoT