What do I do if there is no TLS handshake?

Part 6 of a six-part article:

1. Configuration and Troubleshooting TLS in Exchange Server
2. What is Transport Layer Security protocol?
3. How the TLS Protocol Works
4. How to configure TLS encryption on Microsoft Exchange 2003 server
5. Testing and Debugging TLS protocol on Microsoft Exchange 2003 server
6. What do I do if there is no TLS handshake?

Just because you checked a few boxes on your Microsoft Exchange Server does not mean that there is secure TLS encryption between your domain and another SMTP server that runs TLS. The golden rule that should guide the actions of any IT professional is testing. You must test your program and configuration before you turn it into production. Without a good test, you will create a disaster for yourself or even for the entire company.

There could be several problems with TLS encryption so the question is, how can you debug these problems.

TLS protocol is a handshake protocol. It means that servers that are TLS enabled will exchange greetings and negotiate communication before they send encrypted data. If the handshake fails, the encrypted e-mail will either be sent without encryption or it will be stuck in your server queue and the sender will receive the following message: Delivery to the following recipients has been delayed. In either if these two cases, you need to find out what is happening with the TLS handshake.

Let’s take a look at how the TLS session is established between the SMTP servers. The TLS protocol resides on layer seven of the OSI model, the same layer as SMTP and Telnet protocols. I will use all three protocols to illustrate how the handshake is established. Because the Telnet protocol resides on the Application Layer of the OSI model, it is one of the best troubleshooting tools available to the administrator. In order to successfully authenticate using telnet session, your request must pass through all seven layers of the OSI model. Once you successfully authenticate with SMTP Port 25, you can eliminate problems related to hardware, network routing, TCP and UDP. You passed all the rings of fire and you are at the top. So, let’s telnet to SMTP server and see what is going on.

First you launch a command session on your workstation and type telnet <IP address of the SMTP server> 25. You should receive an SMTP banner that should look like this: