What do I do if there is no TLS handshake?

Part 6 of a six-part article: Just because you checked a few boxes on your Microsoft Exchange Server does not mean that there is secure TLS encryption between your domain and another SMTP server that runs TLS. The golden rule that should guide the actions of any IT professional is testing. You must test your program and configuration before you turn it into production. Without a good test, you will create a disaster for yourself or even for the entire company.

There could be several problems with TLS encryption so the question is, how can you debug these problems.

TLS protocol is a handshake protocol. It means that servers that are TLS enabled will exchange greetings and negotiate communication before they send encrypted data. If the handshake fails, the encrypted e-mail will either be sent without encryption or it will be stuck in your server queue and the sender will receive the following message: Delivery to the following recipients has been delayed. In either if these two cases, you need to find out what is happening with the TLS handshake.

Let’s take a look at how the TLS session is established between the SMTP servers. The TLS protocol resides on layer seven of the OSI model, the same layer as SMTP and Telnet protocols. I will use all three protocols to illustrate how the handshake is established. Because the Telnet protocol resides on the Application Layer of the OSI model, it is one of the best troubleshooting tools available to the administrator. In order to successfully authenticate using telnet session, your request must pass through all seven layers of the OSI model. Once you successfully authenticate with SMTP Port 25, you can eliminate problems related to hardware, network routing, TCP and UDP. You passed all the rings of fire and you are at the top. So, let’s telnet to SMTP server and see what is going on.

First you launch a command session on your workstation and type telnet <IP address of the SMTP server> 25. You should receive an SMTP banner that should look like this:

It basically informs you that the SMTP service is ready and waiting for your next command. The next request you will issue is EHLO <IP address of the SMTP server> and the server will respond with this acknowledgment:

EHLO<server fully qualified domain name> Hello [server IP address]

After you issue EHLO commend the server will respond with the list of supported ESMTP “keywords,” which are also called SMTP service extensions as seen below:

These are the commands that are available and can be executed on the SMTP server to which you just telnet it to. Please keep in mind that not all SMTP servers advertise these SMTP service extensions. Some may only list the server’s domain name, DNS, SIZE, and STARTTLS as seen on the example below. In this case you deal with either a proxy server or some form of secure e-mail gateway.

You can learn more about some of the most common SMTP service extensions by reading the following RFC documents:

8BITMIME 8 bit Data transmission RFC 1652

ATRN Authenticated Turn RFC 2645

AUTH Authenticated SMTP RFC 255

CHUNKING Chunking RFC 3030

DSN Delivery status notification RFC 1891

ETRN Extended Turn RFC 1985

HELP Supply helpful information RFC 821

PIPELINING Command pipelining RFC 2920

SIZE Message size declaration RFC 1870

STARTTLS Transport layer security RFC 3207

STARTTLS is the SMTP service that starts TLS negotiations between the SMTP servers. In order to start TLS negotiations you should type STARTTLS the servers exchange EHLO greetings. You will get a response: 220 2.0.0 SMTP server ready. This response along with the IP address of the SMTP server that you telnet into is logged into the SMTP logs which are on your Microsoft Server. This response indicates that the SMTP servers are ready to start TLS negotiations. Please keep in mind that it does not mean that these negotiations are successful. The server on the other side may reject the handshake for variety of reasons.

For example, you will encounter problems with the TLS handshake when you have two or more virtual SMTP servers on your Microsoft Exchange 2003 server. In this scenario, the TLS handshake will be negotiated between the default virtual SMTP server and the SMTP server to which you are trying to send TLS encrypted e-mail. If you create a dedicated virtual SMTP server for TLS encryption, it will send a STARTTLS request, but the answer could be returned to a default SMTP server which knows nothing of the request that was sent and the TLS handshake will be dropped and a message will be sent unencrypted.

Here is an example of such a situation. Your domain test.com has two Virtual SMTP Servers one is called default-test and has IP address the other is called secure-tls-test and has the IP address You try to send an encrypted e-mail to a remote SMTP server called remote-tls-server.com. The TLS message will be sent from your secure-tls-test Virtual SMTP Server. It will start the STARTTLS service on remote-tls-server.com which in turn will send the negotiation request back to a default virtual server for your domain based on the MX record. So if the MX record for your domain test.com points to, the TLS negotiation will fail because the acknowledgment was sent to a wrong server. It was sent to an IP address of the server that did not initiated TLS negotiations.

The largest source of misunderstanding is that on a mailbox server with multiple SMTP virtual servers, the SMTP virtual servers have some internal mechanism for communicating and selecting a route. In fact, there is no such mechanism. The SMTP simply tells the Windows network stack to provide SMTP with a socket. It does not provide a source IP address to use and, as such, you will notice that the source IP address assigned by Windows will be based on the Windows routing table, not taking into consideration the IP of the SMTP Virtual Server that is delivering the message.

The remote-tls-server.com will send the acknowledgment of the handshake to a virtual server that never sent a request in the first place. Once the handshake drops the TLS encryption is dropped as well and either it will be sent unencrypted or the receiving e-mail gateway will not accept it at all. In either case you have a big problem on your hands.

The troubleshooting will be tedious and in many cases you may be faced with the resistance from the e-mail administrators of the domain to which you try to send TLS encrypted e-mail. They may not want to cooperate because they do not have time, knowledge or because of the internal politics within their organizations.

With this in mind, you should know how you can resolve TLS-related issues using the tools and log files available to you on Microsoft Windows 2003 server. I already reviewed what information you can obtain from the e-mail header and how you can use the telnet protocol to debug issues with TLS. Now we will examine SMTP log files stored on the Microsoft Exchange server, Network Monitor and Message Tracking Center.

The SMTP log files are easy to read for anyone who is familiar in how to send e-mails using telnet session with the SMTP server. The log will show:

  • The IP address of remote SMTP server or a gateway.
  • The FQDN of your e-mail server.
  • Time and date.
  • All the acknowledgments and SMTP services used during the session.
  • The e-mail addresses of the participants.
  • Any encryption negotiations used.
  • Queue status.

The following is the example of the SMTP log: - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?220 SMTP Proxy Server Ready SMTP" 0 27 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "EHLO -?exchange-1.Chi.my-test.com SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?250-ESMTP Server Ready SMTP" 0 22 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "STARTTLS - SMTP" 0 8 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?220 Server ready Ready to start TLS SMTP" 0 35 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "EHLO -? exchange-1.Chi.my-test.com SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?250-ESMTP Server Ready SMTP" 0 22 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "MAIL -?FROM:<AWright@ test.com> SIZE=8453 SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?250 +OK Sender OK SMTP" 0 17 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "RCPT -?TO:<ALONSON@z-test.com> SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?250 +OK Recipient OK SMTP" 0 20 - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "DATA - SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?354 Start mail input, end with "<CR><LF>.<CR><LF>" SMTP" 0 52 - OutboundConnectionResponse [18/Oct/2006:09:43:47 -0600] "- -?250 +OK message queued for delivery. SMTP" 0 36 - OutboundConnectionCommand [18/Oct/2006:09:43:47 -0600] "QUIT - SMTP" 0 4 - OutboundConnectionResponse [18/Oct/2006:09:43:47 -0600] "- -?221 Service closing transmission channel closing connection SMTP" 0 59

In this example you can clearly see that the SMTP server/gateway received a command requesting to initiate TLS hand shake: - OutboundConnectionCommand [18/Oct/2006:09:43:46 -0600] "STARTTLS - SMTP" 0 8 with the acknowledgment that it is ready to negotiate the encryption: - OutboundConnectionResponse [18/Oct/2006:09:43:46 -0600] "- -?220 Server ready Ready to start TLS SMTP" 0 35. However, it does not show if this negotiation was successful. Therefore, relaying only on the SMTP log you may conclude that the e-mail was encrypted but in reality the negotiation failed and the message was sent in clear text. This log basically says the e-mail was successfully queued for delivery and there was an attempt to encrypt it.

Now let’s take a look at how you can use Network Monitor to verify that the e-mail you sent was indeed encrypted. The Network Monitor is a packet sniffer and it should be used carefully because it can put a heavy load on the server. You have to turn it on just before you send a test e-mail and turn it off just after you send it. Before you start capturing the packets, you must specify the interface you are interested in. It will be a NIC card associated with one of your Virtual SMTP Servers. You also should set appropriate buffer. Both settings can be found under Capture on the tool bar. The Buffer Size set to 300MB should be more than enough for your test.

After you stop the capture you should find the IP address of the remote SMTP server/gateway to which you sent your test e-mail and apply the filter so that only the packets sent to that server are displayed. You should see the following screen:

After you apply the filter, you will be able to follow the SMTP packets with ease and, again, a good understanding of how the telnet SMTP session works will help you debug TLS encryption using Network Monitor.

In the picture below, we can see that the local Exchange-1 server successfully exchanged the greetings with the remote server called Test. The remote server sent 220 acknowledgment that it is ready to receive the SMTP traffic.

The Exchange-1 server sent an EHLO command to the Test server:

And the Test server successfully replied to EHLO by sending the information about the SIZE of the message that it supports and it also started STARTTLS service because the request came from the server that has TLS encryption enabled.

The next command issued by the Exchange-1 server is MAIL FROM: and here is the first indication that the e-mail that is being sent is not encrypted and the TLS handshake failed because the sender’s e-mail address is not encrypted.

The remote server accepted the sender’s e-mail by sending 250 acknowledgement.

The recipient command RCPT: and the e-mail address of the e-mail recipient was issued by the Exchange-1 server.

The recipient’s e-mail address was accepted by the Test server.

The Exchange-1 server issued DATA command, which was accepted by the test server and here we can see the header information and e-mail ID number which you can plug into the Message Tracking Tool on Microsoft Exchange 2003 server:

Once you scroll down, you will be able to see the content of the e-mail and in this case it was: Test, test, test, test.

The test server acknowledged that it received the message:

As we could see in from the information provided by the Network Monitor, the TLS handshake negotiations between servers Exchange-1 and Test failed and the message was sent in clear text. The negotiations were initiated correctly by both servers, but the Exchange-1 server failed to respond to the acknowledgement that was sent from the Test server.

Therefore, the problem must lie with the Exchange-1 server. The TLS encryption on the Exchange-1 server could have been misconfigured. The Exchange-1 server could have two Virtual SMTP servers, one configured as a Default SMTP server and the other could have been configured as a dedicated secure TLS Virtual SMTP server, and all TLS enabled connecters used it to route TLS encrypted e-mails.

1 2 Page 1
Page 1 of 2
SD-WAN buyers guide: Key questions to ask vendors (and yourself)