How to configure TLS encryption on Microsoft Exchange 2003 server

Part 4 of a six-part article:

1. Configuration and Troubleshooting TLS in Exchange Server
2. What is Transport Layer Security protocol?
3. How the TLS Protocol Works
4. How to configure TLS encryption on Microsoft Exchange 2003 server
5. Testing and Debugging TLS protocol on Microsoft Exchange 2003 server
6. What do I do if there is no TLS handshake?

First of all, you must have X.509 server certificate issued by a certification authority like VeriSign. There are many vendors who provide TLS\SSL certificates. They can be valid for one or more years. The cost of the certificate will vary from vendor to vendor and the price will be determined by the level of encryption and the expiration date. The rule of the thumb is that the stronger the encryption of the certificate and the longer it is valid the more expensive it is.

Before you receive the TLS\SSL certificate you will be asked to generate a Certificate Signing Request (CSR) for the server where the certificate will be installed. In order to generate a CSR, you will need to create a key pair for your server. These two items are a digital certificate key pair and cannot be separated. If you lose your public/private key file or your password and generate a new one, your SSL certificate will no longer match. You will have to request a new SSL certificate. Most of the time the company that issues the certificate lists detailed, step-by-step instructions on how to create CSR and how to install the certificate they issue for your server.

After you receive the certificate from the CA you install it on the IIS server which runs on Microsoft Exchange Server 2003. The certificate can also be used for secure Web Outlook session. Once you successfully install the TLS\SSL certificate, you can precede with TLS configuration on the Exchange 2003 SMTP server.

Installation and configuration of the TLS protocol on the Microsoft Exchange 2003 Server is very simple. It can be divided into two steps. First you should configure your default SMTP server and force it to accept TLS traffic from the outside servers that also have TLS enabled. The second step is to configure the appropriate routing group connectors for each domain you want to communicate with using the TLS and enable TLS encryption on each of them.

After you open System Manager, follow these steps:

1. Expand the Administrative Groups and navigate to SMTP virtual server located in the Protocols folder.

2. Right click on Default SMTP Virtual Server and go to Properties.

3. In the “IP address:” field, you should click on the down arrow and change the option from “(All Unassigned”) to a specific IP address of your e-mail server.

4. You should also enable login and select NCSA Common Log File Format. The SMTP log files will help you with debugging and troubleshooting issues with TLS.

5. In the authentication Tab you will be able to assign the TLS\SSL certificate that you purchased from CA and configure the TLS protocol for the virtual SMPT server.

6. Click on “Authentication” button and select the following check boxes:

• Anonymous access

• Basic authentication (password is sent in clear text)

• Requires TLS encryption

• Integrated Windows

You want to force TLS encryption on the incoming mail as well, therefore you should check “Requires TLS encryption” box.

7. Click on “Certificate” button and follow the prompts to assign TLS\SSL certificate to the default virtual SMTP server.

The second step is to configure appropriate routing group connectors for each domain that requires TLS encryption. First you navigate to the Routing Groups folder, expand it and go to the appropriate Routing Group. After you expand it, you will go to Connectors folder and right click it, then select New > SMTP Connector which will open the following properties:

You will have to enter the name of the connector and select the option: “Forward all e-mail through this connector to the following smart hosts.” You will have to specify the IP addresses of the remote domains mail servers. You enter them using brackets and for multiple servers you enter semicolons to separate them. You will also specify the Local Bridgehead for the outgoing mail by clicking on Add button. You will be prompted to select appropriate virtual SMTP servers to be associated with this connector.

Once you configure the Local Bridgehead, you will click on Advance tab to configure TLS encryption for this connector.

You will click on Outbound Security and check the TLS encryption box.

You will also have to define the address space for the secure domain by clicking Address Space tab and Add button. You will select SMTP and correctly enter domain name you wish to communicate with. The domain name has to match the IP address of its SMTP server.

After you configure your TLS protocol for encrypted e-mail communication you must test it. You should never assume that it works just because you configure it correctly on your server. You should always make sure the e-mails are indeed encrypted.

< Previous story: How TLS Protocol Works
> Next story: Testing and Debugging TLS protocol on Microsoft Exchange 2003 server >