Mail and firewalls: Oh my - Part I

By Joel Snyder
Network World on Groupware and Messaging, 03/22/99

A few months ago I was raving about how great Microsoft's SP2 for Exchange Version 5.5 was. However, I spent all week digging bugs out of mail systems where SP2 was installed. Conclusion: SP2 is dangerous and should not be installed if you have a stupid firewall.

Here's why. One of the nifty features that SP2 has is Transport Layer Security (TLS). However, if you have a stupid firewall, then TLS actually causes an interoperability failure. Next time, I'll explain what TLS is and why you want it (and why it's different from SASL, which is another thing you want related to security, and why SASL is not SSL, but SSL is really TLS).

Normally, SMTP servers that work with TLS act something like this:

First, the server advertises that it can do TLS during the HELO/EHLO dialog with the client. So an SMTP client connects to an SMTP server, says "EHLO," and the server responds with, among other things, "STARTTLS." This means that the server can do TLS to encrypt the data stream. The SMTP client sees the "STARTTLS," and if the client can also do TLS, then it may choose to start up an encryption algorithm so that everything from here on in is encrypted. This is all well documented: the TLS protocol in RFC 2246 and SMTP plus TLS in RFC 2487 (ftp://ftp.opus1.com/rfc/rfc2487.txt).

Now, with Exchange Version 5.5 SP2, the Exchange SMTP server advertises that it can do TLS even if the security system hasn't been set up. To do TLS, you see, you need a certain amount of configuration: signed keys, and all that jazz. It's not hard to do, but my experience is that 99% of Exchange managers haven't bothered to finish setting up their Exchange systems.

No problem. In a normal Exchange setup, then, the Exchange server says, "I can do TLS;" the client says "OK, let's do TLS;" and Exchange comes back with the equivalent of "just kidding." So it's a waste of packets, but I'm guessing that Microsoft Exchange engineers couldn't figure out an easy way to turn on and off the TLS stuff inside of a patch kit like SP2 and so that's just the way it is.

Enter the Cisco PIX firewall. The PIX has, among other things, an exceedingly stupid SMTP proxy. This is not unusual --- as far as I can tell, all firewalls out there have exceedingly stupid SMTP proxies. But the PIX proxy is even stupider than most because it likes to eat commands that it doesn't understand.

So here's what happens when you stick an Exchange system (with Version 5.5/SP2 installed) behind a PIX. Remember that the Exchange server always says it can do TLS, even when it can't.

So, a client tries to connect to the Exchange server. The PIX intercepts the connection, turns on its SMTP proxy, and shuffles commands back and forth. The server says, "I will do TLS." PIX passes that on to the client. A TLS-capable SMTP client will see this and say "OK, let's do TLS." The PIX, being a wily little box, promptly eats that command and says to the client "OK." So the client thinks "OK, let's begin to bring up encryption." Now there is no other way out of here: once you've both said, "let's encrypt," you have to encrypt. There is no graceful way out. At this point both sides, to the client's way of thinking, have said, "let's encrypt."

Unfortunately the server doesn't have any idea what the PIX has said on its behalf. And now you have total interoperability failure. Mail cannot flow over this pipe. The SMTP client tries to bring up encryption; the PIX doesn't have any idea how to deal with that; and the Exchange SMTP server couldn't do encryption even if the PIX let the command through, because it hasn't been set up.

So through the good graces of Microsoft giving us a feature with no documented "off" switch, and Cisco for providing an SMTP proxy that does more harm than good, we now have chunks of the Internet unable to send mail to each other.

And you thought interoperability problems were a thing of the past ... not so.