RE: Network World NAC Test Results, did Cisco get a fair shake?

It was kind of like reading a review on The best Italian restaurants in America only to find out that the winner was chosen because they also served Indian and Thai food. Hmmm?

Being that I authored the book on Cisco NAC Appliance (NACA) I take a personal interest in analyzing any bake off results dealing with NAC products. I am sure my comments will be controversial (oh well) and this response is my personal response not that of my employer. Thus, here is my personal analysis of Network World’s NAC test results.

If you haven’t seen the test yet go here: http://www.networkworld.com/reviews/2007/073007-test-nac-main.html.

Bottom line is that this test ranked NAC “god box features” way above real world customer NAC requirements. It seems that the test was really about finding the NAC vendors who stuffed the most widgets into a single box and call it NAC. Rather than finding the NAC vendors that excel at performing the commonly accepted NAC functions, authentication, posture assessment, quarantine, remediation, and reporting.

First let’s start with their overall vendor ranking results. They ranked Symantec on top and Cisco NACA almost at the bottom. This result just doesn’t add up with the other industry data, reviews or customer surveys out there. For example, NACA won the 2007 product GOLD award at searchnetworking.com (see here http://searchnetworking.techtarget.com/productsOfTheYearWinner/0,296407,sid7_gci1244774_tax306254_ayr2007,00.html ) Symantec wasn’t even a finalist. And if you look at customer satisfaction surveys or poll data Cisco’s solution consistently ranks above the others ( see here http://www.networkcomputing.com/galleries/showImage.jhtml?galleryID=17&imageID=4&articleID=199201086 and here http://www.networkcomputing.com/showArticle.jhtml?articleID=199204304&pgno=7 .)

And finally, Cisco’s NAC Appliance holds a commanding 47% market share in the cluttered NAC space. So I ask you, are that many customers making horrid buying decisions? The NW NAC test results ask us to believe that they are. However, when we look at customer satisfaction rating for NAC Appliance we just don’t see this, in fact we see the opposite. In general, customers are happy with the Cisco NAC solution.

Second, let’s analyze the scoring criteria the NW test relied on. The test weighted authentication less than endpoint security posture. But wait, the endpoint posture checking process depends on the authentication process to tell it what checks it should perform. In fact, almost every NAC process relies on data gathered during the initial authentication process. Without a rock solid authentication foundation on which to build, all other NAC features will suffer. Additionally, a majority of customers consider the ability to enforce user authentication at the network layer to be the most compelling reason to implement a NAC solution.

The NW NAC test criteria didn’t focus on real world customer requirements for NAC solutions. Top of mind issues that drive the need for NAC, like controlling guest access, non-corporate owned PCs, contractors, and rogue devices (like APs), weren’t addressed in any meaningful way. Additionally, the effectiveness of dealing with non-user devices, like IP Phones, was given only passing consideration in the test.

Third, let’s analyze the test bed topology itself. The only deployment method used in the test was inline deployment. Again, real world considerations were not taken into account here. The easiest path, not the most likely path, was taken. Most customers do not want to deploy NAC inline in a LAN environment due to performance and high-availability concerns among others. If given the choice almost all customers would choose an out-of-band solution for wired ports. The NW NAC test doesn’t mention OOB results because they were not tested, in fact they call out-of-band a controversial option. Huh?? If OOB options would have been tested I guarantee you that all of the 802.1x solutions would have performed less than admirably. Deploying 802.1x for wired is riddled with issues on all sides, the client supplicant, the switches need to support it, guest access support, non-dot1x enabled client support, certificates, OS support, the list goes on and on. Can it be done, yes, but it is a huge undertaking with many caveats, the omission of this info from the test docs is telling. Cisco NAC Appliance should have gotten points just for its ability to deploy OOB without the need for 802.1x!
Using OOB can reduce the cost of deployments by requiring fewer servers.

Fourth, where was the focus on remediation at the host? I felt that very little weight or focus was put on the NAC solutions ability to remediate the issues a posture assessment found. This is a critical piece in the real world. A poor remediation solution results in increased, not fewer, help desk calls. Just simply stating that a vendor can provide a link, launch a program, etc is not enough information. More should be said regarding how that information is presented to the user and how it integrates with 3rd party apps like AV, AS, and WSUS.

My final point is that the review failed to focus on, or score, the deployment options available, their functionality, and their ease of use. Most customers ask for and make buying decisions on a NAC solutions deployment flexibility, functionality, and ease of use.

Here are a couple of the things that the NW test team reported incorrectly in their results write-up of Cisco NAC Appliance:

• “Also, Cisco API required to analyze assessment results.” Not true, a full report of passed and failed checks per host is available in the Manager software natively.
• The agent software gathers minimal information about each endpoint - user role as well as IP and MAC addresses." Again simply not accurate, NAC Manager also gathers OS type/version info, user info, machine info, OS fingerprints, AV version info, AS version info, etc.
• The review stated the following "For guests, a captive portal is used for logging in and distributing Cisco’s dissolvable agent."
  See here: http://www.itbusiness.ca/it/client/en/home/news.asp?id=44580

Cisco doesn’t even have a dissolvable agent yet. It’s just hearsay and feature on our future roadmap. Could it be that Cisco’s NAC was never really tested fully? Could it be that Cisco’s configuration and marketing material were heavily relied on for their results? Who knows.

I realize that this is the NAC test reviewer’s first article on NAC and one of her first product review articles in general, but still the review lacks the real world criteria and testing results that NW readers and NAC customers really need. NAC is not supposed to be, nor was ever intended to be, a do everything “god box” single vendor solution. Basing a NAC product review with that as the overriding theme makes the results fatally flawed in my opinion. What do you say?