UPDATE: NSS Labs would like it noted that the rankings in this story are Ms. Smith's summary of the data, and not NSS' report format.
Advanced endpoint security products don’t do you much good if they can be evaded or eat your time by consistently throw false positives. Since enterprises are expected to defend against sophisticated threats and money in the security budget only goes so far, you might be interested in the results from NSS Labs’ testing of 13 security vendors AEP solutions. The results were released during the RSA conference.
According to NSS Labs’ CEO Vikram Phatak, “The AEP test results provide vendor neutral insight and analysis to help enterprises accelerate their decision process and make informed decisions about when to deploy these products to manage their risk posture.”
NSS Labs said, “To determine the efficacy of the AEP products and validate their claims of technological differentiation with proactive blocking and active detection of known and unknown threats, the leading AEP vendor products were examined for their security effectiveness and total cost of ownership.”
The 2017 AEP security value map, which is free to view, gives a general overview of how well 13 different vendors did in the NSS Labs’ group test. It gives you an idea of the security effectiveness for various vendors’ endpoint protection solutions, but doesn’t show exact scores like the $2,500 in-depth comparative report; in addition to overall security effectiveness, the full report also covers results such block rate, total coverage, total cost of ownership per protected agent and per protected Mbps. Individual reports on each AEP solution are available for $750 each.
Two of 13 received “caution” ratings, one received a “neutral” rating, nine AEP solutions were “recommended” and only one was awarded a “security recommended” rating. Let’s start at the bottom, those that scored the worst, and move to the top.
(#13) Malwarebytes Endpoint Security v.1.7.4.0000 came in last place, being rated by NSS Labs with an overall security effectiveness of less than 60% even though it had “no observed evasions.” The exact score, according to ZDNet, was 57.9%, which earned it a “caution” rating. The detailed AEP test report on Malwarebytes is available here.
(#12) CrowdStrike’s Falcon Host was the second worst product, according to NSS Labs, which scored it with an overall security effectiveness rating of 73.2% and a “caution” rating. This caused CrowdStrike to basically freak out over how damaging it would be to the company’s reputation; CrowdStrike attempted to sue NSS Labs to prevent the release of the report. Since the report was released, the U.S. Federal District Court judge obviously did not agree with CrowdStrikes’ claims. NSS Labs’ AEP test report on CrowdStrike is available here.
*Update: CrowdStrike wants it noted that NSS did not complete three of nine tests on its product and to see its blog for more details.
(#11) ESET Endpoint Security 6.4.2014.0 was rated as the third worst, falling below the “average” line and scoring under 90% for overall security effectiveness. The graph shows ESET was “adjusted for evasions.” The detailed report about ESET is available on NSS Labs here.
The remaining security products of 10 vendors were ranked above average.
Granted, RSA is a busy time for security vendors, but at the time of writing this, a few of those with “recommended” ratings had not issued a press release about it. The following scored over 90% and were recommended by NSS Labs: (#10) Kaspersky Endpoint Security Center 10 had “no observed evasions” (detailed report), (#9) Sophos Central Endpoint Advanced and Sophos InterceptX was “adjusted for evasions” (detailed report) and (#8) Fortinet FortiClient v 5.4.1.0840 had “no observed evasions” (detailed report).
(#7) Trend Micro was pleased with its recommended rating by NSS Labs, saying Trend Micro OfficeScan Agent v12.0.1851 received “one of the highest malware protection scores with no false positives.” The vendor noted that scored as “100% effective against exploits and evasion,” but doesn’t mention the exact score handed out by NSS Labs. The graph notes that Trend Micro had “no observed evasions.”
Unlike CrowdStrike which called the NSS Labs’ methodologies “deeply flawed,” Trend Micro said that the NSS Labs’ endpoint protection public test presents “a truly independent and unbiased public test to help customers understand that user protection must be viewed as a comprehensive solution.” The detailed AEP test report on Trend Micro is available here.
(#6) Symantec Endpoint Protection 14 with ATP Endpoint (EDR) V2.2 (detailed report) came in at sixth from the top, with (#5) McAfee Endpoint Security v10.5 (detailed report) scoring slightly better and coming in fifth.
(#4) Invincea was happy with the results, having placed fourth for X by Invincea v4.2.0-387 which was “adjusted for evasions.” The company announced that it had been awarded a “coveted recommended rating.” You may recall that last week Sophos announced its intention to acquire Invincea and integrate the tech into Sophos next-gen solutions. The detailed AEP test results about Invincea are available in this report.
(#3) Cylance also issued a press release after CylancePROTECT 1.2.1410 was rated third by NSS Labs in the “most comprehensive advanced endpoint security public test to date.” The testing “takes into account advanced threats as well as the completely new ways of detecting and preventing them” and only the top tech solutions earn a “recommended” rating.
Cylance scored 99.69% efficiency after evasions were remediated, but the company is concerned “with the penalty weighting of evasion techniques within this test across all vendors. An arbitrary percentage of the test was based on seven evasion techniques, while the other 1,840 advanced tests used accounted for the remainder. We questioned how NSS had come to that skewed weighting score, and we will continue to work with them to improve overall transparency in their testing methodology going forward.” NSS Labs’ detailed AEP test report on Cylance is available here.
(#2) SentinelOne received the second highest rating for SentinelOne Endpoint Protection Platform v1.8.3#31. It scored a “100% block rating against malware and exploits in six of seven tested categories” and had the “leading TCO rating.” SentinelOne CEO Tomer Weingarten said, “We take pride in knowing that our technology can withstand any attack and we have the third-party validation to back our claims.” The detailed report about SentinelOne is here.
And the “winner” coming in first place was Carbon Black. (#1) Carbon Black Cb Protection v18.104.22.16806 was the only product that stopped all attacks. It “achieved 100% block rate and 100% total coverage score” in NSS Labs’ AEP test. It was the only solution which was rated as “recommended for security effectiveness.” The full report on Carbon Black is available here.