SIEM tools come up short

We deployed all of the SIEM (security information and event management) products in a live, production environment and ran them over the course of several months. We were both impressed by the depth of features that some of these tools have and frustrated by how far they still need to go.

"Thou shalt review thy logs!"

While it wasn't exactly on Moses's tablets, it's a commandment present in just about every IT standard, audit methodology and federal regulation an IT outfit has to document it has followed. Ticking off that particular checkbox on regulatory compliance forms forces IT to acknowledge that its systems and applications are generating event logs, that it is saving that data, and that it is reviewing it on an ongoing basis.

Archive of Network World tests

In reality, most IT personnel do turn to their logs at some point in time — usually after something bad has happened. But monitoring them 24/7? All entries? Every minute of every day of every week? Um…no. Unless of course you've deployed a Security Information and Event Management (SIEM) platform. In that case, ticking off the "yes" checkbox might be a little closer to the truth. SIEM platforms help get logging and event data from distributed points A, B and C to a centralized point C, help store it, monitor it, report on it, purge it when the time comes, and ultimately — so the pitch goes — provide the situational awareness necessary to effectively manage IT operational risk.

But do they deliver?

In a word: somewhat. It's a crowded market full of players that make many promises. Unfortunately, none of them completely deliver the whole package at this point in time. We currently track more than a dozen vendors that lay claim in the SIEM space and we invited a subset of them to participate in our test. CheckPoint, eIQ Networks, High Tower, Q1 Labs, NetIQ and TriGeo all agreed to participate, while ArcSight, Cisco and RSA all declined for a multitude of reasons. (Compare products.)

We deployed all of the products in a live, production environment and ran them over the course of several months. We were both impressed by the depth of features that some of these tools have and frustrated by how far they still need to go. User interfaces were clunky, reports were incomplete, data parsing problems are still around, and when it came to trying to figure out what the heck was going on in our Windows environment, most products left us scratching our heads. (One could argue, however, that this is as much Microsoft's fault as anyone else's.)

We found the products from Q1 Labs, High Tower and TriGeo to consistently be the most useful. In the end Q1 Labs' QRadar just barely came out on top. While its user interface could still use some work, it is the Swiss Army knife of SIEM tools we tested. It performed all of its tasks required by our testing reasonably well.

With that nod to the top scoring product, truth be told, if we could take High Tower's user interface, combine it with NetIQ's event manager grid tool, grab TriGeo's integration with Splunk for log aggregation, and pull in Q1 Labs' correlation engine, we would then have one heck of a product. In their current form, however, these products still show much room for improvement.

However, selecting the right SIEM product is almost entirely based on the use cases an organization is trying to fulfill. For example, if you're a midsize business without a dedicated team of security analysts, your needs and cost sensitivity will vary greatly from that of a large multi-national firm. You will most likely require a healthy amount of out-of-the-box functionality while heavy customization is probably not on the agenda.

Likewise, if your primary reason for deploying a SIEM tool is so you can click that "yes I review my logs" audit checkbox and you aren't looking at spending a lot of time on ticketing, workflow and advanced correlation logic, your needs aren't going to match that of a full-featured Security Operations Center (SOC). Some organizations might require a ticketing and workflow system to cut and paste event data into an incident "package," where others might simply need reports that show a set of metrics and pretty graphs. Perhaps the day will come when data storage, user interface, monitoring, event-reduction, ticketing, visualization and reporting mechanisms are all relatively comprehensive, but today the products remain heavily varied in coverage for those features.

If you're a small to midsize business it's hard to beat the easy deployment, easy to use, simply priced and feature-rich products from TriGeo and High Tower. TriGeo has a better adhoc query mechanism where High Tower's well-designed user interface makes using it a more enjoyable experience overall.

NetIQ's Security Manager will be attractive to larger customers that already use NetIQ's AppManager product on the IT operations side of the house. Its modular approach allows for both scalability and deployment customization. It is, however, a bit of a beast to deploy. And based on NetIQ's per-server pricing model, the larger your environment, the more you'll pay.

EIQ's SecureVue provides a good set of features for midsize businesses, has visualization components that are actually useful, and offers a very helpful ability to gather device configuration information for change control monitoring purposes. Its user interface, event reduction capabilities and reporting features could all use some work, however, and it's really expensive.

CheckPoint's offering is a relatively new entrant to the space and will undoubtedly make the short-list for existing CheckPoint customers, but lacks some of the features like asset weighting and cross-device reporting that can be found in the more mature products.

An evolving space

Tools in the SIEM space are not new by IT standards. Basic log parsing and alerting have existed for decades, and what many consider the first commercial SIEM products from companies such as NetForensics, Intellitactics and eSecurity (now Novell) came to market in the late 1990s. However, even a decade later the products are by no means fully mature.

Looking at the history of these products, you'll find that many started with very few components. Some had reporting engines without any real-time user interface. Others had real-time user interfaces but didn't have event reduction engines. Still more supported only firewalls and IDSs while others homed in on operating system-centric events.

Today the products have evolved to include common components regardless of the product's heritage. Those components include a data acquisition mechanism, a data storage and archiving system, an event parsing and normalization mechanism, a reporting mechanism, a query mechanism, and usually some sort of real-time analysis module. That list completed, our testing showed that the maturity of these modules across products varied greatly, and the forewarned buyer will give some thought to what features will be most important to their organization. (See related story.) 

Starting with the data acquisition mechanism, all of the products provide (at a minimum) a syslog listener to receive incoming events. However, the maturity of the syslog listeners and the accompanying mechanisms that parse incoming event streams vary widely. For example, NetIQ's product is both inflexible in how it receives Windows events and its mechanisms for gathering syslog data are woefully green. We had to put the NetIQ listener for Cisco ASA devices on one system and another NetIQ listener for our Snort IDS on a different system because the listener couldn't handle data streams from multiple device types. This approach would create a nightmare if the network comprises dozens of syslog-based device types. NetIQ says it's addressing this shortcoming in its next revision due out later this year.

By comparison, the more mature listeners and parsers from CheckPoint, High Tower and Q1 Labs allow you to simply point your device – any device – to the appliance and the SIEM platform will automatically accept the feed, identify the format, and figure out which event came from which device of which type (for example, a syslog-based event from a Cisco ASA firewall vs. a Linux host). This is extremely helpful if you happen to have a centralized syslog implementation already in place as you can then "relay" all inbound syslog messages with something like the syslog-ng (Syslog Next-Generation) "spoof source" configuration directive. But even if you don't have a centralized syslog implementation in place being able to point all devices to a single syslog destination helps make device deployment simple.

Other data acquisition features of these products include support for protocols such as CheckPoint's OPSEC LEA, database scraping mechanisms for products from established security vendors such as ISS and McAfee, and proprietary agents that can run on hosts to acquire non-syslog based event data like that found in vulnerability scanner data and Windows event logs. The products from Q1 Labs and eIQ supported the widest assortment of security devices and platforms out of the box but organizations will want to gather their own compatibility requirements when compiling their SIEM evaluation short-lists.

All SIEM products we tested also offer a mechanism for data storage. Most have a general purpose relational database like Microsoft SQL Server or Oracle under the hood, but there's a growing trend toward using simplified, proprietary databases for large volume event storage. The compelling argument is that one doesn't need all of the features of a modern relational database, so it's best to go slim and purpose built for performance. Q1 Labs, for example, uses a proprietary database whereas High Tower uses an embedded MySQL deployment and NetIQ uses a combination of MS SQL and flat-files. We suspect that the proprietary approach will most likely win out in larger-scale deployments, but time will tell.

Another data storage issue to consider is size. Just a few years ago, packing a terabyte or more onto a single appliance was a substantial challenge, but with increased in average processor power, decreased storage costs, and the move to optimized databases, terabyte data stores are more commonplace in the SIEM world. While our testing did not push any of the databases past 512GB, it was evident that the more data you put into these things, the longer your query times could be. But query times were also product-specific. For example, some of our adhoc queries using High Tower's product to search for text strings took minutes to return where the Q1 Labs product was a lot snappier. High Tower acknowledged that this is a known issue and claims it is addressing it this summer.

With modern hardware most organizations won't see a huge performance problem under 1TB of data. However, for those organizations lured by the temptations of large data sets or complex query loads, we highly suggest embarking on performance tests that are more advanced then what we've done in this test, and ideally perform them before making any purchasing commitments.

In addition to storing normalized events in the database back-end, most products also offer the option of storing a copy of unmodified log entries. Some vendors even offer a hashing mechanism to help with evidence admissibility concerns. We've been unable to find an example case where log-based evidence was outright rejected based solely on the lack of a hash (none of the vendors we asked were able to cite a case, either), but it is a feature some organizations are still talking about as a possible SIEM requirement. Given all the other feature areas that are critical to meeting organizations' needs, this was not a top priority in our testing, but we aren't lawyers so you'll want to consult legal counsel regarding how to tackle the issue.

Finally, almost all of the products have a reporting and analysis engine of some sort – ranging from the extremely basic reports found in the High Tower product to the custom reporting engine in Q1 Labs to the more unique features found in TriGeo's integration of the log aggregation tool Splunk and the business intelligence analysis tool QlikView. We delve much deeper into reporting and the other functions that go hand and hand with it – forensics tools and real-time analysis measures -- later in this article.

The reality of deployments

Installation concerns can be a potential headache with just about any technology, but there are a few scenarios that really compound that pain in the SIEM world. The delivery model (software vs. appliance) does make a difference. Our High Tower deployment – an appliance - came up in about 20 minutes. High Tower had the simplest installation process with Q1 Labs, TriGeo and eIQ not far behind. The appliance delivery model is probably the way to go for most SIEM deployments, and all of the products we tested with the exception of NetIQ's come with an appliance option. Because of its reliance on a healthy amount of software that must be pre-installed, it shouldn't come as a surprise that NetIQ takes the cake for the most overbearing installation process; it'll take a day just getting a dizzying array of Microsoft components up and going before you even start in the NetIQ installation.