"Thou shalt review thy logs!"

While it wasn't exactly on Moses's tablets, it's a commandment present in just about every IT standard, audit methodology and federal regulation an IT outfit has to document it has followed. Ticking off that particular checkbox on regulatory compliance forms forces IT to acknowledge that its systems and applications are generating event logs, that it is saving that data, and that it is reviewing it on an ongoing basis.

In reality, most IT personnel do turn to their logs at some point in time — usually after something bad has happened. But monitoring them 24/7? All entries? Every minute of every day of every week? Um…no. Unless of course you've deployed a Security Information and Event Management (SIEM) platform. In that case, ticking off the "yes" checkbox might be a little closer to the truth. SIEM platforms help get logging and event data from distributed points A, B and C to a centralized point C, help store it, monitor it, report on it, purge it when the time comes, and ultimately — so the pitch goes — provide the situational awareness necessary to effectively manage IT operational risk.

But do they deliver?

In a word: somewhat. It's a crowded market full of players that make many promises. Unfortunately, none of them completely deliver the whole package at this point in time. We currently track more than a dozen vendors that lay claim in the SIEM space and we invited a subset of them to participate in our test. CheckPoint, eIQ Networks, High Tower, Q1 Labs, NetIQ and TriGeo all agreed to participate, while ArcSight, Cisco and RSA all declined for a multitude of reasons. (Compare products.)

We deployed all of the products in a live, production environment and ran them over the course of several months. We were both impressed by the depth of features that some of these tools have and frustrated by how far they still need to go. User interfaces were clunky, reports were incomplete, data parsing problems are still around, and when it came to trying to figure out what the heck was going on in our Windows environment, most products left us scratching our heads. (One could argue, however, that this is as much Microsoft's fault as anyone else's.)

We found the products from Q1 Labs, High Tower and TriGeo to consistently be the most useful. In the end Q1 Labs' QRadar just barely came out on top. While its user interface could still use some work, it is the Swiss Army knife of SIEM tools we tested. It performed all of its tasks required by our testing reasonably well.

With that nod to the top scoring product, truth be told, if we could take High Tower's user interface, combine it with NetIQ's event manager grid tool, grab TriGeo's integration with Splunk for log aggregation, and pull in Q1 Labs' correlation engine, we would then have one heck of a product. In their current form, however, these products still show much room for improvement.

However, selecting the right SIEM product is almost entirely based on the use cases an organization is trying to fulfill. For example, if you're a midsize business without a dedicated team of security analysts, your needs and cost sensitivity will vary greatly from that of a large multi-national firm. You will most likely require a healthy amount of out-of-the-box functionality while heavy customization is probably not on the agenda.

Likewise, if your primary reason for deploying a SIEM tool is so you can click that "yes I review my logs" audit checkbox and you aren't looking at spending a lot of time on ticketing, workflow and advanced correlation logic, your needs aren't going to match that of a full-featured Security Operations Center (SOC). Some organizations might require a ticketing and workflow system to cut and paste event data into an incident "package," where others might simply need reports that show a set of metrics and pretty graphs. Perhaps the day will come when data storage, user interface, monitoring, event-reduction, ticketing, visualization and reporting mechanisms are all relatively comprehensive, but today the products remain heavily varied in coverage for those features.

If you're a small to midsize business it's hard to beat the easy deployment, easy to use, simply priced and feature-rich products from TriGeo and High Tower. TriGeo has a better adhoc query mechanism where High Tower's well-designed user interface makes using it a more enjoyable experience overall.

NetIQ's Security Manager will be attractive to larger customers that already use NetIQ's AppManager product on the IT operations side of the house. Its modular approach allows for both scalability and deployment customization. It is, however, a bit of a beast to deploy. And based on NetIQ's per-server pricing model, the larger your environment, the more you'll pay.