SIEM tools come up short

After 10 years on the market, products should be better at reporting, usability and advanced correlation features

1 2 3 Page 2
Page 2 of 3

But there's another critical installation angle: getting the SIEM platform to accept and properly parse the output from devices in your environment. For our testing we used just shy of 30 devices (see How we did it). While this number may not sound very big (and it certainly isn't for a large enterprise) if you have to configure every device entry within a SIEM product, trust us, it's larger then you'd like it to be.

Products from Q1 Labs and High Tower really shine on the device provisioning front as their platforms sport an auto-identification mechanism that simply auto-detects events coming from new devices, make an educated guess on device type based on parsing the inbound data, and then, only prompts the administrator to confirm the finding when they have time. While we had to make some modifications from time-to-time, we found the products' informed guesses to be correct most of the time. By comparison, in NetIQ's Security Manager we had to setup every single device. Painful for us in our small environment, but it would be excruciatingly brutal for anything larger, and possibly unacceptable for many enterprise-class environments.

Support for specific devices is clearly a big deal in the SIEM world. If you want to receive, store and intelligently monitor security events from a wide range of devices and commercial software packages, your SIEM platform has to understand the output of all the sending devices. For example, if you deploy a remote access product and the SIEM doesn't understand the log format you won't be able to easily correlate or report on user authentication attempts. Fortunately in 2008 most firewall, IDS, IPS, and general networking devices are supported in most SIEM platforms. What gets a bit uglier is comprehensive support for operating systems and more specialized applications.

For example, Q1 Labs and NetIQ did a better job on the Windows front than most, but unfortunately all of the products we tested were confused at some point by the obnoxious range of Active Directory authentication events. Linux support was a bit better but still far from comprehensive. The lack of coverage isn't a show-stopper, but it is annoying and it does demonstrate the general immaturity of the SIEM space.

Another consideration when it comes to device support is the ability to monitor the events from custom applications. The push for SIEM systems to interface with applications that are less IT-security specific – take home-grown banking applications as an example -- will only continue to rise. If this prediction pans out, then two elements become more critical: first, flexible agents that can perform tasks such as scraping events from local logs and databases, and, second, extensible parsing mechanisms that can be customized for proprietary applications. Fortunately most of the products we tested are headed in this direction with NetIQ, CheckPoint and High Tower all shipping parsing development toolkits. This is a substantial change from just two years ago when one couldn't even see the parsing logic, much less use a toolkit to edit it.

We put parsing modifications to the test with the help of NetIQ when we modified one of the NetIQ Cisco ASA parsers to address a log parsing problem caused by a recent Cisco software update. The process is doable but keeping your regular expression skills sharp is highly advisable if you're going down the path of customizing any SIEM product; every product we tested relied on regular expressions for their parsing logic.

When it comes to transporting data, historically many security practitioners have taken a black-and-white stance in the agent vs. agentless debate. On one side of the argument, no one wants to deploy and maintain yet another agent in their computing environment. On the other side of the argument, agents can provide features such as bandwidth throttling, batch transfers and an improvement over the laughably insecure transport method of syslog over UDP.

It's no longer an either/or issue (even if some vendor documents suggest the black-and-white view isn't dead yet); in some scenarios you won't want an agent, and in others, you might. For example, in our deployment we had an office in India that we wanted to tie into our logging infrastructure. We didn't have a ton of log data we needed to backhaul to North America, but had that been the case we would have wanted to either throttle or batch-transfer logs during local off-hours. Q1 Labs, NetIQ, and TriGeo are shipping agents and High Tower has one in the works, but we wouldn't consider any of them as fully functioning as they should be. If buying an SIEM within the next year, selecting the vendors that are expanding functionality in this area would be a wise move.

The smarts in SIEM

Three of the most critical subsystems of SIEM platforms are the reporting engine, the forensics and investigation system, and the real-time analysis or "monitoring" component. The use cases often seen that drive the majority of SIEM product selections typically involve at least one of these subsystems which is why they were at the center of our testing efforts.

On the analysis and monitoring front, the two areas we focused on were methods of event reduction and user-interface usability of the data being filtered up to you. Event reduction is what most of the SIEM vendors tout as their initial value proposition. Anyone who has centralized and then attempted to review their event logs will tell you it's impossible to accomplish the task without some technology to separate the wheat from the chaff.

Even at rates of five to 10 events per second – which is quite low by enterprise standards - you're looking at numbers exceeding 400,000 events per day, a load that will crush even the most battle hardened of security geeks. Ultimately a core function of SIEM is to watch over all of this data and provide the answer to the simplest of questions: what are the few important things that require a deeper look within the event log?

"Correlation" has long been the buzzword used around event reduction, and all of the products we tested contained a correlation engine of some sort. The engines vary in complexity, but they all allow for basic comparisons: if the engine sees A and also sees B or C, then it will go do X. Otherwise, file the event away in storage and move onto the next. We'd love to see someone attack the event reduction challenge with something creative like Bayesian filtering, but for now correlation-based event reduction appears to be the de facto standard.

But even on the correlation front not all engines have been created equal. Q1 Labs' product have the most powerful correlation language that was still intuitive to use. Q1 Labs employs a "building block" model that allows you to layer pieces of logic that are essentially written in English and assemble them into an alert rule. For example, one of the use cases we tackled was the monitoring of login attempts from foreign countries. We wanted to keep a particularly close watch on successful logins from countries in which we don't normally have employees in. To do this, there are a few things that had to be in place: We had to have authentication logs from the majority of systems that would receive external logins (IPsec and SSL VPN concentrators, Web sites, any externally exposed *NIX systems); we had to have the ability to extract usernames and IP addresses from these logs; and, we had to have the ability to map an IP address to a country code. Not rocket science to do without a SIEM, but not entirely trivial, either.

Q1 Labs' QRadar had all of the functionality to do this, and we were able to build a multi-staged rule that essentially said, "If you see a successful login event from any devices whose IP address does not originate from one of the following countries, generate an alert". Because of the normalization and categorization that occurs as events flow into the SIEM, it's possible to specify "successful login event" without getting into the nuances of Linux, Windows, IIS, VPN concentrators. This is the convenience that SIEM can offer.

Most modern SIEM products also ship with at least a minimum set of bundled correlated rules, too. For example, when we brought a new Snort IDS box online, there was a deluge of alerts, the majority of them considered false-positives. Because of useful reduction logic, there was only one alert out of 6,000 that actually appeared on our console across all of the products tested. That alert was based on a predefined correlation rule that looked for a combination of "attack" activity and a successful set of logins within a set period of time.

Our Q1 Labs dashboard claimed an ongoing data reduction ratio of 500,000:1 which sounds about right to us, but an apples-to-apples reduction comparison during our test was not possible because of the variances in supported devices across all platforms. Regardless of correlation method, the "alert vs. event" approach is essential for all products because it allowed us to look for SIEM-generated alerts (the wheat) and only dig into the raw event data (the chaff) when we needed to.

If you're an Apple fan or have any appreciation for good user interface design you're going to be disappointed by every product we tested. High Tower's product was - by far - the simplest and easiest to use: the user interface is laid out relatively cleanly and intuitively, the navigation bar allows you to hop between the logical functional groupings: Incidents, Cases, Assets, Rules, Reports and Administration, and the interface reacts the way you'd expect it to. One user interface, one tool and the designers adhered to basic user interface design principles such as anticipation and coherence (among others).

Unfortunately it went downhill from there: CheckPoint and NetIQ force you to use multiple applications, Q1 Labs' interface was useable only once you got familiar with it, TriGeo's interface was acceptable but not stellar, and eIQ violated a healthy share of basic design principles.

After using these products for a few months we found areas that were slightly annoying at the first go, only grew more aggravating over time. For example, if when performing basic searches on things such as usernames and IP addresses you have to walk through at least six menus (NetIQ), after a few dozen times of repeating meaningless steps it's hard not to get frustrated.

In comparison, if you're in the middle of creating an incident ticket and you can add a piece of event data with nothing more than a right click (High Tower) you're going to really appreciate that convenience. The bottom line is that if you're going to use a component in the products for any reasonable amount of time you'll want an intuitive, easy to use user interface that doesn't frustrate you.

On the reporting front, the two most common reporting scenarios we've seen typically involve scheduled reports that are relatively static and adhoc queries that are typically event driven or used in forensic situations. Static reports often include top-10 lists, incident summaries, items that exceed normal environment thresholds, general user access reports, and pieces of compliance checklists, to name a few. Adhoc queries are typically driven by an investigative action: looking for where user X logged in from, how many places we've seen IP address X, or taking a look at all login activity in a certain region over a certain period of time.

On the static reporting front NetIQ and TriGeo delivered the most polished reports, most likely because of wise choices in the reporting technology (Microsoft SQL Reporting Services and QlikView and Splunk, respectively). Q1 Labs had the easiest to use report designer (although the reports didn't look as good), CheckPoint's were sufficient, with High Tower and eIQ holding up the rear. All of the products came with sample reports and provide the user the ability to design custom ones, and all of the products had the ability to schedule reports and deliver them via e-mail.

We used the adhoc querying mechanisms primarily to investigate suspicious activity triggered by correlated alerts, although we could certainly see how it would be useful for other efforts ranging from full investigations to basic troubleshooting. In most cases the adhoc reporting mechanisms were sufficient, and in one case above average: TriGeo (via the Splunk technology mentioned above) made searching for events easy to do, and also delivered some good reporting functionality. Q1 Labs was a close second.

A look ahead

When looking at the future of SIEM products, take into account the major changes that lie ahead for enterprise security management.

For starters, some of the pricing models are obnoxious given the value (or lack thereof) that these systems deliver. Few enterprises in this tight economy are going to be able to swallow the large numbers required for sizeable deployments.

On the reasonable side of the pricing spectrum are High Tower and Q1 Labs, which have gone the simple route by selling appliances that are based on nothing more than the approximate event-per-second (eps) loads that the SIEM platform will be experiencing; if you're looking at approximately X eps you buy appliance model A, if you're looking at Y eps you should look at appliance model B. Prices start at $18,000 and $19,000 respectively.

1 2 3 Page 2
Page 2 of 3
IT Salary Survey 2021: The results are in