The convergence of SIEM and log management

Though Security Information and Event Management and log management tools have been complementary for years, the technologies are expected to merge. Here’s a look at what you can expect in second-generation log management and SIEM solutions.

SIEM emerged as companies found themselves spending a lot of money on intrusion detection/prevention systems (IDS/IPS). These systems were helpful in detecting external attacks, but because of the reliance on signature-based engines, generated a large number of false positives.

The first-generation SIEM technology was designed to reduce this signal-to-noise ratio and help surface the most critical external threats. Using rule-based correlation, SIEM helped IT detect real attacks by focusing on a subset of firewall and IDS/IPS events that were in violation of policy. Traditionally, SIEM solutions have been expensive and time-intensive to maintain and tweak, but they solve the big headache of sorting through excessive false alerts and effectively protect companies from external threats.

While that was a step in the right direction, the world got more complicated when new regulations such as the Sarbanes-Oxley Act and the Payment Card Industry Data Security Standard mandated stricter internal IT controls and assessment. To satisfy these requirements, organizations are required to collect, analyze, report on and archive all logs to monitor activities inside their IT infrastructures.

The idea is not only to detect external threats, but also to provide periodic reports of user activities and create forensics reports surrounding a given incident. Though SIEM technologies already collect logs, they process only a subset related to security breaches. They weren't designed to handle the sheer volume of log data generated from all IT components, such as applications, switches, routers, databases, firewalls, operating systems, IDS/IPS and Web proxies.

With an emphasis on monitoring user activities rather than external threats, log management entered the market as a technology with an architecture to handle much larger volumes of data and with the ability to scale to meet the demands of the largest enterprises.

Chart of SIEM, log management

As companies implement log management and SIEM solutions to satisfy different business requirements, they are also finding the two technologies work well together. Log management tools are designed to collect, report and archive a large volume and breadth of log data, whereas SIEM solutions are designed to correlate a subset of log data to surface the most critical security events.

If you take a look at an enterprise IT arsenal, you’ll likely see both log management and SIEM. Log management tools often assume the role of a log data warehouse that filters and forwards the necessary log data to SIEM solutions for correlation. This combination helps optimize the return on investment while also reducing the cost for implementing SIEM.

In these tough economic times it's likely we'll see IT trying to stretch its logging technologies to solve even more problems. It will expect its log management and SIEM technologies to work closer together and reduce overlapping functionalities.

Next-generation SIEM and log management

One area where the tools can provide needed help is in compliance. Corporations increasingly face the challenge of staying accountable to customers, employees and shareholders, and that means protecting IT infrastructure, customer and corporate data, and complying with rules and regulations as defined by government and industry.

Regulatory compliance is here to stay, and under the Obama administration, corporate accountability requirements are likely to grow. Log management and SIEM correlation technologies can work together to provide more comprehensive views to help companies satisfy their regulatory compliance requirements, make their IT and business processes more efficient and reduce management and technology costs in the process.

IT organizations also will expect log management and intelligence technologies to provide more value to business activity monitoring and business intelligence. Though SIEM will continue to capture security-related data, its correlation engine can be re-appropriated to correlate business processes and monitor internal events related to performance, uptime, capability utilization and service-level management. We will see the combined solutions provide deeper insight into not just IT operations but also business processes. For example, we can monitor business processes from step A to Z and, if a step was missed, we'll see where and when.

In short, by integrating SIEM and log management, it is easy to see how companies can save by de-duplicating efforts and functionality. The functions of collecting, archiving, indexing and correlating log data can be collapsed. That will also lead to savings in the resources required and the maintenance of the tools.

It gets even more exciting when you can apply log-based activity data and security-event-inspired correlation to other business problems. Regulatory compliance, business activity monitoring and business intelligence are just the tip of the iceberg. Leading-edge customers are already using the tools to increase visibility and the security of composite Web 2.0 applications, cloud-based services and mobile devices. The key is to start with a central record of user and system activity and build an open architecture that lets different business users access the information to solve different business problems.

Levin is executive vice president of strategy at LogLogic in San Jose.

Copyright © 2009 IDG Communications, Inc.

The 10 most powerful companies in enterprise networking 2022