Log and event management is now a requirement for organizations that need to monitor security and IT policy enforcement, document compliance, and achieve IT operations excellence without increasing head count. However, current approaches to log and security event management force customers to purchase and integrate two or more products for each discipline. This approach is complex, costly, and difficult to deploy and manage for enterprises with large data centers, distributed operations and/or branch offices.

In a typical organization, millions of logs are generated by every system, application and device on the network every day. According to the SANS Institute, logs represent up to 25% of the total data created in a typical enterprise.

While most logs are not important or meaningful, a small percentage are extremely valuable. They contain insights and warnings about the health of the network, security issues, compliance violations and operational problems.

To unlock the value of logs, a new class of appliance has emerged that combines universal log-data collection, analysis, event management, automated report distribution and incident response. They employ a building-block approach that allows organizations to start with a single appliance then add more devices as the number of log sources and volumes grow. A single management console makes expansion seamless.

Click to see: Diagram of how log and event management works

These new log- and event-management appliances perform the following continuous cycle of functions:

* Log collection: Log sources can include servers, applications, databases, firewalls, switches, routers, point of sale (POS) systems and more. Anything connected to the network is likely generating logs. Logs can be delivered to the appliance via standard network-logging protocols such as Syslog and Netflow. They can be pulled from Windows hosts (event logs) and any database compliant with Open Database Connectivity. Logs also can be collected by agents from remote sites and flat-file sources (that is, Web server logs) and forwarded to the appliance.

* Log management: Since log formats are as varied as the log sources themselves, once logs are collected they must be normalized. Log normalization includes classifying logs so they can be correlated, stored, reported on and managed. Normalization is a key step in transforming logs from raw data to valuable information. During the normalization process, the appliance also automatically synchronizes the time stamps of all log entries to single ‘normal time’ for reporting and analysis purposes.