Today I’d like to take a look at a start-up vendor whose product processes massive amounts of log data from a variety of sources, stores it in a highly compressed format, and then analyzes the data to discover malicious activity and understand user behavior.

Addamark’s Omnisight consists of two key components: a log server that provides a high degree of scalability and compression of raw log data, and a log analyzer that provides ad hoc query capability via a Web-based interface. Although compression capabilities vary widely, depending upon the type of log data stored in the system, a typical compression ratio is 10 to 1.

The advantage Addamark’s product has over other types of log analysis tools is the breadth of the data that can be analyzed. Because Omnisight consolidates into a single data store information from a variety of sources - such as messaging systems, Web site logs or firewalls - it’s possible to correlate different types of log data that would otherwise be difficult. For example, Lehman Brothers can now do a query in 10 minutes that used to take eight hours to complete.

Addamark’s system would be useful for administrators who wanted to correlate messaging activity with other types of activity within a company. For example, administrators could correlate employees’ use of the messaging system with use of other enterprise systems to understand how these systems were used together.  They could correlate messaging system activity on weekends with log data from entry door card-locks. Or they could analyze the use of instant messaging and e-mail to gain a better understanding of how both systems are used.

Omnisight is not intended to be a real-time alerting system to let an IT administrator know, for example, when a hacker is breaking into the system. However, it could yield useful data when trying to understand information security procedures and how to improve these procedures.

The company has completed two rounds of venture funding and currently has deployed its technology for 12 customers.

Michael Osterman is principal analyst of Osterman Research.