- Silicon Valley's 19 Coolest Places to Work
- Is Windows 8 Development Worth the Trouble?
- 8 Books Every IT Leader Should Read This Year
- 10 Hot Hadoop Startups to Watch
Network World - The buzz in security circles about "big data" goes something like this: If the enterprise could only unite its security-related event data with a warehouse of business information, it could analyze this Big Data to catch intruders trying to steal sensitive information.
Other News: Inside Apple's iPad world-wide ubiquity
This is the security angle to the Big Data hopes that are rising along with the popularity of vast big Data repositories, often based on the open-source scalable software Hadoop, being adopted in enterprises. This is leading to anticipation a new type of "data scientist" job will emerge in IT around Hadoop. Among security professionals and analysts, there's now talk that that Big Data will also lead to security-focused data scientists who will have the tools and knowledge to pinpoint attacks by stealthy intruders out to steal highly sensitive data.
Catching cyber-thieves in the act across sprawling networks has proven hard to do, and "Big Data" is offering new hope. But it is warranted?
Scott Crawford, analyst with consultancy Enterprise Management Associates, thinks so. "Statistical analysts will identify anomalies but not understand the security," he commented during an analysts panel at the recent RSA Conference in San Francisco on the topic of Big Data and how it could help security.
Crawford predicted eventually there will emerge "a market for security algorithms" for big data. He noted firms such as Red Lambda and Palantir are tackling this today in math-heavy analysis aimed at spotting anomalies.
The "bad" attacker intent on hiding is an anomaly to the generally "good" behavior of network users inside the network, behind which the attacker often hides, according to some. Today, stealthy attackers are getting past traditional defenses, such as intrusion-prevention systems, firewalls and anti-virus, pointed out Gartner analyst Neil MacDonald, who spoke about this during the RSA panel.
These devastating attacks to infiltrate and steal highly sensitive data, sometimes called advanced persistent threats (APT), are driven by human actors able to effectively hide their malevolent presence within networks. Today, says MacDonald, we just don't know what "goodness" and "badness" looks like in terms of network activity. "You have to know what goodness looks like" to understand "deviations from goodness," he points out.
Big Data is offering new possibilities for security analysis, which could mean that one type of security tool used today, security information and event management (SIEM), and tools like it that may not properly adhere to that genre, will have to evolve, analysts contend.
To some extent that has started already today, says MacDonald, pointing to RSA's threat-detection product NetWitness and the HP ArcSight SIM, among others. Some start-ups, including CrowdStrike, are claiming they will tackle the APT problem in new ways.
But will SIEM evolve to be able to process business-related big data or not? And is the whole idea that business data be added into more traditional SIEM data from a variety of firewalls, servers, IPS and the like to provide meaningful intelligence on an attacker simply a pleasant illusion?