Splunk powers enterprise search with Google-like scalability

Enterprise IT search vendor Splunk this week introduced a revamped version of its flagship product that company executives say will enable IT managers to more quickly search network device and systems logs using commodity hardware.

Splunk emerged with an interesting concept years ago when it delivered an IT troubleshooting product that automated the process of collecting logs from devices and systems across an enterprise environment. Now company executives say the product, Splunk 4, required a major update to enable the level of search needed in today’s enterprise IT environments.

“The original Splunk product was designed as an enterprise product, not unlimited scalability,” says Erik Swan, Splunk CTO and co-founder. “The dramatic improvement in this version is that we cleaned up the architecture around scalability and the individual indexers are efficient and can scale horizontally to no end.”

With this release, Splunk developers took advantage of a software framework introduced by Google: MapReduce. The framework enables computing to be distributed across clusters of computers, similar to what is proposed with grid computing, Swan explains. This distributed process technique takes small units of work and “farms them out to lots of boxes and have them work in collaboration on a problem,” he says.

Splunk says beta users can confirm that Splunk 4 can perform searches up to 10 times faster and increase indexing speeds twofold, helping IT managers more quickly resolve performance problems, troubleshoot errors and ensure compliance -- without investing in expensive servers.

“We re-architected our search language so it works in that manner so customers can use fewer boxes and commodity hardware to search terabytes of data more efficiently,” Swan says.

The vendor also updated several usability features in Splunk 4, which the company says will enable non-technical end users to take advantage of the software’s search capabilities. This version of the software also offers customers the option to develop and customize Splunk 4 with additional applications and dashboards. The vendor also works with partners such as F5, VMware and Blue Coat to incorporate IT device and system data from additional devices in enterprise searches.

Available immediately, Splunk 4 is priced starting at $7,500. Free trials are also now available for download here. 

July 31, 2009 will mark the 10th annual System Administrator Appreciation Day. I’d like to know from IT pros what their perfect SysAdmin Day would entail from start to finish. How can companies show you their appreciation? How do you want to spend this year’s SysAdmin Day? Let me know at ddubie@nww.com.

Do you Tweet? Follow Denise Dubie on Twitter here. 

Denise Dubie is senior editor with Network World.