Few data scientists happy with current state of 'big data' analytics

A poll of attendees of the largest gathering of statisticians and data scientists in North America reveals a near-unanimous belief that "big data" analytics needs improvement.

Conducted by Revolution Analytics at the Joint Statistical Meeting held in Miami from July 30 through Aug. 4, the survey shows that 97% of data scientists believe "big data" analytics technology currently is falling short of enterprise needs.

STUDY: World's colleges lack BI, analytics concentration

The other 3% believe global warming is a hoax.


Specifically, the 200 or so scientists surveyed highlighted three obstacles to running analytics on big data:

* the inherent complexities of big data software

* problems applying valid statistical models to the data

* a general lack of insight into what the data means

One slight hitch in the survey is that there appears to be no consensus on the definition of "big data." According to Revolution Analytics, some data scientists consider the "big data" threshold to be a terabyte, some say it's a petabyte, and some consider "big data" to be "just above what can be reasonably managed for any given job."

Now that's a flexible definition.

Commenting in an official statement announcing the survey results, Revolution Analytics COO Jeff Erhardt said, "Big data is changing the way we analyze and interpret data. We're now in the age of 'big analytics.'"

Revolution Analytics sells software and services based on the open-source R project for statistical computing. Customers include Google and Bank of America.

Other technologies being used to analyze "big data" include cloud computing platforms, massively parallel processing databases, the Apache Hadoop Framework and distributed databases.

In a June special report on big data, research firm Gartner said efforts by IT leaders to manage "big data" often comes at the expense of other aspects of information management, leading to problems down the road.

Gartner also warned that "big data" also is "heavily weighted toward current issues and can lead to short-sighted decisions that will hamper the enterprise's information architecture as IT leaders try to expand and change it to meet changing business needs."

Learn more about this topic

Watson teaches 'big analytics'

Open source analytics options grow

IBM to invest $100 million for big-data analysis research

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2011 IDG Communications, Inc.