Big data's dirty little secrets

Takeaways from a recent discussion with SAP execs about developing big data trends.

Big Data is one of the hottest trends in enterprise IT right now, and for good reason. The ability to collect, store, and analyze unprecedented quantities of all kinds of data promises to remake everything from weather forecasting to consumer marketing.

But while that promise is being realized at a few pioneering companies, from Amazon and Macy’s to Google and Facebook, many other organizations are still struggling to get the expected returns out of their big data investments.

That wasn’t the theme at SAP’s dinner for journalists and analysts in San Francisco last night, but as I chatted with company execs about SAP’s analytics strategy and what its customers were doing, a few key points came into focus:

Big Data is a concept, not a product. You can’t go out and buy a box of Big Data. In fact, even the concept is a bit fuzzy around the edges. While IBM’s four Vs (Volume, Velocity, Veracity, and Variety) get a lot of attention, that’s still a pretty vague definition on which to hang a huge, transformative business.

Much of what gets called Big Data isn’t really that big. Jason Rose, SAPs VP of Business Intelligence Marketing, noted that only the top 1% of companies are using truly enormous data sets. The vast majority are still working with data sets not much larger than they were a few years ago. Incorporating new data sources like social media adds complexity, Rose said, but once you take out the video and images, the actual amount of new data is relatively moderate. That means many so-called Big Data applications could actually be handled by existing technology approaches, including relational databases.

Big Data is still hard to use. Even as companies like SAP roll out new technology stacks intended to “democratize” the application of Big Data for use by business analysts and even end users, an ongoing shortage of data scientists is trapping Big Data insights deep inside corporations instead of allowing them to use them to make critical decisions. Big Data is getting easier to use every day, but there’s plenty of work to do.

Big Data technology is way ahead of its uses. Many companies are investing in Big Data solutions without a clear idea of what business problems they are trying to solve. Because of the hype surrounding Big Data, companies feel they need to invest now, even if they haven’t yet figured out the business problems they hope to solve. Not surprisingly, not knowing what benefits you’re looking for often leads to disappointing returns on investment.

Culture eats technology for lunch. Jayne Landry, SAP’s Global Vice President of Business Intelligence, noted that just because Big Data makes something possible, that doesn’t mean a given company will – or even should – do it. Landry cited a sports equipment company whose HR department was using Big Data techniques to identify next logical career steps for its talented managers. That sounds great, but the company worried that using the results to suggest career paths might actually turn off many execs.

Right now, I think that’s the biggest Big Data disconnect. Analyzing the data and even identifying the proper actions often turns out to be much easier than changing corporate culture and processes to actually implement those actions, especially when the recommendations cut across multiple parts of the organization to challenge existing stakeholders. Despite the technological issues mentioned above, Big Data technology is way out in front of the cultural and organizational progress needed to fully leverage the insights it creates.

And that, not technology limitations, is Big Data’s most important obstacle.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.