IT pros quantify their organizations' big data initiatives, investments, and priorities
Big data tactics and budgets vary, but IT pros agree data analytics will play an increasingly vital role in business operations.
The hype around big data is feverish, and it’s driving enormous spending growth.
Research firm IDC expects the big data technology and services market to grow at a rate of 27% year-over-year, reaching $32.4 billion in 2017. To put that in perspective, the big data market is growing at a rate that’s six times stronger than the overall information and communication technology market, IDC says.
Though the money is flowing, it’s not always clear whether enterprises are finding success as they move past early adoption stages. To find out, researchers polled IT executives and managers, who shared details about their organizations’ big data initiatives, investments, and priorities.
+ALSO ON NETWORKWORLD: Big data’s biggest challenges | Inside CIO paychecks | 12 jobs getting the biggest raises in 2014 +
We’ve highlighted some key findings of the 2014 Big Data Survey, conducted among the audiences of Network World and five fellow IDG Enterprise publications (CIO, Computerworld, CSO, InfoWorld and ITworld).
The survey polled companies that are in various stages of their big data deployments. Some have already implemented (19%) or are in the process of implementing (25%) big data projects. Others have plans to deploy — over the next 12 months (16%) or within the next 13-24 months (16%). The remainder (23%) is the least far along; these respondents are likely to implement big data projects in the future but are struggling to find the right strategy or solutions.
In the big picture, big data is becoming more important to companies of all sizes and across all industries. When asked what level of importance their organizations place on big data initiatives, 53% deemed their projects either critical or high priority, and another 34% said moderate priority. Just 12% said it’s a low priority.
Here’s what else they had to say:
1. Businesses want to make better decisions … faster.
Why big data? Because it plays a vital role in the quality and speed of corporate decision-making. The two most popular business goals that are driving companies to invest in big data are to improve the quality of decision-making (cited by 59% of respondents) and to increase the speed of decision-making (53%).
Close behind in the ranking of big data objectives are to improve planning and forecasting (47%); develop new products/services and revenue streams (47%); improve new customer acquisition/customer retention (44%); and build new business partnerships (34%).
2. Petabytes of data? A reality for more large companies.
Companies have amassed enormous datasets about their customers and the business. Among all the survey respondents, the average amount of data currently being managed is 164 terabytes. When asked to look ahead and estimate how much data they expect to be managing 12 to 18 months from now, respondents projected an average of 289 terabytes — a 76% increase.
Today, 6% of respondents are already managing more than 1 petabyte of data; that percentage will climb to 14% in 12 to 18 months. Among the largest companies (with at least $1 billion in annual revenue), 31% expect to manage more than 1 petabyte of data.
3. Companies feel the consequences of data overload.
When there’s an influx of large quantities of data, there are corporate consequences. Such as: losing business because of an inability to quickly find needed information (a frequent occurrence for 11% of respondents, occasional occurrence for 31%); a delay in making an important decision (frequent for 14%, occasional for 39%); users becoming overwhelmed by data (frequent for 19%, occasional for 46%); and a lapse in data security (frequent for 4%, occasional for 15%).
4. Companies are prepared to invest; ROI isn’t a major roadblock right now.
Limited budget is the most pressing big data challenge, the survey found. Among respondents, investment tallies vary. At the low end, 19% of respondents said they’ll spend less than $100,000 on big data initiatives in the next year. At the high end, 29% said they’ll spend more than $1 million (including 2% who claim they’ll spend $100 million or more).
The money will go to a variety of different investments related to big data, including:
- Developing or buying software applications (cited by 38%)
- Investing in additional server or storage hardware/software (37%)
- Utilizing an open source software framework (e.g. Apache Hadoop) (30%)
- Migrating to cloud-based services for storage (28%)
- Increasing network bandwidth (27%)
- Migrating to cloud-based services for analytics (26%)
Yet while budget is the most common concern, ROI is less pressing. Just 26% of respondents cited the ability to demonstrate ROI from investments as a major hurdle right now.
5. Companies are feeling the talent crunch.
Companies are worried about whether they have the right people — knowledge workers, data scientists, for instance — within the company to make the most of their big data initiatives. On the personnel front, 34% of companies are hiring staff with analytics skills, and 26% are contracting outside big data expertise.
When asked which skill sets they plan to hire within 12 to 18 months, data scientist (cited by 27%) topped the list, followed by: data architect (24%), data analyst (24%), data visualizer (23%), business analyst (21%), research analyst (21%), director or manager of analytics (19%), and database programmer (19%).
6. Data security is a concern, but it’s not the most pressing.
Limited budget and scarce talent are the two most pressing big data challenges, according to the research. Respondents also cited security issues (35%), development time (35%), legacy issues such as integration of existing tools (33%), poor data quality (32%), and difficulty integrating or analyzing real-time data (30%).
On the topic of security, nearly half of respondents (49%) said their existing security solutions and products provide adequate security for their big data projects. Twenty-nine percent said they’re not adequate, and 22% aren’t sure.
7. What’s harder: business integration or cultural adoption?
Researchers asked how challenging is it to integrate big data initiatives into the organization’s business processes and culture. Based on the percentages of people who said it was either extremely challenging or very challenging, cultural integration (54%) is tougher than business process integration (50%) by a slim margin.
8. IT leads big data initiatives, but success depends on cross-functional partnerships.
IT-led projects are the norm for survey respondents: 46% said senior or executive IT leads big data initiatives at their organization, and 44% said IT management runs their big data projects. But they don’t do it alone: 36% said executive business leaders are also involved. In addition, respondents said their big data efforts are being supported and sponsored at the CEO (cited by 47%) or LOB level (34%). Both IT and business leaders agree that big data projects succeed best when implemented as a jointly owned project designed to solve specific business challenges.
9. Momentum will continue for many companies.
Looking to the future, nearly half (48%) of respondents expect that big data usage will be widespread across the company in three years, while another 26% expect mainstream use at one or more business unit, department or division. Just 5% think their big data plans will fizzle out after the hype dies down, and another 5% aren’t sure. The remaining 16% expect to still be experimenting with big data, but it won’t be in mainstream production.
Ann Bednarz covers IT careers, outsourcing and Internet culture for Network World. Follow Ann on Twitter at @annbednarz and reach her via email at abednarz@nww.com.
