Understanding big data analytics

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.

Now that you have initiatives to migrate to the cloud, extend to the mobile web and integrate all the pieces from the bring your own device (BYOD) movement, the next hurdle is going to be keeping an eye on all these moving pieces.

IT Operations Analytics (ITOA, what some people call Big Data analytics or Advanced Analytics) has been dubbed the solution to bring order to this rapidly growing complexity. Some are asking “What does this term really mean, what’s the big deal?"

Here’s the big deal: the advantage to ITOA is it allows real-time monitoring of huge volumes of data and makes sense of it for you. You can now:

  • Know what’s going on with your full IT / application stack at a glance.
  • Prevent incidents and crises before they happen.
  • Shorten response times to incidents when they do happen through automated responses.

While predictive analytics have been promised by the industry before, the promise is finally being met. Conventional analytics and problem-solving responses generally only respond to events that have occurred. The new generation ITOA solutions, rather than just spewing out mountains of system stats that you have to spend hours combing through after an event has occurred, can send a targeted message, in real-time, warning that a specific KPI (key performance indicator) is deviating from norm.

Will Cappelli, a leading IT operations management software analyst and Research Vice President at Gartner, explains that at the heart of these ITOA systems are pattern matching technologies and analytics engines that can use complex event processing (CEP), machine event and log indexing and search, behavior learning engines (BLE), architecture mapping and discovery, and multidimensional analysis databases.

These technologies (application behavior learning engines in particular) enable the capture and analysis of multiple variables to generate patterns and identify deviations from those patterns - the keys to predictive analytics in the IT application data center stack. An example of these variables include:

  1. Real end-user application experience, such as responsiveness, slowness, availability.
  2. IT infrastructure KPIs, like CPU utilization, database IOPs, active connections.
  3. Naturally occurring load or application usage patterns.

These metrics and usage patterns determine how your system is supposed to behave and then compares that assessment with what is actually going on at any given moment.

Interestingly, once you prevent problems from happening, you can creatively “raise the bar on your IT offense” as well.

Consider the rout of the Denver Broncos by the Seattle Seahawks in the Super Bowl last February. Seattle was known for their best-in-class defense. It was this defense that enabled the Seahawks to shut down Denver, build the confidence of the entire team, and take more risks on offense.

For consumer marketing, social networking, data centers, financial systems – the proper use of ITOA is going to have a huge impact in the next 10 years, as companies learn that they will be able to head off problems before they blow up into full-on crises. They will also leverage analytics to enable data center and application operations to have an increasingly direct impact on business results.

Here’s a real-world example:

A major Internet banking platform was set up so that most alarms would get triggered when systems reached static thresholds like 80% utilization. Advanced analytics were checking on the system in this case too. The analytics software noticed that a CPU was showing 15% utilization when it should be at 5%. Something was amiss. But it wasn’t a crisis yet, so the admins said, “Well, there’s still 85% to go.”

At about 4 p.m., a whole bunch of users started getting off work and trying to deposit their checks and withdraw money at the end of the day. At that point, a request that had been stuck in a loop finally kicked loose. And a whole day’s worth of transactions that had been bottled up behind that clog in the system all came flooding in at once.

Think of it like a garden hose that had been clogged, gradually building up pressure all day, bulging and swelling like in a cartoon – and then just blowing through like an uncontrollable fire hose. The system was down for critical hours, right at the end of the day, when the users were counting on it the most.

Real-time analytics can give you an early warning that you have a kink in the hose and help you locate it and straighten it out, before it washes up everything downstream.

Does ITOA sound like it’s too good to be true? Maybe. But Gartner predicts the sector will grow from essentially zero a couple of years ago to over $2 billion in 2018 global sales. We are also starting to see and hear about more and more real-world use cases. A number of us in the industry have been working on the problem and we are finally seeing the kind of compelling results we've all been looking for.

The three things that you should do now  

* Research. There are a lot of analysts and companies parroting a lot of the right phrases, but who don’t really understand this market yet. They are learning too. Be careful! Take your time. Learn the space. Research the researchers and the software or service providers. Learn what can really be done with Big Data analytics today, and start talking with your own internal teams about how it can apply to your business processes.

* Test drive a real use case. After you’ve done some research, figure out a few real technical or business problems to address. You’ll want multiple use cases because different vendors may be able to help with different problems. The technology may be ready to handle some problems and not others. Just like different screw drivers in the toolbox can’t all be used for all jobs, the same applies here with analytics tools.

Find a real problem to address because it is still early for this sector and implementations can be challenging. You want the results to pay off, generate real value. Working on a real problem – albeit a manageable one – will go a long way to proving whether or not ITOA is going to perform for you.

Once you start applying ITOA in your operations and getting your hands dirty, you’ll learn what it is that you can actually accomplish – and start using what you learn to figure out how you can apply Big Data analytics to the rest of the enterprise as well.

* Get Involved. There are communities springing up all over to discuss ITOA. Hacker Dojos are a good place to start; if there isn’t one in your community it might be a great opportunity for you to start one. Another place to look would be 160 Frontiers, which is an organization that coordinates advanced analytics meetups – particularly in Silicon Valley.

These communities are great places to share knowledge with others in your geographic area about what they’re doing with ITOA, and to work out ways that you might be able to collaborate or share expertise.

Contrary to what you may hear at some industry conferences, at this stage of the game, ITOA is very much not a fire-and-forget kind of solution. In particular, be careful of claims from large integrated software providers who say they have advanced or Big Data analytics and easy ways to integrate it into your environment, as well as smaller vendors who claim their tool can solve all problems. These custom solutions can be more like consulting engagements than software that can be rolled across an organization.

Links and Resources

To learn more about ITOA, and how it is being used, check out these online articles and organizations:

*  This site bills itself as the “big data knowledge platform with big data strategy information, trends, jobs, use cases, tools and big data startups for anyone interested in big data.” It has a lot of links to real-world examples of companies leveraging big data, list of meetups and upcoming events, discussion forums, and much more for CIOs looking to keep current with what’s happening in the world of Big Data.

* Similarly, ITOA Landscape says that they want to be “your leading source of insights and up-to-date information, focusing on the exciting field of IT Operations Analytics.” They have an interesting series of interviews with thought leaders on ITOA, and a gallery of the Top 50 vendors for you to check out. (Full disclosure: Appnomic appears in that gallery. And yes, we are pleased about that.)

* Frontier Real-time Big Data Cloud Meetup Group This Bay Area group meets every Tuesday to discuss the applications and implications of leveraging Big Data. Their motto is: “Big Data is being powered by Cloud Computing, while Cloud Computing is being driven by Big Data; Big Data Analytics must be Scalable, and Cloud Computing must be Elastic. Both spaces still remain to be wild frontier, opportunity is always in the hands of those who are well prepared, and we are here to help.”

* Hacker Dojo is a workspace in Silicon Valley for programmers, hackers and Big Data geeks to get together and try to create something awesome in their spare time. Their mission statement says that they want to “act as a catalyst to enable its members to provide these things to the Hacker Dojo community and beyond. The longer term focus of the Hacker Dojo is to deepen its roots in the community here in Silicon Valley and inspire people around the world.” Cool place to talk with developers and test out your ideas, investigate the edge of big data analytics.

* BMC Software White Paper on the Value of ITOA This white paper reiterates many of the same points we have made in this article, but at slightly greater depth. BMC’s take is that ITOA, at its core, allows “IT staff to gain predictive capabilities to ensure both IT service levels and user expectations are met, no matter where users are located or how they access their services.”

* Gartner report on “Seven ITOA Errors to Avoid” It costs $195 to download this document (unless you are already a Gartner client), but it might be worth it if you are about to purchase an ITOA solution and want to do a last-minute “idiot check.”

* “How to take the mystery out of Wi-Fi using performance management tools” explores ways to figure out whether your local wi-fi network is actually delivering the speeds that are promised on the box.

* The “7 keys to delivering better applications faster” article explores ways to bridge the gap between the business side and the technology side of an enterprise.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2014 IDG Communications, Inc.