Anomalies in data are everywhere. They appear in the performance of just about anything you can measure. Sometimes, if you’re lucky, they’re rare. Other times they aren’t. The problem in both cases, however, is figuring out what is truly an anomaly and what just appears to be one. This is why we need statistical methods to sort the WTF from the chaff, so to speak.
Now, using stats is fine but when you don’t know exactly how to characterize anomalies, you wind up wasting a lot of effort identifying the correct and then optimal methodologies to identify the stuff you need to find which is why handing over the problem to a computer is a much better solution.
But what computer technology to use? Numenta, founded by Jeff Hawkins, the founder of Palm Computing turned artificial intelligence researcher, argues that the best solution lies in artificially intelligent system that perform complex pattern detection, use automated modeling, and implement adaptive learning, all of which is the basis for the company’s new offering: Grok for IT.
Numenta claims their solution to the pattern detection problem:
… leverages sophisticated algorithms to analyze connected datastreams, such as those from AWS CloudWatch. Through complex pattern analysis, Grok identifies abnormal conditions or gradual trends – situations that tools based on thresholds or simple statistics can easily miss.
IT professionals can use these early insights to prevent potential problems while avoiding false positives or annoying alerts.
The company also claims that Grok for IT learns continuously in real-time so as conditions and architectures change and evolve their system adapts to ensure that anomalies are detected and false positives minimized even though conditions have changed. The result is a continuously updated model of “normal” performance that makes anomaly detection efficient and effective.
Grok’s features include the ability to monitor AWS environments via CloudWatch or other streaming data from:
- Elastic Block Store (EBS)
- Elastic Compute Cloud (EC2)
- Relational DB Service (RDS)
- Elastic Load Balancing (ELB)
- Simple Queue Service (SQS)
- AWS AutoScaling Groups
- Other system data via custom metrics
Grok has also created a construct for monitoring a dynamic group of logically related EC2 instances, called an Autostack. Every five minutes for Autostacks, Grok will calculate average CPU usage across all servers, average network input, average network output, average disk read bytes, average disk write bytes.
For the Free Grok for IT plan you get:
- Advanced anomaly detection
- Continuous learning of your systems
- Simple setup and automatic maintenance
- Android user interface
- Custom metrics and auto-scaling support
This plan allows you to monitor a maximum of 200 instances or custom metrics with a dedicated EC2 AWS Instance. The Free Grok for IT plan is available from the AWS Marketplace. At present, no other Grok for IT plans are available.
So, where would Grok for IT make an operational difference? Numenta has a selection of use cases that include abnormal latency changes, bad code pushes, and unusual server access patterns. In fact, Grok for IT should be exceptional for identifying Day Zero attacks as these should involves network behaviors that are anomalous.
Numenta’s Grok for IT product, built upon the company’s novel HTM technology, is one of the most impressive AI products available. If you’re in IT you need to check Grok for IT out; it could make your life much easier.