To thrive in a digital age, businesses must look beyond log data

With the amount of data in the world predicted to increase at least 50-fold between 2010 and 2020, how we store that data has come into sharp focus. Collecting large volumes of raw log data from multiple applications and infrastructure components and sending it to a central location for storage and processing, for example, increases the size and cost of storage. And as the volume of data grows and storage and processing costs increase dramatically, businesses risk undermining the advantages big data brings. Furthermore, the surging demand for data has environmental implications; by 2020, 12 percent of the world’s energy consumption will be taken by our digital ecosystem, and this is expected to grow annually at approximately 7 percent until 2030. 

Data storage hindering business growth

Since the costs associated with exporting large volumes of data from the cloud to an on-premises data center are frequently prohibitive, businesses often store log-collected data locally in the cloud, which requires a considerable amount of space. In an attempt to reduce some of the large volumes of data held, administrators may be forced to decide which logs to erase and which to keep. While this approach can help reduce storage space and costs, it is also inefficient, time-consuming, and prone to human error, meaning valuable and irreplaceable information can be lost from the log data set.

In addition, because log data is collected from a wide variety of vendors and systems, a such as load balancers, other network appliances, servers, databases, and service enablers, it lacks a common schema and structure and can differ from system to system. This is further compounded by the fact that the application developers for these systems decide which events to log, creating huge inconsistencies and potentially abstracting the bigger picture. It is also impossible to access information in real time due to the time it takes to collect data. With 99 percent of IT and business decision makers noticing an increasing pace of change in today’s connected world,  the inability to act in real time presents a major obstacle to success.

Getting smart with data

To address these challenges, we need to fundamentally change our approach.

Forward-thinking enterprises are therefore looking beyond log data to adopt a smart data approach, which distills the essence of the traffic flows, also known as wire data, that traverse their service delivery infrastructure. These traffic flows include IP packets, segments, sessions, and application data streams. The intelligence is derived from these flows at the source of the instrumentation points, and is then compressed into metadata. As a result, businesses have access to the valuable information they need to gain meaningful and actionable insights and can ensure that only the most relevant data is kept. This in turn leads to very high levels of compression and dramatically reduces storage costs by only holding valuable information. Unlike log data, smart data is normalized, organized, structured, service-contextual, and available in real time. Further efficiencies are then driven by the fact that all data is processed, optimized and contextualized at the source, with some of it converted to metadata in real time. This enables data to be rapidly compressed, substantially reducing the volume of data stored by an order of magnitude or more. This efficiency also enables businesses to store smart data for extended periods of time for forensic and back-in-time analysis of past incidents and events.

Another key advantage of smart data is its consistency. While log data is based on information selected by programmers and engineers with a specific perspective of their domain in mind, continuous monitoring of the wire data that traverses the key service performance indicators enables businesses to obtain full granularity. Rather than simply having a select snapshot of sampled information, they instead have complete access to data that is continuously produced based on analysis of all wire data, which is then contextualized to provide real-time, actionable insights across the entire IT infrastructure.

However, while storing log data across all systems for extended period of time may be prohibitive for the vast majority of use cases, using log data in the right context to accomplish a specific IT service, operations, or business management task can be very efficient. Take, for example, in an incident management case related to service performance degradation. Once the intelligence from smart data has identified the service-level root cause based on triage of complex dependencies across the service delivery infrastructure, log data can be effectively used for analysis of the “last mile.” In this case, log data can be pulled in context from the system that was identified as the incident root cause to analyze system performance. The amount of log data in this instance will be manageable, and the intelligence derived from this data will complement the insight obtained from the smart data analysis. Through this approach, businesses can gain maximum visibility into their networks, and thus add much-needed insight into service delivery and business operations.

Powering businesses to advance faster

With smart data proven to reduce storage size, improve coherency and structure, and provide consistent real-time intelligence, its advantages are clear. With more than one-third of IT professionals listing “moving faster” as their top goal for 2018, such time savings will help businesses that plan to harness big data efficiently achieve a competitive edge. When combined with the processing and storage-related cost savings that enable organizations to store data for extended periods of time for forensic and back-in-time analysis, as well as the wealth of deep insights smart data can provide to service, operations, and business management, we expect it to quickly become a mandatory weapon in digital enterprises’ arsenals.