In search of agility and low overhead, companies are putting as many applications as practical in the cloud. But the resulting hybrid IT environments, where certain applications remain on-premise for security or other reasons, can result in data integration issues that reduce efficiency drags and hamper agility.
In fact, cloud integration is much more demanding than many people want to believe.
As an applications intelligence company that has built its customer-facing and internal operations primarily on cloud applications within a hybrid environment, AppDynamics has had considerable experience with cloud data integration. Here are the five essential data integration capabilities that any company serious about harnessing the cloud should have in its pocket:
- As businesses move toward a “scattered” business applications strategy, the ability to support a variety of data latencies – all the way up the latency ladder from batch to real time – is more important. Consequently, a variety of latency-influencing capabilities need to be available in a hybrid environment. These can include change data capture (CDC) for capturing changes to source data and sending it quickly to targets, push-down optimization schemes that harness database resources to process data transformations, and grid-based deployment that can scale to any number of processing nodes.
- Data integration is not just about transporting data. It is about turning data into something meaningful and trusted, and that can and should be seamlessly accomplished inline as your data is being processed. This can include data profiling for understanding data usefulness, including its condition, structure, relevance and relationships; data quality, including data validation, matching and cleansing; and pivoting (e.g., pivoting records from multiple rows to a single column).
- It is all-too-easy to violate the integrity of an application (particularly in a hybrid environment) when you do not understand the relationships between data objects. Effective integration requires knowing what data elements are needed to ensure a complete data object transaction from the application context (e.g., “an order object is always related to order detail”).
- The goal of lowering overhead and increasing agility by introducing business self-service is what encourages many companies to explore cloud computing. Effective integration requires a business focus, not a technical focus. Application administrators should have the ability to easily create, manage, restart and track sophisticated integrations themselves, without the need for coding and in-depth knowledge of shell scripting and so on.
What is required is an integration solution that can understand the schema and underlying metadata of the cloud application, so data can be exposed in business terms. This empowers business users to directly access the data they need, when they need it.
What is also required is for integration to be enabled within the context of the application itself, creating a seamless user experience for cloud application users. This requires support for native user interface (UI) generation for SaaS applications, such as Salesforce Sales Cloud, so IT does not have to be saddled with writing custom UI code.
This kind of business self-service not only saves considerable time and money, it also encourages low-risk exploration and innovation while helping ensure business needs for data are met correctly.
That said, it is also important to enable collaboration between business users, integration developers and central IT resources to guarantee and accelerate the successful development of any “heavy lifting” integrations. What you don’t want is an integration environment designed solely for developers with little consideration of business user needs. Surprisingly, such developer-centric environments still exist, particularly with Enterprise Service Bus (ESB)-based integration solutions.
The other thing you do not want is an integration environment that allows your business users to run around unfettered, which is something we will cover in the fifth and final core integration capability: hybrid security.
- Security is the gorilla in the cloud chamber. At AppDynamics, we wanted to put as many applications as we could in the cloud, preferring SaaS vendors to those that wanted us to host the software ourselves. There are a few applications that remain on-premise for security reasons, and we made certain our security policies extended to our cloud implementations.
You cannot run a business without security policies. You also cannot allow just any user to launch full API-level queries into your business systems. In the new hybrid world, it’s critical to have a single integration layer for collating data in a secure fashion and for managing who can query the data and what they can get out of it.
It is the only way you can scale an enterprise information environment safely. Security can be built into this layer in the form of secure agent technology, data masking (dynamic data masking for role-based security in production environments, passive data masking for data privacy in development sandboxes) and support for advanced security standards.
Align on these precepts
Cloud integration is a complex undertaking, but you cannot go far wrong if you keep in mind the following precepts:
- You cannot always predict what kind of data your business users are going to need tomorrow to drive decisions and grow the business, so be prepared with a full range of data access, latency and transformation capabilities.
- You will eventually find yourself managing scores if not hundreds of best-of-bread cloud applications, so invest in implementing a flexible, scalable and secure integration layer.
- You can go broke hiring business intelligence consultants and IT programmers to support this kind of scattered application environment, so hand over responsibility for querying data and creating reports to your business users to save both money and time.
- Your head may be turned by solutions with a low upfront price, but nothing beats a low total cost of ownership (TCO).
Like many companies, AppDynamics lacked the time and resources to build the cloud integration infrastructure we wanted, so we looked to commercially available tools and techniques. Some techniques, such as the ESB model mentioned above, are fine for transporting data, but inadequate for integrating, transforming and managing data for business analysis and decision making. We knew we would quickly outgrow such approaches and would have wasted whatever resources we had invested in their implementation.
In the end, we implemented a data integration platform solution, Informatica’s iPaaS (integration Platform as a Service), which provided the five core essentials for cloud integration and aligned us with the precepts above. There were other options with a lower upfront cost, but we were investing for the long run, not the easy sprint. The TCO advantage of a platform becomes apparent very quickly as you integrate more and more applications and their data.
Do not let anyone tell you that cloud data integration is a slam-dunk. There is more to doing it effectively than meets the eye. Select your solution wisely, as it will help drive the long term success of your company.
Shafique is chief customer success officer and senior vice president of Technical Operations, and Brian Hoyt, director of Business Information Systems at AppDynamics.