How to help your files speak S3

Among the many clouds in the storage sky, Amazon Web Services is the dominant leader by far. The annual AWS re:Invent trade show is taking place in Las Vegas this week, and given how many enterprises have some data in the AWS cloud, the event is expected to sell out.

It’s no secret that the cloud is one of today’s top storage industry disruptors, significantly reshaping enterprise IT architectures and investments. Cloud object storage has a well-earned reputation for cutting costs since enterprises can avoid both upfront capital and costly operational expenditures by storing data off-premises. However, the challenges of getting data to the cloud means many enterprises are only getting started on adoption. There is much more growth ahead, as the cloud’s elastic scaling can also deliver agility, but so far, most enterprises are relegating cloud storage for archival.

Integration ain’t easy

My company, Primary Data, recently surveyed IT pros about the cloud. Adoption ranked among the top IT challenges, coming in closely behind delivering performance, managing migrations, and budget concerns. This is not surprising, as the majority of the IT professionals surveyed estimate that about 60 percent of their data is cold, not recently used, or “dark,” meaning that its use is unknown. Admins also noted that their biggest challenge to cloud adoption is the difficulty of integrating object storage into existing infrastructure, and the challenge of having to adapt applications to connect with the cloud.

There is significant complexity at the root of these adoption challenges. Most enterprise data is stored on NAS file systems and accessed over NFS or SMB protocols. In fact, many enterprises already have a mix of these protocols in place throughout their infrastructure. That means that today, most enterprise applications don’t speak the Amazon S3 protocol. This issue reaches down into the very heart of IT architecture, significantly complicating cloud adoption.

Another challenge limiting the cloud’s role as an archival tier was also foreshadowed in the storage survey. Migrations are the second-biggest headache for admins today. Data has been anything but easy to move, and the gravity of data means that the cloud has mostly been used for storing data in bulk when it will likely never be used again, or if it can’t be deleted for governance reasons. Fixing data migration pain is critical to transforming the cloud into the flexible, agile resource it is destined to be.

A virtual solution

Luckily, it is now possible to easily connect file and object stores using metadata engine software. A metadata engine virtualizes data by mounting all storage in a global namespace. It provides analytics about data’s use, finally making it possible to create awareness between applications and infrastructure. Different NFS, SMB and S3 protocols can be integrated through the power of virtualization. With a global namespace, heterogeneous data stores can be made simultaneously available to all applications, writing data transparently to object storage as files, which increases the granularity, visibility, and control admins have when managing data across primary, object and public cloud storage.

However, speaking the same language is only half a solution. The other half comes from getting the right data off your performance storage and onto your AWS cloud (or other) object store. Metadata provides application performance telemetry and insight on when files were last opened, edited, modified and more, making it easy to know what data is active and which files are candidates for archival.

When file and object resources are integrated into a global namespace, metadata engine software can then transparently move data between primary, object, and public cloud storage according to IT-defined policies, without disrupting applications. This technology solves the migration problem, once and for all. Data can then move automatically, without the need to plan months or years in advance, since there is no downtime needed or application interruption when data is on the move.

Pick a cloud, any cloud

Given the conflicting challenges of meeting performance and watching budgets, IT professionals don’t have an easy job, but object and cloud storage can help by archiving cold data while preserving performance storage to meet the high expectations for critical applications.

Powerful data management platforms can automatically identify cold data, accelerate cloud transfers, provide seamless snapshots to the cloud, and automate movement of data between on-premises storage and multiple clouds, including granular retrieval of data back on-premises so organizations can use it as a cost-effective active archive. Whether you build a private cloud on-site, leverage services like AWS, Azure or Google Cloud, or adopt a mix of multiple solutions, software can now make it easy for you to get the right data where you want it, and manage data across a single pane of glass.