The issue of storage within regulated industries

As I spend more and more time looking at information lifecycle management it’s the compliance issues – a fundamental part of what drives companies to invest in ILM – that seem to get thornier and thornier.  Obviously, I am not alone in my thinking here.  Many of my readers have expressed similar concern over the plethora of regulations and industry guidelines with which they have to comply. 

Even more of my readers have little idea regarding whether or not they are even in a regulated industry when it comes to data-related compliance issues.  What, for example, are the compliance guidelines regarding retention of records in higher education, a reader from Colorado asks?  A reader from Washington State asks about the record-keeping requirements for architectural drawings of public buildings – are there any rules regarding how long they should be kept, and under what level of security?

Consider what is known as “SEC 17a-4,” a regulation from the Securities and Exchange Commission that manages how the U.S. securities industry must store and manage e-mail.  

A glance at the business section of most newspapers shows that SEC regulators take very seriously issues that relate to records retention and messaging compliance. 

In the case of SEC 17a-4 this means that any firm in the business of buying and selling stocks and bonds must have written and enforceable data retention policies. Specifically, that data must be readily retrieved and easily viewed, it must be stored on non-rewriteable media and be indexed to facilitate its easy retrieval. A copy of the data must be safely stored in an off-site location.

There are, of course, lots of ways of going about this – but none of them are necessarily easy.  At the very least, plan on the following:

When it comes to data retention policies, volumes are likely to be defined and retention periods are likely to be set based on individual records within the volume.  Content addresses will verify that records are stored accurately, and will be associated with some mechanism that demonstrates that the record contents have not changed (think content addressable storage – CAS – a term you are likely to hear much more of over the next few years).

For easy retrieval and viewing, indexes will have to be pervasively active.  The controlling application (the e-mail archiving program, for example) must ensure rapid access to indexes and to the records with which they are associated.

Storing data on non-rewriteable media will mean an investment in read-once write-many (WORM) technology to ensure that records cannot be erased or changed during the period required by regulation.  Whether or not the media turns out to be optical, tape or spinning disks will probably have less to do with cost (which has been the historical determiner) than it will with how easily the media answers the previous requirement.

Even the idea of having copies of the data stored in an offsite location is mandated.  This may prove the easiest pill to swallow though, as many of you already are taking advantage of existing vendor capability to mirror to remote locations.

A few high-end providers offer these capabilities already, and some larger firms have already implemented them.  But these regulations apply equally well to mid-tier and smaller firms, and buying enough technology to satisfy these requirements will be a challenge to many of them.  Expect to see vendors roll out some solutions for these groups as well over the next few quarters.