* RAID 5 rebuilds and other horrors
Today, we continue our discussion of RAID 5. When it comes to RAID 5, rebuild times are affected by four things.
The first parameter is the number of drives in the array: the more drives you have, the longer the rebuild will take. Next, when it comes to RAID stripes, size really does matter: the bigger the stripe, the faster the I/O – and consequentially, the faster the rebuild as well. The third consideration is the size of the drive itself: bigger drives add time to the rebuild process. The final issue is whether or not user I/O continues during the rebuild process; if it does, the process will be much slower than if the system is taken offline so that 100% of the system resources can be devoted to rebuilding the RAID set.
Beyond these four, many systems also allow for “load balancing” between I/O and rebuild operations, but using this feature never comes close to delivering the performance levels users expect to get.
In all cases, the size of the disks and the issue of doing rebuilds while the system is still operational are the two major obstacles.
Keeping in mind that with RAID 5 the loss of a second disk always leads to complete data loss, it is invariably in everybody’s best interest to rebuild the array immediately so as to pre-empt any possibility of data loss. But how is that rebuild likely to impact IT and business operations? The answer may surprise you.
Consider a RAID 5 array with four 9GB disks. If the system cannot be taken offline – and thus you can expect I/O requests to continue while the system is rebuilt – the rebuild will progress at something like a rate of 25Mbytes/sec (obviously this is a system-dependant number, but this is a reasonable, if somewhat optimistic, rate). At this rebuild time of 1.5Gbytes/min, we have a rebuild time of about 24 minutes for the 36GB array, not including any time for setups, finding a spare drive, and so forth.
A few years ago, 9GB drives were pushing the envelope when it came to disk drive density; today of course, drives are much bigger. Now consider a system with the same rebuild speed, but this time the RAID 5 configuration is using four modern 500GB drives. Your four-disk array, now upgraded to a 2TB RAID set, will have a rebuild time exceeding 22 hours.
Next year, my friends in the drive business tell me those drives will each hold a terabyte of data. You do the math.
All this naturally assumes that the system stays operational, and that users are willing to grit their teeth and live with sharply degraded performance. There are of course two major alternatives to that, but in most cases neither is a likely scenario.
Option 1: Keep the system offline during the rebuild, thus making the data unavailable. The line of business managers relying on the data to do their jobs will just love you for doing this, of course.
Option 2: Use data that has been replicated to another array. This first assumes you have a replicated volume (or one that has been backed up very recently and that you can restore relatively quickly), and that you have an addition local array that can be used. In may situations, neither is true.
The overhead for calculating the parity data never goes away. At some point with the new, large disks, it will probably be desirable to find an alternative RAID level for large arrays, one that avoids parity calculations altogether. A RAID 10 (combining RAID 1 and 0) answer may be appropriate in many cases; perhaps even using lower-cost SATA drives. The good news is that this now becomes as much a business decision as a technology-based one, and that all the variables are pretty well known.
The new, larger disks are substantially changing the playing field. If in the future you are intelligent in your decision-making when it comes to matching RAID types to disk and array sizes, you may find that the relatively rare breakdown is at worst a significant inconvenience. If however you are an admin trying to rebuild a very large RAID 5 array, you may simply decide that sometimes it just doesn’t pay to get up in the morning.
