Google: DRAM error rates vastly higher than previously thought

A study released this week by Google Inc. and the University of Toronto showed that data error rates on DRAM memory modules are vastly higher than previously thought and may be more responsible for system shutdowns and service interruptions.

The study ( download .pdf ), which used tens of thousands of Google's servers, showed that about 8.2% of all dual in-line memory modules (DIMM) are affected by correctable errors and that an average DIMM experiences about 3,700 correctable errors per year.

"Our first observation is that memory errors are not rare events. About a third of all machines in the fleet experience at least one memory error per year, and the average number of correctable errors per year is over 22,000," the report states.

"These numbers vary across platforms, with some platforms seeing nearly 50% of their machines affected by correctable errors, while in others only 12%-27% are affected."

The median number of errors per year on a Google server that had at least one error ranged from 25 to 611.

A memory error is marked by bits being read differently from how they were originally written. Memory errors can be caused by electrical or magnetic interference or by hardware corruption.

Memory errors are classified as soft errors, which randomly corrupt bits but do not leave physical damage and can be corrected, and hard errors, which corrupt bits (cells) within the DRAM that become a physical defect that repeats data errors. Soft errors are often caused by radiation or alpha particles, which naturally occur in organic materials, including the epoxy that DRAM chips come packed in. Hard errors are most often caused by chip contamination at the manufacturing facility, but they often don't show up in testing and only surface after the memory chip warms after hours of use, according to Jim Handy, an analyst with Objective Analysis in Los Gatos, Calif.

The Google/University of Toronto study included memory from multiple vendors as well as multiple types of DRAM (dynamic random access memory), such as DDR1, DDR2 and FB-DIMM.

The study covered the majority of servers in Google's data centers and was conducted over two-and-a-half years, from January 2006 to June 2008.

While the study focused on servers and stated that error rates are not climbing with the latest, more dense generations of DRAM, the results show that PCs will eventually need error correction codes (ECC) technology as the size of memory chips become more and more dense, Handy said.

ECC on special chips is used to detect and correct errors introduced during data storage or transmission.

Today, DRAM uses 50 nanometer lithography technology but is migrating to 40 nanometer technology. The smaller the bits, the more susceptible they are to soft errors due to normal levels of radiation, Handy said.

For example, while a server with error correction technology can continue to function after a soft error, a PC would need to be rebooted. A hard error would also be corrected each time a processor attempted to read from a bit on a server card, but the DRAM in a PC, because it has no error correction, would need to be replaced because it would cause a system or application using the memory to crash, Handy said.

For more enterprise computing news, visit Computerworld. Story copyright Computerworld, Inc.