Last August, Greenplum and Aster Data made a very appealing case for enterprise use of DBMS-integrated MapReduce. Despite slow adoption, I still think the case has merit. Monday, however, was a bad night for the MapReduce advocates. First, famed MapReduce skeptics Michael Stonebraker and David DeWitt released a series of benchmarks that suggest MPP database management systems far outperform MapReduce. Computerworld should be posting a related story soon. I piled on by posting some thoughts from even-more-skeptical eBay, which thinks MapReduce is 6-8X slower than MPP database managers for comparable tasks.
That doesn't mean MapReduce advocates need to jump off of a ledge. Much of what these benchmarks show is the should-have-been obvious point that MapReduce shouldn't be used to replace DBMS for tasks DBMS are good at. MapReduce applications tend to be concentrated in four areas:
and the benchmarks didn't really speak to any of those. But some of those areas may equally fall victim to the "Don't reinvent the wheel argument." For example, LinkedIn is one of the more famous users of MapReduce for text processing, but LinkedIn's text processing is ghastly.
MapReduce is surely an appealing paradigm for lightweight, reliably-parallel programming. At least for research into parallel algorithms, it has much to recommend it. But whether MapReduce will play a major role going forward in production use seems at this point to still be an open question. (Facebook and Cloudera certainly think it will.) Stay tuned for further research.
Curt Monash is a leading analyst of and strategic advisor to the software industry. Praised by Lawrence J. Ellison for his "unmatched insight into technology and marketplace trends," Curt was the software/services industry's #1 ranked stock analyst while at PaineWebber, Inc., where he served as a First Vice President until 1987. He subsequently co-founded Evernet, Inc., a $40 million networking systems integrator. Since 1990, he has owned and operated Monash Research, an analysis and advisory firm covering software-intensive sectors of the technology industry. In that period he also has been co-founder, president, or chairman of several other technology startups.
Curt has served as a strategic advisor to many well-known firms, including Oracle, Microsoft, SAP, AOL, CA, and Netezza. Curt earned a Ph.D. in mathematics (Game Theory) from Harvard University. He has held faculty positions in mathematics, economics and public policy at Harvard, Yale, and Suffolk universities.