Key findings from research on the network impact of Hadoop implementations
With all the talk about big data, we hear comparatively little about the challenges it’ll pose for the WAN. So it was particularly interesting to hear what one university learned when they ran Hadoop over their WAN.
For the most part, Hadoop implementations tend to run within a single site and for good reason. The Hadoop Distributed File System (HDFS) assumes uniformity in terms of latency and bandwidth among the nodes that will be scheduling the jobs, Joey Jablonski explained to me the other day. Jablonski is the principle analytics architect at Data Direct Networks, a data storage infrastructure provider. WANs are hardly predictable and while Hadoop can be tuned for WAN’s longer delays, said Jablonski, it’ll lower the performance of the Hadoop cluster.
MORE ON HADOOP: Hortonworks’ Hadoop distro debuts
IDC: Explosive growth expected for Hadoop, MapReduce-related revenues
Often, though, organizations need secondary repositories of the data for big data implementations. These repositories are used to break up workloads on the primary database, particularly the tasks that can work with slightly older data or which only need access to a subset of the data. Analytics, for example, are often run outside of the main tier.
Such was the case with the researchers at the University of California, San Diego (UCSD). They represent one of several organizations that participated in the analysis of data from the Large Hadron Collider experiment. The experiment generates petabytes of data used by researchers from all over the world. While the primary data is kept on site, copies of the data are kept at various locations for the use of researchers. The UCSD team is one such location, running a Hadoop storage node, which is connected back to Switzerland with dual 10-Gbps links, using GridFTP to move up to 15 Gbps of data at any one time.
The paper looks at Hadoop performance across some 2000 WAN links and in particular the links run by the UCSD. Some of the findings they reported were:
- Optimize individual data transfers. HDFS cannot support asynchronous writes so a single-session stream per GridFTP transfer is the best solution.
- Out of Order Packets (OOPs) are a big issue. Organizations that use multiple GridFTP streams per transfer will require large buffers to keep all the out-of-order packets. This increases the cost of the hosts as they need more system memory.
- Watch that buffering. The use of local buffer will introduce significant overhead in the system I/O before the data is written to HDFS, which has negative impact on the sustainable throughput.
For complete results, read the paper here.
