But beware of tradeoffs when it comes to documentation, management tools and failover
One alternative to buying expensive storage-area networks or other hardware-based dedicated storage is to deploy open source storage software on existing server hardware. For this test, we evaluated three such open source storage products, GlusterFS 3.3, Ceph 0.72 and Apache Hadoop 2.2.0.
All three did a good job, but as you might expect, there is a tradeoff when it comes to using open source storage. This is a DIY project: the documentation might not be as comprehensive as you might like, installation can be tricky, GUI-based management tools might not be available and if anything goes wrong, you’re pretty much on your own.
We liked GlusterFS for its hashing algorithm, which for the most part eliminates the bottleneck and single point of failure risk associated with products that use centralized management. However, GlusterFS, which is being developed by Red Hat, lacks GUI-based management tools.
Ceph also impressed us with its algorithm model. We also liked how Ceph provided object, block and file storage in one system. However, while Ceph is an interesting product to keep an eye on, it’s not ready for prime time deployment in the enterprise. The vendor does not yet recommend CephFS (the file system) for production environments.
+ ALSO ON NETWORK WORLD 25 free open source projects IT pros will love +
Apache Hadoop is a popular, full-featured product with a nice, web-based management console. Our concern with Hadoop HDFS is the potential bottleneck and single point of failure of the centralized server that stores the metadata. Currently there are ways to manually failover to a secondary metadata server and the vendor is working to make failover an automatic feature, but at the time of publication this feature was not yet available.
Here are the individual reviews:
GlusterFS is a POSIX-like distributed file system currently being developed by Red Hat. In addition to being available as an open source storage solution, Red Hat is also increasingly integrating GlusterFS with Red Hat Enterprise Linux (RHEL) products such as the latest RHEL 6.5 release. GlusterFS can be used with commodity hardware as well as virtual and cloud resources.
It utilizes existing disk file systems like ext3, ext4, xfs and others to store data. GlusterFS, although owned by Red Hat, is not tied to Red Hat OS products; in fact we tested it with Ubuntu. The only requirement as far as the file system goes is that it needs to support extended attributes, which enables users to associate computer files with metadata not interpreted by the file system.
GlusterFS fits between traditional storage solutions like Network Attached Storage (NAS) and more expensive models such as SANs. It essentially allows for linear scaling, taking advantage of aggregated storage and memory. Scaling does not affect the user and with GlusterFS you can add and remove servers on the fly. The minimum server requirement to get started is dual-core CPU with 4GB RAM. For storage you can use JBODs (Just a Bunch of Disks), DAS (Direct Attached Storage). RAID is not required with GlusterFS open source, but RAID6 is required if GlusterFS is used as part of RHEL.
GlusterFS uses FUSE (Filesystem in Userspace) to hook itself into the Virtual File System (VFS) layer. FUSE is a mechanism that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space with the FUSE module providing a “bridge” to the actual kernel interfaces.
Unlike some other open storage solutions, GlusterFS does not use a distributed or centralized metadata model, but instead relies on a hashing algorithm, specifically what the vendor refers to as the Elastic Hash Algorithm, which manages the distribution method of the GlusterFS aggregated storage. It hashes the location (path/file name) using a unique identifier similar to md5sum. This means each time the same location is hashed it will produce the same result. The hash is not applied to a specific ‘brick’ (GlusterFS term for unit of storage) location, but rather a virtual volume. This allows the virtual volume to be applied to multiple bricks.
We tested version GlusterFS 3.3 by creating a basic cluster consisting of two servers running Ubuntu Server version 12.04. We wanted to test the product using a non Red Hat OS and also keep the test environment open source from A to Z. Once we had the OS up and running, we needed to configure the disks before downloading and installing GlusterFS by setting up at least one brick (a unit of storage used as a GlusterFS building block) on each server.
Once the disk was prepared, installing GlusterFS was a straightforward operation that involved issuing a few commands from the command prompt. However, getting to the correct commands for our installation type was not so straightforward, as installation parameters vary significantly depending upon the setup environment (virtual, bare metal or cloud install). We would have liked to see more comprehensive, step-by-step instructions broken out for each type of installation. We also seemed to handicap ourselves a bit by using Ubuntu, since the how-tos and instructions were all Red Hat/Fedora/CentOS-centric.
After completing the installation, we created a trusted pool using our two commodity Ubuntu servers. This was done by issuing a single command on each of the servers that were part of the storage pool. Next we created a GlusterFS volume by issuing a simple “volume create” command. A volume is a group of bricks “passed through” translators and presented to the end user as the actual share.
When creating the volume you specify several parameters such as the number of servers to replicate across and also the location of the storage bricks on each server. Also there are a couple of daemon instances that need to run at least once on all peers in a cluster. Glusterd is the elastic volume management daemon and it runs only once on each peer. The glusterfsd is the brick daemon that manages the bricks, it runs once for each brick and there may be multiple running on a given peer depending on the number of bricks.
Once a GlusterFS volume has been created and started, it can be accessed via NFS, CIFS or with the native GlusterFS client. The native client is FUSE-based and is the recommended method as it provides high performance and transparent failover on Linux clients. GlusterFS volumes can be mounted manually or automatically by configuring the ‘fstab’ file, which applies to either the native client or NFS v3. We installed the GlusterFS client piece on a separate Linux desktop and mounted our test volume manually from the command line. Installing the client requires only a few commands, but first you need to download and install the GlusterFS software on each client.
The GlusterFS Command Console is a command line utility that is used to configure and manage the GlusterFS storage environment. On the server storage side, most of the management is accomplished by issuing various commands at the prompt, using a “gluster” prefix. Commands can either be entered as gluster + the command, or you can switch to the gluster shell by typing “gluster”; that way you do not need to type gluster + the command each time. As an example, adding servers to a storage pool is accomplished with a simple “gluster peer probe servername” command.
Currently, GlusterFS does not come with a native Web or other management GUI, however oVirt, a third-party open-source virtual management tool, provides support for GlusterFS in its current release. There are also some third-party non-GUI tools available, such as the open source Puppet-Gluster, which allows administrators to streamline the GlusterFS installation by automating some of the installation tasks (install packages, partition/format bricks). It can also help manage the glusterd service, open firewalls and create volumes.
GlusterFS supports geo replication, replication over LANs or WANs. GlusterFS claims it can handle up to 72 brontobytes of data. This of course would be a function of the available “commodity” resources such as hard drives large enough as well as adequate network, CPU and RAM capabilities. In the public cloud, GlusterFS is currently supported on Amazon Web Services (EC2 and EBS).
Ceph
Originally developed as a doctoral dissertation and funded by the Department of Energy and National Nuclear Security Administration, Ceph is currently in its fourth release since 2012. Ceph’s Reliable Autonomic Distributed Object Store (RADOS) provides object, block and file system storage in a single cluster. The Ceph Filesystem (Ceph FS) is a POSIX-compliant filesystem that uses a Ceph Storage Cluster to store its data.
+ ALSO ON NETWORK WORLD 26 helpful open source network management tools +
Like GlusterFS, Ceph relies on an algorithm to compute the location of data in a cluster. Both the Ceph clients and OSD daemons use the CRUSH (Controlled Replication Under Scalable Hashing) algorithm, specifically developed for placement of replicated data, instead of relying on a centralized lookup table. Ceph OSD daemons store all data as objects in a flat namespace (e.g., no hierarchy of directories).
Although Ceph is designed to run on commodity hardware, its metadata servers do require quite a bit of CPU power to distribute the load. Ceph recommends quad-core processors for the metadata servers and dual-core processors for the OSD daemon which stores the actual data. As for RAM, 1GB per OSD daemon is recommended on the metadata server and at least 500MB on the OSD daemon itself. However, during recovery operations a lot more memory may be needed and Ceph recommends 1GB per 1TB of storage per daemon. As for storage, a minimum of 1TB on each OSD daemon is recommended.
Ceph recommends a cluster with three storage nodes plus an admin node as a good starting point and that’s what we proceeded with, using several Ubuntu 12.04 servers to create our test cluster. There aren’t too many prerequisites to get started and Ceph provides a handy online preflight checklist. First we needed to install an SSH server (openssh).
We also created a password-less key and copied this to all nodes. It is also recommended that you set up a user with root privileges on each of the nodes. Then it was time to add Ceph packages to the repository on the admin node and install the “ceph-deploy” tool. “Ceph-deploy” is a stand-alone method of deploying and de-commissioning Ceph clusters.
It should be noted that we ran into some issues when trying to run the update and deploy command as the “precise” release packages were not found. In order to move beyond this critical roadblock we had to perform a series of web searches in order to find suitable workarounds, which involved changing some of the configuration files. After some trial and error, we were able to get back on track, but this definitely caused some time delays.
Once the preflight was completed, we configured the Ceph node cluster and installed the Ceph software on each of the nodes. Using a series of commands with the aforementioned “ceph-deploy” tool, we also made one of our nodes an Admin node and two of the other nodes OSD daemons. Once a cluster has been created it can be automatically started with “sysvinit” or “upstart”, depending on which Linux flavor you’re running.
Ceph data is stored and replicated across the cluster dynamically. A Ceph cluster stores data in pools, which is essentially a logical group. The CRUSH map is at the core of a Ceph cluster; it is used to determine how data is stored and retrieved and allows clients to interact directly with the OSD daemons without going through a centralized server. When first setting up a Ceph cluster, a default storage map is created, but for larger clusters you can customize the map to improve performance.
According to Ceph, the most common use of Ceph block devices is to provide block images to virtual machines. Block devices can also be used by kernel modules and cloud-based systems through OpenStack and CloudStack. In order to use a Ceph block device, a Ceph client needs to contact one of the Ceph monitors before it can access the cluster to read and write data. A user can mount Ceph as a provisioned block device using Ceph’s object storage system. The Ceph block device is currently supported on virtualization platforms such as OpenStack and OpenNebula.
In order to use the Ceph filesystem, at least one Metadata Server needs to be created with a “ceph-deploy” command. Importantly, Ceph does not currently recommend using CephFS for production data. Mounting CephFS on a client can be done both manually and automatically using a kernel driver or FUSE (file system in user space). We were able to mount CephFS on one of our test machines by adding a configuration parameter to the file systems table using a command similar to the following:
id=nwwuser,conf=/etc/ceph/cluster.conf /mnt/ceph2 fuse.ceph defaults 0 0
Applications can access the Ceph RADOS (reliable autonomic distributed object store) through software libraries for Java, Python, PHP, C and C++. RADOS also provides a RESTful interface that works with Amazon S3 and Openstack APIs.
Monitoring and extending a Ceph cluster is performed at the prompt using a variety of “ceph” commands. For example, to check on the health of a cluster you can use commands like “ceph health” or “ceph status”. These commands will write out parameters to the screen. This is helpful on the fly, but for the bigger picture we would have liked to see some sort of Web GUI. Ceph does not currently include any sort of GUI tools, but there is a management API available for C and C++ that provides interaction with Ceph monitors and daemons. Third-party vendor Inktank offers a commercial solution named Ceph Enterprise that provides a management GUI (Calamari) together with 24/7 support options.
For the most part we found the Ceph online instructions to be adequate although some additional information for troubleshooting and setup configurations would have been helpful.
Apache Hadoop
Apache Hadoop is an open-source data management framework that provides distributed storage and processing of large data sets across clusters of computers. It can scale from a single machine to thousands of servers. The basic components of the Hadoop project are the Hadoop Distributed File System (HDFS) and Hadoop MapReduce. MapReduce is a framework that works with HDFS to perform high volume data processing. Since our main focus for this review was software-defined storage, we did not test the MapReduce feature.
We tested Hadoop version 2.2.0, the first 2.x general availability release. As for hardware requirements, we did not locate any specific baseline recommended by Apache, but commercial users recommend quad-core CPUs, 8-16 GB plus RAM and 2 TB plus hard drives as a good starting point. Hadoop is built to scale out well on budget friendly equipment and manage hardware failures, so there is not necessarily a need to spend resources on features like redundant power supplies and RAID hardware.
+ ALSO ON NETWORK WORLD 16 of the weirdest places you’ll find Linux +
With the newly released version 2.2.0 in hand we set off to install Apache Hadoop as a simple cluster on a pair of CentOS 6.4 servers. Since Hadoop is written in Java and our servers did not already have Java installed, we went ahead and installed Java 1.7 (OpenJDK). Hadoop communication between nodes requires SSH so we installed openssh and generated a private key. Following the online instructions we made changes to a couple of configuration files prior to starting the actual installation (.bashrc and hadoop_env.sh files to specify the JAVA_HOME directory, tells Hadoop where to look for Java).
After downloading the latest Hadoop files we were ready to do the actual install. The software installation is fairly straightforward and involves unpacking the software on each of the nodes in the cluster or just installing the RPMs. Once the install is completed, there are two types of configuration files (read-only defaults and site-specific configurations) needed in order to get a cluster set up properly. This is especially true when setting up large complex clusters. These configurations are done in a series of XML files and although many of the default values can left as-is, there are installation specific parameters that most likely need to be configured, such as host names and ports.
A Hadoop cluster consists of a namenode which manages the file system metadata and one or multiple datanodes that store the data. The namenode uses a block map to keep track of where each block is stored on each data node. The metadata on the namenodes is organized into directories and files, with files divided into uniform block sizes having a default size of 64MB, but this can be changed in the configuration files to suit specific storage requirements. It should be noted that HDFS is “rack aware”, meaning that not only does the namenode know which datanode the data is stored on, it can also know which rack the datanode resides in and its geographical relation to other datanodes. The datanodes communicate with the namenode through pings and if a datanode fails, the namenode will automatically replicate the data to a different datanode.
Client access to data stored in HDFS is done through a JVM (Java Virtual Machine) requesting access to a file or directory from the namenode. If the namenode approves the request, it will provide information about the datanodes and related blocks where the requested data is stored. The application performs data operations (such as a read, write or delete) directly on the datanode without needing to access the namenode, thus reducing the namenode workload. Storage integrity in Hadoop is achieved through replication across multiple hosts, this removes the need for RAID. HDFS in not currently fully POSIX compliant, but by relaxing some of the POSIX requirements to enable streaming access to file system data, Apache claims HDFS offers better performance.
Various fuse-dfs projects allow HDFS can be mounted as a standard filesystem on most Linux/Unix flavors using the mount command. Most traditional operations such as mkdir, rmdir, cp, rm, cat and mv are supported with HDFS mounted with FUSE; however certain permissions-related operations (chown and chmod among them) are not supported with HDFS and FUSE.
For our review we first installed the NFS utilities on one of our test servers. Then we enabled NFS in the HDFS XML configuration files. This can be done using Ambari or from the command prompt. In order for the HDFS enabled NFS to work, we first needed to stop the native Linux NFS services and then launch the HDFS services. Once these services have been started, HDFS can be mounted on different clients (Linux, Windows, Mac) provided a user with HDFS permissions is created on the client machine.
We mounted HDFS as a file system on a separate Linux client machine and were able to perform file tasks such as copying data off of HDFS onto the local file system. Hadoop can be managed from the command line using hadoop commands. For example, hadoop fsck –locations will print out the location of all blocks. There are also various Web interfaces such as an internal Web server included on each data and name node. These display basic information such as statistics about the cluster. These web interfaces can be used to browse the file system and also view the logs. Then there is Apache Ambari, which can be used to provision, manage and monitor Hadoop clusters and is a good alternative for those who prefer a graphical management tool.
We decided to try Ambari for ourselves using instructions found on the Apache website. Although Ambari is still in the incubating stage at Apache (we tested version 0.9), we found the documentation to be good, and after some minor configuration wrinkles we had the Ambari dashboard up and running in our browser. Although we had used Ubuntu as our go-to OS for this review, as it turned out we needed CentOS to run Ambari, as Ubuntu is currently not supported. The Ambari dashboard provides an at-a-glance overview of the Hadoop cluster status and metrics.
Values such as HDFS capacity, memory load, network and cluster load are displayed with the ability to drill down to view additional detail. Ambari has a cool heat map feature that allows you to view usage data such as “disk space used by host’ and “HDFS garbage collection time”. There is also a tab for services where you can monitor, start and stop specific services, and a host tab from which hosts can be added/removed as well as managed and monitored.
Compared to GlusterFS and Ceph, we found Hadoop a tad more cumbersome to install and configure. There are decent instructions on the Apache website along with several third-party sites that provide bits and pieces that are useful, but we would have liked to see a more comprehensive set of step-by-step instructions from the vendor.
Hadoop is widely used. Yahoo claims to be the largest Hadoop user and currently has more than 40,000 nodes in its cluster handling close to 400 petabytes (PB) of data. Facebook is reportedly storing hundreds of petabytes in its Hadoop cluster. Several big name vendors provide commercial implementations for Hadoop (EMC, Dell, Microsoft and Cloudera to name a few). Other third-party vendors such as Hortonworks, Nagios and Ganglia offer various levels of support and extensions for Hadoop.
Perschke is a web and database developer with 15+ years of industry experience. You can reach her at susan@arcseven.com.
