Emerging Linux filesystems – Part 2

As a comparison, I ran my test script against a set of standard filesystems in the Linux kernel that most users will be familiar with.

Watch a slideshow of the graphics with this article.

As well as being of general interest, it also gives a good range of values to use as a baseline when comparing the emerging filesystems.

ext2

The ext2 filesystem was introduced into the Linux kernel in January 1993 and was the principal filesystem until the introduction of ext3 in 2001. It is a fixed-block-size filesystem and has no journalling capabilities.

Timing Results

Bonnie Results

ext3

The third extended filesystem (ext3) was introduced into the mainline Linux kernel in 2001 to provide a filesystem that was backwards- and forwards-compatible with ext2 but which provided journalling of both metadata and (optionally) data.

In its default mode of data=ordered, it also provides a higher guarantee of filesystem consistency by ensuring that file data is flushed to disk before the corresponding metadata.

Timing Results

Bonnie Results

XFS

SGI’s XFS began life in the mid ’90s in Irix, their Unix variant, but in 1999 the company announced it was going to contribute it to Linux. It finally arrived in Linux in the 2.5.36 kernel on Sept. 17, 2002, and then in the 2.4.24 kernel on Feb. 5, 2004.

XFS is a 64-bit extents-based filesystem capable of scaling up to 9 exabytes, though on 32-bit Linux systems there are kernel constraints that limit it to 16TB for both filesystems and individual files.

Timing Results

Bonnie Results

JFS

JFS is a filesystem developed by IBM. It first appeared in the 2.5.6 development kernel on March 8, 2002, then backported to 2.4.20 which was released on Nov.28, 2002.

JFS is an extents-based filesystem and can extend to 4 petabytes with 4KB block sizes.

Timing Results

Bonnie Results

Reiserfs

Reiserfs (actually Reiserfs Version 3) was the first journalling filesystem to be included into the mainline Linux kernel, arriving in the 2.4.1 release on Jan. 29, 2001.

It uses a novel tree structure for files, as well as directories, and claims space-efficiency through its “tail-packing” of small files, though this feature can have performance impacts too, and can be disabled if necessary.

Timing Results

Bonnie Results

ChunkFS

*     Authors: Amit Gud, Val Henson, et. al

*     Website(s): http://linuxfs.pbwiki.com/chunkfs

Background

ChunkFS is based on ideas from Arjan van de Ven and Val Henson to counter the “fsck problem” caused by seek times not keeping up with disk sizes and bandwidth. It was discussed at the 2006 Linux Filesystems Workshopand again at the 2007 Workshop.

In their Usenix paper they describe the filesystem, saying:

“Our proposed solution, chunkfs, divides up the on-disk file system format into individually repairable chunks with strong fault isolation boundaries. Each chunk can be individually checked and repaired with only occasional, limited references to data outside of itself.”

There are two early implementations of ChunkFS at present; one is a straight kernel filesystem and the second is implemented as a user space filesystem using FUSE. Both use the ext2 filesystem code underneath the covers.

Installation

Both the FUSE and kernel versions, as well as their associated tool sets, are available via git from http://git.kernel.org/.

FUSE version

To retrieve the FUSE version of ChunkFS, I installed the cogito package which provides a higher-level interface to GIT and then cloned the repository thus:

# apt-get install cogito

# cg-clone git://git.kernel.org/pub/scm/linux/kernel/git/gud/chunkfs.git

Because it is a FUSE filesystem, it requires some dependencies to be pulled in, which are documented in the INSTALL files:

# apt-get install e2fslibs e2fslibs-dev fuse-utils libfuse-dev libfuse2

There is one undocumented dependency too:

# apt-get install pkg-config

To build it the usual routine is followed:

# ./configure –prefix=/usr/local/chunkfs-fuse-trunk

# make

# make install

It appears the FUSE version lags somewhat behind the kernel version, though the FUSE version is easier to debug because it is fairly easy to run it under GDB.

Kernel module

I had expected the straight kernel filesystem version of ChunkFS to be simply a kernel module, so on trying to clone it I was surprised to find out it was its own self-contained kernel tree! Thus began a rather painful experience.

I began by importing my existing kernel config file for the 2.6.22.1 build I already was using, and enabled it as a module. I also took notice of the text on the ChunkFS PBwiki page, which says:

Compile with CONFIG_CHUNKFS_FS set and CONFIG_BLK_DEV_LOOP to “y”. NOTE: No xattrs and xips yet, CONFIG_EXT2_FS_XATTR and CONFIG_EXT2_FS_XIP should be “no” for clean compile.

Unfortunately I was then greeted with this error during the build:

ERROR: “shrink_dcache_for_umount” [fs/chunkfs/chunkfs.ko] undefined!

ERROR: “super_blocks” [fs/chunkfs/chunkfs.ko] undefined!

ERROR: “sb_lock” [fs/chunkfs/chunkfs.ko] undefined!

Telling it to build directly into the kernel worked, however, so it appears that (for the moment) ChunkFS will not build as a module.

However, when trying to boot this kernel, it would panic on boot. I turned to Amit Gud, one of the developers, and he was able to supply me with an extremely cut-down kernel .config that he was using. I was able to take that and by carefully enabling only what was essential (PCI device support, SCSI support, Adaptec AACRAID driver, networking, and so forth), I was able to get the system to boot.

However, this certainly is not yet recommended for beginners!

Configuration

To be able to create a ChunkFS filesystem for either version of the filesystem, you need the ChunkFS version of mkfs which again is available through git.

# cg-clone git://git.kernel.org/pub/scm/linux/kernel/git/gud/chunkfs-tools.git

# cd chunkfs-tools

# ./configure –prefix=/usr/local/chunkfs-fuse-trunk

# make

# make install

As I had seven drives in the RAID-0 stripe, I decided to set that as the number of chunks to create within the ChunkFS filesystem, like this:

# /usr/local/chunkfs-fuse-trunk/sbin/mkfs -C 7 /dev/md0

However, it was rapidly apparent that this was incredibly slow and that it appeared to be rather a lot of I/O for each chunk. A quick investigation into the code showed that a missing break in the argument-checking code was causing the -C option to drop through into the -c option and setting the variable to enable bad block checking. Fortunately pretty easy to fix!

NOTE: This patch has been submitted upstream, but it has not been committed out to the kernel.org git repository, so you may want to check for yourself if you wish to experiment.

FUSE

Mounting the FUSE filesystem is the same as running any other user process. Here we pass through the -o option to specify the chunks we created previously with the custom mkfs command. You will notice that the same device is referenced seven times, once for each chunk specified with the -C option. Finally, we specify the mount point.

# /usr/local/chunkfs-fuse-trunk/sbin/chunkfs -o chunks=/dev/md0:/dev/md0:/dev/md0:/dev/md0:/dev/md0:/dev/md0:/dev/md0 /mnt

Kernel

The kernel filesystem is much simpler.  Even though we have seven chunks, we don’t need to tell it what we have created.

# mount -t chunkfs /dev/md0

Tests

Timing Results

Here we look at the FUSE and the kernel version separately.

FUSE

The FUSE variant seemed rather fragile. A reproducible crash before the results above was tracked down to a buffer overrun when the name of a file to be unlinked was being passed through. This code only appears in the FUSE variant, not the kernel version, because that doesn’t require this glue layer.

Bonnie Results

As can be seen above, the FUSE version of ChunkFS was not robust enough to survive the testing, so no Bonnie results were available.

Kernel

Sadly, the kernel version of ChunkFS locked the machine up hard while trying to extract the kernel source tree, and it was not possible to track down where this was happening in the time available.

NILFS

l     Authors: The NILFS Development Team, NTT Laboratories

l     Website(s):

Background

NILFS is a log based filesystem developed in Japan by NTT Laboratories and designed to provide continuous “checkpoints” (as well as on demand) that can be converted into snapshots (a persistent checkpoint) at a later date, before the checkpoint expires and is cleaned up by the garbage collector. These snapshots are separately mountable as read-only filesystems and can be converted back into checkpoints (for garbage collection) at a later date.

It has what appears to be a rather nicely thought out set of user commands to create (mkcp), list (lscp), change (chcp) and remove (rmcp) checkpoints.

Installation

Installation of NILFS is reasonably straightforward. I grabbed the 2.0.0-testing-3 versions of the nilfs kernel module and the nilfs-utils package and extracted them into their own directories.

The kernel module builds as an out-of-tree module so it just a matter of:

# cd nilfs-2.0.0-testing-3

# make

# make install

to get the necessary kernel module installed into /lib/modules/2.6.22.1 ext4/kernel/fs/nilfs2/.

The utilities package uses the standard autoconf tools, I built them with:

# cd nilfs-utils-2.0.0-testing-3

# ./configure –prefix=/usr/local/nilfs-utils-2.0.0-testing-3

# make -j4

# make install

Then I found out that it it didn’t completely honour the —prefix option I had passed through, as it did:

/usr/bin/install -c .libs/nilfs_cleanerd /sbin/nilfs_cleanerd

/usr/bin/install -c mkfs.nilfs2 /sbin/mkfs.nilfs2

/usr/bin/install -c mount.nilfs2 /sbin/mount.nilfs2

/usr/bin/install -c umount.nilfs2 /sbin/umount.nilfs2

This will be to allow for the way that mount, mkfs, etc., work when passed the -t option.

Configuration

Creating a NILFS filesystem is very easy:

# mkfs.nilfs /dev/md0

Mounting is again very simple:

# mount -t nilfs2 /dev/md0 /mnt

Tests

Timing Results

Bonnie Results

btrfs

l     Authors: Chris Mason, Oracle

l     Website(s): http://oss.oracle.com/projects/btrfs/

Background

On the 12th of June 2007, Chris Mason announced btrfs, a new filesystem that both checksums and does copy on write logging of all file data and metadata information. It also includes such features as space-efficient packing of small files, which should not be surprising, given Chris’s history with the reiserfs project.

Unlike most filesystems, the root of the btrfs filesystem is not for users, it is instead a place to keep various “subvolumes”. These subvolumes are described as named b-tree which can be be optionally allocated a fix number of blocks which it cannot exceed (a quota). These subvolumes can be snapshotted and either the original or the snapshot can be written too, with the filesystem using copy-on-write to constrain changes to that b-tree.

NOTE: As the announcement and Web site say, this filesystem is in very early alpha, the on-disk format isn’t fixed yet, and it doesn’t handle a filesystem running out of space gracefully (or at all, really).

Installation

To build btrfs I grabbed btrfs-0.5.tar.bz2 and btrfs-progs-0.5.tar.bz2 from the downloads page and extracted them. Btrfs builds as an out-of-tree kernel module, so all that was needed was:

# cd btrfs-0.5

# make

There is no make install option for this version of btrfs, so to load the kernel module it was necessary to first load the 32-bit CRC module and then the btrfs module, thus:

# modprobe crc32c

# insmod btrfs.ko

The btrfs-progs package required no configuration, just supplying a Makefile, which simply required an edit to change the prefix variable to /usr/local/btrfs-0.5 and then doing the usual:

# make

# make install

This time everything was installed directly under the prefix directory.

Configuration

Creating a btrfs filesystem follows the standard pattern of:

# mkfs.btrfs /dev/md0

Mounting that filesystem is again through the standard method of:

# mount -t btrfs /dev/md0 /mnt

Unlike the other filesystems, you will find that you cannot write into the top-level directory, but you will find a default directory already there. This is because btrfs reserves that top-level directory for subvolumes and snapshots.

You can use the btrfsctl -s command to create a new subvolume, or just work in the default subvolume, which is what I chose to do.

Tests

Timing Results

Bonnie Results

Checksum Errors

After the testing, I happened to do a dmesg and noticed the following checksum errors reported back, though they didn’t appear to affect any of the testing that was being done.

[208284.710000] btrfs: md0 checksum verify failed on 273600

[208422.630000] btrfs: md0 checksum verify failed on 295841

ext4

l     Authors: Too many to mention! Based upon ext3, ext2, ext..

l     Website(s): http://ext4.wiki.kernel.org/index.php/Main_Page

Background

The workhorse filesystem of the Linux kernel is ext3. It entered the mainline kernel in 2001 when 2.4.15 was released and can currently support filesystems as large as 8TB and files as large as 2TB (if your hardware supports 8KB pages, it can reach 32TB and 16TB, respectively). Those limits are now becoming a problem and patches to increase those limits have been circulating for a while, but have never been merged, precisely because ext3 is so important. In June 2006 Linus wrote on the Linux filesystem development list:

So near the end of June 2006 Ted T’so posted his “Proposal and plan for ext2/3 future development work” to the linux-kernel mailing list, giving his four-point plan for ext4 development, phase 1 (where we are at present) being:

The only difference with the current state of play is that the filesystem has ended up being ext4dev rather than ext3dev.

Installation

Because ext4 has arrived in the mainline kernel as an experimental filesystem, enabling it was just a matter of selecting “Ext4dev/ext4 extended fs support development” (CONFIG_EXT4DEV_FS) as a module under File systems in the kernel configuration, before building the kernel.

There are no ext4-specific packages required for ext4, development of the current e2fsprogs package to support ext4 is happening in git and you will only require these if you need to build an ext4 filesystem that requires 64-bit addressing.

NOTE: Be warned though, that ext4 support in e2fsck is fairly new and may not handle all the new features yet!

Configuration

Creating an ext4 filesystem is identical to creating an ext3 filesystem:

# mkfs.ext3 /dev/md0

It is only when you mount the filesystem that it becomes ext4:

# mount -t ext4dev -o extents /dev/md0 /mnt

NOTE: In early July a patch was submitted to make the extents option the default (which can be disabled with noextents) and is currently in the mainline GIT repository and the prepatches for 2.6.23.

WARNING: Once you have mounted an ext3 filesystem using ext4 with extents enabled (as it appears they will be by default in 2.6.23), ext3 will not be able to mount it again!

Tests

Timing Results

Bonnie Results

Reiser4

l     Authors: Namesys

l     Website(s): http://www.namesys.com/v4/v4.html

Background

On July 24, 2003 Hans Reiser of Namesys posted to linux-kernel, asking for people to look at his benchmarks of the new Reiser4 filesystem compared to Reiserfs and ext3.  He asked for inclusion into the 2.5 development kernel ready for the release of 2.6.

It is now more than four years later, and Reiser4 still is not included in the mainline kernel, though it has at least made it into Andrew Morton’s experimental “-mm” series in August 2004 when 2.6.8.1-mm2 was released. The reasons are many, and rather than go into them here, I would direct interested people to this 2005 LWN article on the matter.

Installation

Because Reiser4 isn’t in the mainline kernels yet, I picked what was at the time the latest in Andrew Morton’s “mm” series, 2.6.22-rc6-mm1. All that was necessary was to import the .config file I was already using and tell it to include Reiser4 support when prompted in the filesystems section, as below:

Reiser4 (EXPERIMENTAL) (REISER4_FS) [N/m/y/?] (NEW) m

The kernel built with no issues.

Configuration

Ubuntu happens to package Version 1.0.5 of reiser4progs in its main repository, even though their packaged kernels do not support it yet. Installing them is as easy as

# apt-get install reiser4progs

You can then create a reiser4 filesystem simply by doing:

# mkfs.reiser4 /dev/md0

Mounting it is also trivial, again we just need to specify the filesystem name.

# mount -t reiser4 /dev/md0 /mnt

Tests

The first attempt with reiser4 ended rather unhappily with a crash part way through the testing that killed the filesystem.

Example 1. Reiser4 crash

[   81.600000] Loading Reiser4. See www.namesys.com for a description of Reiser4.

[   81.620000] reiser4: md0: found disk format 4.0.0.

[  162.730000] reiser4[pdflush(265)]: disable_write_barrier (fs/reiser4/wander.c:234)[zam-1055]:

[  162.730000] NOTICE: md0 does not support write barriers, using synchronous write instead.

[  862.270000] reiser4[fixdep(4426)]: parse_node40 (fs/reiser4/plugin/node/node40.c:672)[nikita-494]:

[  862.270000] WARNING: Wrong level found in node: 1 != 113

[  862.270000] reiser4[fixdep(4426)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:662)[]:

[  862.270000] WARNING: reiser4_write_tail failed

[  862.270000] reiser4[fixdep(4426)]: release_unix_file (fs/reiser4/plugin/file/file.c:2378)[nikita-3233]:

[  862.270000] WARNING: Failed (-5) to convert in release_unix_file (82406)

[  862.270000] reiser4[fixdep(5010)]: parse_node40 (fs/reiser4/plugin/node/node40.c:672)[nikita-494]:

[  862.270000] WARNING: Wrong level found in node: 1 != 113

[  862.270000] reiser4[fixdep(5010)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:662)[]:

[  862.270000] WARNING: reiser4_write_tail failed

[  862.270000] reiser4[fixdep(5010)]: release_unix_file (fs/reiser4/plugin/file/file.c:2378)[nikita-3233]:

[  862.270000] WARNING: Failed (-5) to convert in release_unix_file (82407)

[  862.270000] reiser4[fixdep(4292)]: parse_node40 (fs/reiser4/plugin/node/node40.c:672)[nikita-494]:

[  862.270000] WARNING: Wrong level found in node: 1 != 113

And so on. Not a pretty sight.

The second attempt a few days later, after remaking the filesystem, was more successful.

Bonnie Results

ZFS/FUSE

l     Authors: Ricardo Correia (FUSE port), Sun Microsystems (original ZFS)

l     Website(s): http://www.wizy.org/wiki/ZFS_on_FUSE

Background

ZFS, originally the Zettabyte File System, now just the initials, was created by Sun Microsystems and is a log-based, copy-on-write filesystem with reference counting, snapshots and checksums of data on disk.

The basic concept of using ZFS is to put physical disks or arrays into a storage pool using some RAID-like strategy (striping, mirroring, RAIDZ, or some combination of these) and then carve off filesystems from that pool for use.

The filesystems carved off can have different attributes set on them, such as compression, storing multiple copies of data for resilience, and so forth, and can inherit these attributes from their parents.

There has been much interest from outside the Solaris community about ZFS, and there are already working ports in the current development versions of Apple’s MacOSX and FreeBSD. A direct Linux port was made much harder by the decision of Sun to create a new license, the CDDL, for their open-source version of Solaris called OpenSolaris, and the CDDL is widely believed to be incompatible with the GPL used in the Linux kernel.

In 2006 Google’s Summer of Code program included an application to provide the ZFS filesystem for FUSE/Linux, taking advantage of the fact that by running the filesystem in user space with FUSE, the licensing problems would not be an issue.

Although the Summer of Code 2006 has finished, Ricardo has carried on working on the port, and it has now reached beta stage.

WARNING: On July 25, 2007 Riccardo posted to the ZFS/FUSE mailing list saying:

“Raid-Z is supported, but currently there is a chance that the pool might completely corrupt, so don’t use it on important data. This issue is being investigated.”  Unfortunately he’s not yet responded to a query about whether this is limited to ZFS/FUSE (probable) or ZFS in general (which should have been picked up by now, you would have thought).

Installation

ZFS/FUSE uses Mercurial for its source code control, and builds using scons rather than the usual make program. To install both those, it was simply a matter of doing:

# apt-get install mercurial scons

Grabbing the trunk of ZFS/FUSE and building and install ZFS/FUSE is then done with:

# hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk

# cd trunk

# cd src

# scons -j 4

# scons install install_dir=/usr/local/zfs-trunk

The ZFS commands now live directly under /usr/local/zfs-trunk.

Configuration

I first created a new “pool” of storage (called zfs, but it could have been called anything) from which to later create ZFS filesystems and checked its status. The pool name is also used as the top level mountpoint (which will be /zfs in this case), but I could have changed that by using the -m option.

# zpool create zfs /dev/md0

# zpool status

  pool: zfs

 state: ONLINE

 scrub: none requested

config:

        NAME        STATE     READ WRITE CKSUM

        zfs         ONLINE       0     0     0

          md0       ONLINE       0     0     0

errors: No known data errors

The “scrub” mentioned above is a periodic check of the checksums of all the data that can be requested to try to spot problems with data that hasn’t been accessed for a while.

Now I can create a ZFS filesystem called testing from the zfs pool to use for testing and then look at what is available, thus:

# zfs create zfs/testing

# zfs list

NAME          USED  AVAIL  REFER  MOUNTPOINT

zfs           130K   469G    19K  /zfs

zfs/testing    18K   469G    18K  /zfs/testing

By default, each zfs filesystem in that pool shares the same free space, though you can constrain that through quotas should you wish.

Quotas are just one of a myriad of options that can be set on a ZFS filesytem. Others that may be of interest are compression (to enable or disable compression on a filesystem) and copies (to store multiple copies of data for extra redundancy). Here is the full list as returned by the zfs command. Note that several of them are, as of the moment, unimplemented, including the snapdirvisible option, which isn’t noted as such below.

The following properties are supported:

        PROPERTY       EDIT  INHERIT   VALUES

        type             NO       NO   filesystem | volume | snapshot

        creation         NO       NO  

        used             NO       NO  

        available        NO       NO  

        referenced       NO       NO   

        compressratio    NO       NO  

        mounted          NO       NO   yes | no | –

        origin           NO       NO  

        quota           YES       NO   | none

        reservation     YES       NO   | none

        volsize         YES       NO  

        volblocksize     NO       NO   512 to 128k, power of 2

        recordsize      YES      YES   512 to 128k, power of 2

        mountpoint      YES      YES   | legacy | none

        sharenfs        YES      YES   on | off | share(1M) options  # not yet implemented

        checksum        YES      YES   on | off | fletcher2 | fletcher4 | sha256

        compression     YES      YES   on | off | lzjb | gzip | gzip-[1-9]

        atime           YES      YES   on | off

        devices         YES      YES   on | off

        exec            YES      YES   on | off

        setuid          YES      YES   on | off

        readonly        YES      YES   on | off

        zoned           YES      YES   on | off

        snapdir         YES      YES   hidden | visible

        aclmode         YES      YES   discard | groupmask | passthrough

        aclinherit      YES      YES   discard | noallow | secure | passthrough

        canmount        YES       NO   on | off

        shareiscsi      YES      YES   on | off | type=  # not yet implemented

        xattr           YES      YES   on | off

        copies          YES      YES   1 | 2 | 3

Sizes are specified in bytes with standard units such as K, M, G, etc.

User-defined properties can be specified by using a name containing a colon (:).

Tests – pre O_DIRECT support

At the time of testing ZFS/FUSE was going through work to convert it to using O_DIRECT for file I/O to prevent the kernel duplicating ZFS’s own caching. This gave me the opportunity to compare ZFS/FUSE both before and after.

Timing Results

Yes, the tests really did run for almost 6 hours. Ricardo is very clear that he has not yet started on performance optimizations in ZFS/FUSE; he is currently concentrating purely on getting it functional rather than fast.

Bonnie Results

Tests – post O_DIRECT support

Timing Results

Bonnie Results

Comparing before and after, we can see that the implementation of O_DIRECT has negatively impacted performance.  When I asked Ricardo about this on the ZFS/FUSE mailing list, he responded, “The reason bonnie is slower is that I/O is now being done synchronously, but I am planning to fix that shortly.”

Unfortunately those fixes haven’t yet appeared in the Mercurial repository. At the time of writing, the last commit is changeset 244, the final part of the O_DIRECT work, which is more than four weeks old.

ZFS on OpenSolaris

l     Authors: Sun Microsystems

l     Website(s): http://www.opensolaris.org/os/community/zfs/

Background

Given that the ZFS/FUSE filesystem is so new to Linux, it would be very interesting to see how alternative implementations under OpenSolaris and FreeBSD would fare on the same hardware.

Installation

Unfortunately, this turned out to be much harder to try than was initially expected, because Linux’s excellent hardware support had raised expectations for these other operating systems. Early hopes for painless installs were dashed. The latest snapshot of FreeBSD 7 available at the time of testing, 200706, could not recognize the RAID card. Neither could the latest releases of the OpenSolaris derived distributions Nexenta or Belenix.

In the end, success was only possible with the latest snapshot of Nexenta available, ncp_beta1-test3-b68, which had an AAC driver new enough to recognize the Adaptec 31605. It was installed using ZFS as its root on a single drive.

Nexenta is sometimes called “GnuSolaris” because it is loosely based on the Ubuntu userland, providing a mixed OpenSolaris and GNU environment with an OpenSolaris kernel. The usual apt-get commands are present, but the snapshot release had some problems doing an upgrade.  However, these issues did not prevent testing using the standard install.

Configuration

ZFS is a standard part of OpenSolaris, and Nexenta was able to install into a ZFS root partition using a pool called syspool containing just a single drive.

The seven spare SAS drives were then added in to make another pool called, as before, zfs.

You can also see that between the first and second zpool status commands, I had done a scrub of the system pool by issuing the following command to see if it could detect any issues.

Tests

Timing Results

It was not surprising that kernbench failed, because it parses /proc/meminfo and /proc/cpuinfo under Linux to learn how much RAM and how many CPUs are available. The GCC error appears to be an issue with the build process; it appears to decide it needs to rebuild java/parse-scan.c from the original yacc file, but bison is not yet packaged for Nexenta so the builds fails at that point. Of course, with those failures the overall run time cannot be compared to other tests.

Bonnie Results

Filesystem scaling

Testing ZFS under OpenSolaris allowed testing of Bonnie with various numbers of disks being striped over in the pool to see how well it scales.

FIGURE 2: ZFS performance scales as the array grows from one to seven drives under OpenSolaris.

Comparisons

Standard filesystems

Timing Results

Bonnie Results

FIGURE 3: File operations per second for the standard Linux filesystems

FIGURE 4: Block operations (KB/s) for the standard Linux filesystems

Emerging filesystems

Timing Results

Bonnie Results

FIGURE 5: File operations per second for the emerging Linux filesystems

FIGURE 6: Block operations (KB/s) for the emerging Linux filesystems

All together now!

Timing Results

Comparison Graphs of Individual Tests

The scale on the lefthand side is in seconds, where less is better (faster). Filesystems with no bar shown failed that test.

FIGURE 7: Extract kernel sources  (in seconds, lower is better)

FIGURE 8: Extract GCC sources  (in seconds, lower is better)

FIGURE 9: Recursive random file test (in seconds, lower is better)

FIGURE 10: Configure GCC (in seconds, lower is better)

FIGURE 11: kernbench test (in seconds, lower is better)

FIGURE 12: Build GCC with “-j16 bootstrap” (in seconds, lower is better)

FIGURE 13: Remove kernel sources  (in seconds, lower is better)

FIGURE 14: Bonnie benchmark, file operations (in seconds, lower is better)

FIGURE 15: Remove GCC sources  (in seconds, lower is better)

FIGURE 16: tiobench threaded I/O test (in seconds, lower is better)

FIGURE 17: Bonnie intelligent I/O test (in seconds, lower is better)

Bonnie Results

Graph of file operations per second for all filesystems

(FIGURE 18)

Graph of bandwidth measured for Block I/O for all filesystems

(FIGURE 19)

Conclusions

Innovation

There is a great range of innovation in the various emerging filesystems tested here. It ranges from the rather conservative ext4 model of extending an existing filesystem with some new features, all the way through porting an existing stable filesystem to a new platform (ZFS/FUSE) to completely new code bases that integrate some of the current hot concepts in filesystem design (btrfs & nilfs).

The filesystems that have impressed me most are ZFS/FUSE for its comprehensive feature set, btrfs for creating a kernel level filesystem with checksum support, and nilfs for its creative continuous checkpointing system and the ability to convert a checkpoint to and from a persistent snapshot at will.

Stability

Let’s face it, emerging filesystems are probably not what you’d really be wanting to use for your business critical data, or even your working home directory, unless you’re very brave and have good backups. Consider this article a look into the future.

ChunkFS has a long way to go, from what I have seen of it. The ideas behind it are nice, but it’s far too much on the painful side of bleeding edge at the moment.

Whilst btrfs didn’t crash during testing, you do need to keep in mind that the on-disk format isn’t fixed yet, so you’re highly likely to find that later versions won’t read current filesystems The fact that it (apparently) blows up spectacularly if it runs out of disk space, should keep you on your toes.

Reiser4 has been around since 2003, so I was rather surprised to have it crash messily on me the first time around. While it is possible that it could have been related to some transient hardware issue, it does seem unlikely.

ZFS/FUSE currently has a shadow hanging over it regarding potential corruption of RAIDZ arrays, though there are not enough details about the problem to know quite what is going on here. The mirroring option, however, is apparently not affected by this.

The final two filesystems, nilfs and ext4, are also still under active development, and so you should expect all sorts of interesting things to possibly happen to your data!

Performance

OK, so we all know that filesystems benchmarks are really good at showing which filesystems excel at filesystem benchmarks, but not necessarily which are good for real-world use. My hope is that somewhere in the range of benchmarks chosen here are one or two metrics that may be of help to you when looking at which filesystems to experiment with.

My biggest surprise here was that two existing stable filesystems, ext2 and XFS, so roundly trounced all the other filesystems on block writes — even Reiser4, for which Namesys has made such great claims.

ZFS/FUSE performs badly all round, which is not surprising, given the fact that no performance tuning has yet happened in the FUSE port. ChunkFS also doesn’t do too well in the tests it does complete in.

The only other standout was that both ext2 and nilfs are exceptionally good at removing large numbers of files sequentially.

The Future

My gut feeling is that btrfs is going to get quite a bit of attention. The fact that it checksums the data on the platter will give people a nice sense of security, while the ability to do snapshots is valuable. With the current version, a user can’t write the top-level directory of a btrfs filesystem, and author Chris Mason is still planning a new version of the “mount” utility to allow users to mount btrfs subvolumes or snapshot volumes as normal filesystems. For test use, if you mount your btrfs filesystem as /home and just create subvolumes in there for users, it may not be too bad.

I also think that nilfs is one to watch, because it has a really well-thought-out checkpointing/snapshot system and user interface. Ext4 will probably succeed ext3 in the kernel, though the fact that once you’ve mounted it as ext4 as written a file, you can no longer mount it as ext3 will cause some pain.

Whatever happens, we are living in interesting times in filesystem development.  The next few years are going to be really fun!