Linux developers are pointing out three major technical holes that will take serious development effort to close.
As the Linux Foundation plans its first Collaboration Summit for June 13 through 15 at the Google campus, Linux contributors are speaking out about kernel gaps that have no solution readily in sight.
Torvalds adds what he calls the “development flow” to the list of concerns. “We’ve always had issues with how certain subsystems end up having development problems due to some infrastructure issue, or just personality clashes, or some methodology being broken. Sometimes it’s not ‘subsystems,’ but hits at an even higher level, and we end up having to change how we do things in general,” he says.
Linux has already been through major shifts in development process. With the release of the 2.6.0 kernel in December 2003, Torvalds and other developers stopped maintaining separate stable and research development trees for the kernel, although a new line of stable kernel releases began again with the 2.6.11.1 kernel in 2005.
File systems
“We have a lot of good file systems, and I think most people are happy with them. But I think we could do better,” Torvalds says. Morton was blunter in his talk, saying, “Basically, I think we need a new file system.”
File-system developer Val Henson points out that disk capacities are likely to grow by a factor of 16 by 2013, but that bandwidth will only grow by a factor of 5, and seek time by a factor of 1.2. That means that the file-system-checking utility, fsck, will take longer and longer to run. “Fsck on multiterabyte file systems today can easily take 2 days, and in the future it will take even longer! Second, the increasing number of I/O errors means that fsck is going to happen a lot more often — and journaling won’t help,” Henson wrote.
Torvalds points out that the standard Linux file system, ext3, does some wasted work on ordinary-sized disks, too. “Ext3 is ubiquitous, but actually doesn’t do very well on some ‘simple’ cases like fsync, where it ends up flushing basically the whole journal, even if we just want to sync a single file,” Torvalds says.
The “fsync” system call is required only to write to disk the data associated with a single file. There is a separate “sync” system call to flush all buffered data to disk. On a busy server, flushing extra journal data would slow down an application that was trying to “fsync” just one file.
Sun’s ZFS is the hot new file system and blends what on Linux are separate RAID, logical volume management and file-system layers into a single subsystem. ZFS, however, is under the open source, but GPL-incompatible, CDDL license and so isn’t available to become part of Linux directly.
Implementing a ZFS-compatible file system, even with access to the source code, may not be an option because of Sun’s patents. Torvalds adds that Network Appliance has patents on some other useful file-system techniques.
Developer Ricardo Correia has devised a way around the licensing beef, with a system for running Sun’s own ZFS code in user space, employing the FUSE technology. Although the project is much slower than a conventional in-kernel file system, Correia claims that “FUSE-based file systems can have comparable performance to kernel file systems, as the bottleneck is usually the disk(s), not the CPU.” There is precedent for handing off a key system role to a specially privileged user-space program — the X Window System used on Linux has always run as a user-space application. One developer has even built an experimental Linux distribution that integrates ZFS on FUSE into the installer, but the approach doesn’t appear to have seen a severe test yet.
In his own “kernel report” talk at the Embedded Linux Symposium, kernel developer and author Jonathan Corbet points out two other file-system research efforts, neither of which is likely to make the mainstream soon. Ext4 is a research extension of ext3, and the process of adding Reiser4 to the kernel was stalled even before inventor Hans Reiser’s arrest in California last October. (Kernel updates syndicated from Corbet’s Web site, LWN, appear on LinuxWorld.com as the weekly Kernel Space feature.)
While current file systems are designed around the disks, Torvalds suggests that in the future the assumptions may need to change. “Another interesting issue is that I don’t think that the day when we’ll see more solid-state storage is a pipe dream any more,” he says. “Sure, some people want terabyte hard disks, but for others, the multihundred GB disks just lay mostly unused. And flash storage has a totally different performance profile, which means that things that we do for traditional disk reasons, may not actually make as much sense if you actually aimed for a flash disk in the first place.”
Power management
“We’re still pretty stupid about power management,” Morton says. Devices are able to support low-power states. For example, a network card that’s not receiving incoming traffic can go to a lower-power state. But Linux supports only on and off, and Morton says, “we’re having trouble getting off and on working.”
Torvalds says that there’s a difference between a driver that works — Linux ships with more than any other OS — and a driver that implements advanced features of the hardware, such as power-saving modes. Many Linux drivers still lack the latter. “We have a ton of drivers, but 99% of all people worry a lot more about basic functionality. Many people would like for the driver to also use power efficiently, and handle suspend/resume well, but it’s definitely a secondary concern to ‘working’,” he says.
Imad Sousou, director of Intel’s Open Source Technology Center, says several Intel developers are working on the power problem.
Among the projects is power-saving functionality for multicore processors. On one dual-core processor, putting one core into a low-power “nap” can reduce the processor’s power use from 34 to 12 watts, Sousou says. But while power saving for the processor works well in existing Linux installs, much work remains to be done on drivers for PCI devices, he says.
Where Intel controls the hardware and driver, power is a priority, Sousou says. But all the drivers for PCI devices, from a variety of hardware makers and driver authors, need attention in a power-saving project. “You need to do it on a wide spectrum of devices to be effective. We’re doing our part,” he says.
Unlike big splashy new kernel features, such as new virtualization technologies, the power problem will be a series of little jobs. “I don’t think there’s anything big that’s going to make things better — it’s going to be one thing at a time,” Sousou says.
One recent step forward in kernel-side power management is still waiting for other programs to catch up. Development work from Ingo Molnar of Red Hat and Thomas Gleixner, founder of embedded Linux development firm Linutronix, gave Linux a power-conservation feature called “tickless” that shuts down Linux’s regular timer interrupt, which typically goes off from 100 to 1,000 times a second. In place of the periodic timer, a tickless kernel wakes up only when needed.
Although tickless holds the potential for putting millions of Linux boxes into cool, power-sipping sleep instead of heat-spewing wakefulness, two limitations hold it back. First, it works only for 32-bit kernels, not the 64-bit ones commonly used on new servers; 64-bit support is said to be coming soon. Second, many user programs still wake up the kernel when they don’t have to — for example, to check for the existence of new files when they could be using functionality called “inotify” to let the kernel tell them when new files appear.
Fedora kernel developer Dave Jones became the spokesman for the trend of power troubleshooting in user space with a talk last year, “Why user space sucks.”
This May, the hunt for power-sucking applications got a boost when Intel developer Arjan van de Ven announced the powertop utility. Powertop pinpoints any running programs that, as the project web site puts it, “ruin the party” by waking up the processor.
The One Laptop Per Child project asked for tickless and is in a position to benefit from it quickly, because it’s working with a finely tuned set of software. But with powertop available, the information is out there to put the pressure on more of the diverse user applications found on conventional desktop and server boxes to clean up power-hoggish coding.
Instrumentation
“I don’t think we expose enough stuff to sophisticated programmers to tell them what’s going on in the kernel,” Morton says. But of the three big holes, this is probably the one with the most work going on. Morton listed quite a few high spots in instrumentation, including per-task I/O accounting and per-process memory footprint monitoring.
He also mentioned Matt Mackall’s “PSS” and “USS” metrics for memory used by a process as a good step forward. Current memory-use stats don’t break out how many processes share a page of memory, so a user can’t get an accurate measurement to answer a simple question, such as “How much memory will I free up if I kill this process?” Mackall’s work makes it easier to assign memory-hogging blame to programs.
Currently, IA-64 Linux has access to that platform’s hardware-performance counters via perfmon, and Morton says, “We’ll get there eventually” for other platforms.
Torvalds doesn’t see any hurry.
The problem is that, yes, we could get better data, but quite often, the gap between “data” and “information” then ends up being so big that it’s not necessarily always even worth getting the data — because turning it into information won’t happen.
For example, we can get a _lot_ of profiling events on pretty much all modern CPUs, and you can do things like cache miss profiling, etc. In the end, most of the real performance problems end up being at a higher level, and in fact, simple cycle profiles tend to be quite sufficient to get enough low-level performance to know what the low-level problem is: and getting to the *high-level* problem is usually not about getting more low-level data.
So as far as I’m concerned, this is more of a “we could do better in gathering some data,” but at the same time, I personally suspect that people already aren’t even taking full advantage of the data that we *do* offer, and that sometimes the “I don’t have information about detail XYZ” is used as an excuse for not doing the things you can already do.
Perhaps what Linux needs at this point is some good old-fashioned IT industry Fear, Uncertainty and Doubt. When Linux on Apache lost out to a Microsoft OS and Web server in tests run by Mindcraft in 1999, the competition made a PR splash of it.
It was a short-lived one. “Two months after the benchmark, the TCP stack was fully SMP threaded; five months after that the entire networking subsystem was fully parallelized. The result: last month’s Specweb results, which blow away everything else,” Corbet wrote a year later.
