Connecting new hardware to the Linux market: Ed Cashin

Don Marti, LinuxWorld: So tell me a little bit about Coraid and the hardware that the company makes.

Ed Cashin, Coraid: We are a small but growing company. And we’re located on both coasts of the United States.And we have been around for a few years now, doing scalable network storage that we make available to folks, usually Linux folks, but also other people, by way of open source software.

LinuxWorld: The first I heard about Coraid was when I saw your name and e-mail address go by in a kernel changelog. Can you tell me a little bit about the process of getting the open source driver for Coraid’s ATA over Ethernet into the kernel?

Cashin: Well sure, it really happened before I was working at Coraid. I got a chance to follow the linux-kernel mailing list and noticed who was who and how the patches usually flowed through the mailing list and through the different kernel trees. And then once I started working at Coraid, it became apparent what needed to be done.

A lot of that was the fact that the kernel folks were really trying to articulate what needed to be done. And part of that was the work by Greg Kroah-Hartman, who was trying to codify a lot of the style and principles of design that go into drivers. And he was trying to make it easier for driver developers to get into the kernel. And so, one of the ways he did that was to create some new documentation and update some old documentation. So Greg was asking for people to do exactly what we wanted to do.

I went to the Ottawa Linux Symposium and finally tracked down Greg on the last night. And met him and he was very friendly and helpful. And basically, it just consisted of getting a loop started where we got our kernel driver ready and tried to make it fit the design principles of the Linux kernel and be acceptable to the kernel community. And we submitted it.

They had a lot of feedback. Some of it, they wanted to know, "Why do you do this? Why do you do that?" Some of it was suggestions on things that could be made better or things that could be improved so that it would fit better with the rest of the kernel. We took those suggestions, implemented them in our driver, resubmitted and after a couple of iterations of that loop, we saw folks start to push our patches up towards Linus Torvalds. And from then on, they started coming back out to the distributions: Red Hat, SUSE and everyone.

LinuxWorld: How long did it take between the first time you showed driver code to someone outside the company and the time that your driver made it into the mainstream kernel?

Cashin: Well, I’m talking about my experience. They actually did submit an earlier version of the driver before I got to Coraid. And so I’m not taking that into account, because that was the 2.4 kernel. And so if you just look at it in terms of our 2.6 kernel driver, probably it took about four to six months.

LinuxWorld: When you maintain new updates to the driver, do you maintain that in a git tree or do you send patches to other kernel developers?

Cashin: Well, I happened to be using git personally. But I always interact with the Linux kernel community through patches and e-mails. And it just works for me. The changes aren’t like a torrent. They’re more like small bursts. And so using just e-mails with regular patches in it facilitates discussion online. And that’s very helpful to us.

LinuxWorld: Do you get a sense of how many Coraid customers find out about the company and the existence of the hardware through just happening to see that driver code and for that configuration option in the kernel?

Cashin: Well, it’s hard to estimate a percentage of customers. But I know it’s significant. If I go to a conference, I’ll often hear people say, "I had no idea what ATA over Ethernet was until I saw you in the linux-kernel mailing list," or something like that. So I know that the fact that we’re developing in public is actually making people notice this more. And it’s a good thing because our customers tend to be the type of people who are aware of what’s going in Linux kernel development. And so we're branching out into different types of customers. But most of our customers right now are a little geeky, a little more technical.

LinuxWorld: Early adopters?

Cashin: They’re early adopters, yes, or at least, interested experts.

LinuxWorld: You maintain a version of the driver that people can download directly from the company, as well as the version that’s in the kernel. How does that work, syncing up your version with the version that goes into the kernel.org kernel?

Cashin: Well, we were actually a little concerned about that when we started getting going. But it hasn’t turned out to be nearly as big of a deal as you might think. It’s actually been very convenient for us. And basically, it comes down to a situation where we have a driver on our Web site. We know that not all of our customers are running the latest, greatest kernel from kernel.org. In fact, most of them aren't. So the driver at our Web site has a series of configuration patches that is applied selectively based on the kernel that the customer is using. So the main file can select the appropriate patches and configure the driver to fit their kernel. And we can do that because we participate in the development community. And we’re made aware of changes to the Linux kernel that affect our driver.

So, as long as we’re keeping track of the changes, we can create a configuration step if necessary. And our customers are able to run the latest driver from our Web site on any Linux kernel. And usually we develop for a while, and when we add a feature or something that goes into our Web site driver. And after it settles down and we have a coherent collection of changes to our driver, that’s when we push them up to the linux-kernel mailing list. And they get a peer review. And they get incorporated upstream. And they start making their way down into the distributions again. So usually we release a bunch of related changes in the form of a few well-tested patches.