Should Linux include a virus scanning layer? Kernel developers debate the best way to protect virus-vulnerable OSs from malware stored on a Linux server.
The TALPA malware scanning API was covered in LWN in December, 2007. Several months later, TALPA is back – in the form of a patch set posted by a Red Hat employee. The resulting discussion has certainly not been what the TALPA developers would have hoped for; it is, instead, a good example of how a potentially useful idea can be set back by poor execution and presentation to the kernel community.
The idea behind TALPA is simple: various companies in the virus-scanning business would like a hook into the kernel which allows them to check for malware and prevent its spread. So the patch adds a hook into the VFS code which intercepts every file open operation. A series of filters can be attached to this intercept, with the most important one being a mechanism which makes the file being opened available to a user-space process as a read-only file descriptor. That process can scan the file and tell the kernel whether the open operation should be allowed to proceed or not. In this way, the scanning process can prevent any sort of access to files which are deemed to contain bits with evil intentions.
There are a few other details, of course. A caching mechanism prevents rescanning of unchanged files, increasing performance considerably. There is also a hook on close() calls which can trigger the rescanning of a file. Processes can exempt themselves from scanning if it might get in their way; scanning can also be turned off for specific files, such as those used for relational database storage. But the patch set is relatively small, as it really does not have that much to do.
This capability could well prove to be useful. Even if one is not concerned about malware infections on Linux systems, a lot of files destined for more vulnerable platforms can pass through Linux servers. There is also the potential for the detection of attempted exploits of the Linux host. Normally, in the Linux world, the way we respond to knowledge of a specific vulnerability is to patch the problem rather than scan for exploits, but there may be systems which cannot be restarted on short notice, and which could benefit from an updated scanning database while running code with known vulnerabilities. Also, as Alan Cox pointed out, this feature could be useful for entirely different objectives, such as efficient indexing of files as they change.
What might be best of all, though, is that this hook could replace a number of rather less pleasant things being done by anti-malware vendors now. Some of these products use binary-only modules, plant hooks into the system call table, and generally behave in unwelcome ways. Moving all of that to a user-space process behind a well-defined API could be beneficial for everybody involved.
The patches have gotten a generally hostile reception on the kernel mailing lists, though. Some developers are uninspired about the ultimate objective:
So you are going to try to force us to take something into the Linux kernel due to the security inadequacies of a totally different operating system? You might want to rethink that argument.
That’s an objection which can be worked around; the kernel developers do not normally want to determine which applications will or will not be supported by the system as a whole.
Another objection, though, might be harder: this hook is said not to be the best solution to the problem. Instead of putting a hook deep within the VFS layer, the anti-malware people could simply hook into the C library (perhaps with LD_PRELOAD), put the malware scanning directly into the processes (mail clients or web servers, say) which are passing files through the system, or embed the scanning into a stackable filesystem implemented with FUSE (or a similar mechanism). That has led to counterarguments that scanning implemented in this manner could be evaded by a hostile application – by performing system calls directly, for example, instead of going through the C library. Certain kinds of attacks, it is said, could get around a purely user-space solution.
That argument, however, highlights the real problem with this posting. The patch includes a set of 13 “requirements,” including intercepting file opens, caching results, exempting processes, and so on. But none of these requirements describe the problem which is really being solved. In particular, as noted by Al Viro and others, there is no description of the threat which this patch is intended to mitigate:
Various people had been asking for _years_ to define what the hell are you trying to prevent. Not only there’d been no coherent answer (and no, this list of requirements is _not_ that – it’s “what kind of hooks do we want”), you guys seem to be unable to decide whether you expect the malware in question to be passive or to be actively evading detection with infected processes running on the host that does scanning.
If the scanning host could be infected, then a scanning mechanism which could be circumvented by a rogue program is indeed a problem. But that is a very different threat than simply trying to prevent evil attachments from creating mayhem on Windows boxes; it does not appear to be a threat which these patches are trying to address.
The lack of a clearly described problem has caused the discussion of these patches to go around in circles; it is not possible to evaluate (1) whether the goals of these patches are worth supporting, or (2) whether the patches can actually be successful in achieving those goals. The code, in other words, cannot be reviewed. Until the TALPA developers can clarify that situation, their work will look like an example of “shoot first, then aim.” That kind of code tends not to make it into the mainline, even if it could be useful in the end.
