While reverse engineering malware may not be something you've ever even considered, after a three day class, I can say that, yes, it's difficult, but also that it's well within the grasp of ordinary mortals (at least those ordinary mortals who can wrap their brains around assembly language and system calls) -- especially if they take the right class, get the right tools, and practice.
What is it?
Reverse engineering malware is the process of taking a captured executable (a stand-alone executable or a library file, such as a DLL) and doing the computer equivalent of an MRI. You should always take some care in where and how you do this kind of work. Analyzing malware in a "safe" environment -- such as a virtual system on a machine that isn't connected to your network -- would be the option of choice. At least this would allow you to snapshot your OS and revert to it whenever you inadvertently kick the malware under your microscope into action. Unix admins who have some experience using the strings command to identify strings defined within binaries or hex dumps provided by commands such as od will have some rudimentary knowledge of the contents of executables, but there is lot more that you can see if you examine the malware file using a debugger (a tool that allows you to step through a program, watching for the effects that each step makes) or a disassembler (a tool that turns the machine language of your malware into assembly language). With a disassembler, you would perform what is called "static analysis". With some experience, you will come to see how the malware you are analyzing was intended to work. Maybe it inserts itself into another process, maybe it creates a file, maybe it changes user settings ... With a debugger, you can break and change the code "on the fly". You can exploit test conditions that may have been designed to make the malware appear benign under certain conditions. You can jump over code that isn't worth your time to analyze. This process is often referred to as "dynamic analysis".
The tools
Some debuggers and disassemblers are free while others cost a bundle, but if you are going to be doing serious work with malware analysis, you should keep in mind that any tool that helps you get answers in hours instead of days should be considered a good investment (do the math!), especially if you will be using it often. Some of the tools that you are likely to learn about if you take a class or do some online research are IDA Pro, Immunity Debugger and Olly Debugger. The IDA tools are available on Mac OS X as well as Linux. In the AoE class that I took, I had opportunities to work with all of these tools and much appreciated that I had someone walking around the classroom willing to answer all kinds of questions. While these tools all provide options that will help you delve into your malware, I don't think I would have wanted to learn any of them without some start-up help to guide me on how to make the best use of them.
The class
The class that I took was offered by the Art of Exploitation (http://www.artofexploitation.com/academics_malware_rev_eng.aspx) and was both eye-opening and chock full of tips designed to help the new malware analyst past the stumbling blocks and into profitable analysis fairly quickly. Having a chance to become familiar with some of the better tools and getting tips on how to make the best use of them will help you make good decisions when it comes time to acquire your own. We spent time ...
- looking for system calls that clue you into what the malware is doing
- watching out for techniques that malware architects use to make it difficult for us to reverse engineer their creations
- understanding how the system stack is used -- or not -- in passing arguments
- turning what first appeared to be arbitrary data into clearly articulated code (de-obfuscation) and
- identifying lots of malware "indicators"
This class provided me with ...
- surprising insights both into how malware works and how I can analyze it
- super tips on what to watch out for and what to ignore
- surprising insights into how malware authors try to make their agents of evil hard to analyze -- more than I would ever have imagined
- ways to make the process of recognizing malware and identifying what it does considerably faster
- how to focus on the "deliverables" -- the answers that your management is waiting to hear
This AoE (Reverse Malware Engineering) class ranks in my eyes as best of breed, top notch, and highly recommended.
Read more of Sandra Henry-Stocker's Unix as a Second Language blog and follow the latest IT news at ITworld, Twitter and Facebook.