In these tough economic times many companies are hunkering down and trying to keep their security program running while minimizing spending. Although it requires you to invest some time to get it moving, log monitoring is one low-cost technology that can pay big dividends.
Why monitor logs? Log files are your early warning system that something is amiss (not to mention that by regularly monitoring logs you will be compliant with numerous information security frameworks). Whether someone is using an application inappropriately or a piece of hardware is beginning to fail, all of that shows up in logs. The key is to find, activate, monitor and follow up on your logs on a regular basis.
Reading log files can help meet numerous goals. But log management is not an easy because 1) there is a lot of data, and 2) you need resources to analyze it. Security Event Information Systems (SEIM) attempts to achieve this through automation. If you don’t have automation, then you have to either do it manually or semi-automate the process with tools you can afford.
But before you even decide how, you must decide what. Here’s where a risk assessment is helpful. If you are able to decide through a proper risk analysis what logs you should focus your attention on (not just the default ‘everything’), this will make your endeavor more likely to succeed. Your list to evaluate includes:
* Operating system (both server and clients)
* Applications (this is a broad category that covers everything from Web servers to accounting packages to antivirus software)
* Network devices (everything from firewalls to routers and switches)
* Storage-area networks
This is a short list, but keep in mind that almost every that is worth something in your network world has a log file. That’s why having a risk assessment that will help set priorities is important. For example, you might decide that your priority is monitoring only application activity for unauthorized access. This is straightforward and will make the next steps easier. But if you decide that you really need to examine everything at least once and a while (a good way to go) then you need to set up your process and tools to accommodate. At this stage, the only investment made is time.
Once you determine what is most important to you, then you are ready to dive in to base lining your process. We’ll just assume you have done a risk assessment and have an idea of the devices that need to be reviewed on a regular basis. If you have ever cracked open a log file, you’ll know that this can be daunting if done manually. But, if you break it down into manageable steps, you can achieve your goal.
First step: make sure that your logs are turned on. I know that this may be hard to believe, but many log files are not on by default. If you are in a Windows shop, implement a Group Policy Object (GPO) to universally turn them on for clients and servers. The key is to ensure that you also set a size limit where the log file will roll over and/or delete itself, not just run until you fill up the disk.
On Linux/Unix systems, your log files are probably running, but perhaps they aren’t logging everything you want. Ensure your syslog.conf file is configured for important information. If you are unsure what you want to log, leave the defaults and move on to step 2. You may find there are areas where you’d like to get more granular and you can tweak the setting accordingly.
Applications will vary on type of logging and how it’s configured. Somewhere in the setup or management console there is a place to check this. Make sure this information then gets into your operational documentation so you can find it later.
Step 2 in the baseline process: use your system/application prioritized inventory to pick some of your most important systems and applications. Let’s say you’ve picked your servers to start with (don’t try to tackle everything at once; get comfortable with the servers, move on to clients, then on to applications, then to network devices). Pick your highest risk servers and take a look at the system logs. Read a few weeks worth of logs and look for trends.
Do you see errors or warnings? Do you see them frequently? Do you know what they mean? Cut and paste them to a single document to follow up on the logs so you can return to the specific issue later. Try to get through a whole log before you start to investigate, otherwise you will get distracted and never finish.
Let’s look at a couple of examples.
Say you are reviewing your Sun OS logs because you are having problems starting up the Domain Name Service (aka named) and you see this:
Apr 11 18:32:30 monkey named[3521]: [ID 873579 daemon.notice] starting BIND 9.3.5-P2
Apr 11 18:32:30 monkey named[3521]: [ID 873579 daemon.error] /etc/named.conf:110: change directory to ‘/var/named’ failed: file not found
Instantly you know that either you are missing a file or perhaps your permissions aren’t quite right for the service to start.
On another example, say you are on a routine review when you spot:
May 4 22:43:53 monkey2 sshd[17904]: reverse mapping checking getaddrinfo for hostname [10.10.1.130] failed – POSSIBLE BREAK-IN ATTEMPT!
May 4 22:43:53 monkey2 sshd[17904]: Invalid user admin from 10.10.1.130
May 4 22:43:53 monkey2 sshd[17904]: Failed none for invalid user admin from 10.10.1.130 port 51502 ssh2
May 4 22:44:00 monkey2 sshd[17904]: error: Could not get shadow information for NOUSER
May 4 22:44:00 monkey2 sshd[17904]: Failed password for invalid user admin from 10.10.1.130 port 51502 ssh2
May 4 22:44:16 monkey2 sshd[18017]: reverse mapping checking getaddrinfo for hostname [10.10.1.130] failed – POSSIBLE BREAK-IN ATTEMPT!
These entries may point to either someone who is attempting to break in by trying different accounts through Secure Shell (sshd) or that someone has forgotten their legitimate login credentials and tried a couple of possibilities. Either way, this will be something to look further into.
Lastly, you are reviewing your Vista client for issues, and you find this message: Your computer was not able to renew its address from the network (from the DHCP Server) for the Network Card with network address 000FFE54093C. The following error occurred: The semaphore timeout period has expired. Your computer will continue to try and obtain an address on its own from the network address (DHCP) server.
This error could mean a couple of things: either your NIC could be failing or perhaps you are having issues on the DHCP server itself. Again, worth a look into.
Step three: Once you have your follow up list, talk to your peers, go online or look up errors in the manual to see what the issues mean. If they mean that something is misconfigured or failing, make sure these get addressed. Once you do this analysis a couple of times, you will weed out the messages that you aren’t interested in and be able to focus on the ones that are interesting.
Repeat this process until you work your way through your list of systems and applications that need monitoring. Once you feel comfortable with them, you can set up a schedule to sample the logs and it will take less time because now you know what you are looking for.
There are some excellent free tools that may help to automate this process, or at least bring together logs from disparate systems to review in one place. For example, Splunk has a free tool to help consolidate and analyze several disparate log file types. As the Web site says: “Splunk is a self-contained software package that runs on lots of different operating systems. Just pick your platform, download and install.” And Kiwi has a free syslog management server (along with products for purchase).
And why not check out Event Log Analyzer? Its free version is limited to monitor five hosts.
Once you have a process and/or a tool that will help you monitor your log files, it’s imperative that you keep it up. This program cannot be successful if you do it once and then never look at the files again, or only look at them when you have a problem. Proactive log management can address risks before they become problems; therefore, helping to contribute to your organization’s bottom line. Isn’t that worth the investment of your time? In these shaky budgetary times, it’s a must.
Westphal is an IT professional of 16 years with specific experience in the area of information security. She is the CISO of the Arizona Department of Economic Security. She is skilled in troubleshooting and process analysis, with specific expertise in security areas including forensics, operating system and network security, intrusion detection, incident handling, vulnerability analysis and policy development. Wetphal has been a CISSP since 2001 and a CISA since 2008. You can reach her at kmwestphal@cox.net.
