Unix: Looking for evil in your firewall logs

Firewall logs. There's never enough time to review them, but you can't ignore them. Here's one way to look for malicious connections without spending a lot of time at it.

Firewall logs always contain far too much data for you to look into. With the likelihood that you're collecting millions -- if not tens of millions -- of records every day, you don't stand a chance of gathering meaningful insights from them unless you summarize or extract meaningful content. In today's post, we're going to look at a simple script that will tell you, given a list of known hostile addresses, whether any of them have connected to your systems (whether they initiated the connections or not) and how many times this has happened. This Perl script expects to find two files. I'm referring to them as log.txt (the firewall log) and bad.txt (a list of hostile IP addresses). Obviously, you can switch the names of these files in lines 5 and 6. This script should also be modified to reflect the name used for your external interface. This script assumes it is called "outside" and that your firewall logs will contain strings such as "outside:" showing the IP address and port of each external connection. Modify the regular expression shown in line 10 -- outside:(\S+)\/ -- if this is not the case. The \S+ extracts the name or IP address of the external system so that it can be added to a hash that also counts how many times we see this system as we comb through the log file one line at a time. The \/ specifies that a / follows the address, the \ being used as an escape to ensure the following / is taken literally.

<p>#!/usr/bin/perl -w</p>
<p>my %outside=();</p>
<p>open LOG,"<log.txt";<br>
  open BAD,"<bad.txt";</p>
<p># create a hash containing all external systems<br>
  while ( <LOG> ) {<br>
  if ( ! exists $outside{$ext} ) {<br>
  $outside{$ext}=1;			# add to hash<br>
  } else {<br>
  $outside{$ext}++;			# increment connection counter<br>
<p># look through list of hostile IP addresses to see if any have been seen in log<br>
  while ( <BAD> ) {<br>
  if ( exists $outside{$_} ) {<br>
  print "FOUND: $_ $outside{$_} time(s)\n";<br>

Once we have combed through the entire log and built our hash showing how many times each connection has occurred, we run through the list of known hostile addresses and look for a corresponding hash entry (i.e., evidence that we have had connections to the hostile systems). If we find any matches, we display a message such as "FOUND: 3982 time(s)". Say you have a list of known to be hostile IP address that starts like this:

The script will run through the second while loop once for each of these addresses looking to see if any match the addresses we have collected in our hash. You could do the same thing with grep, of course, but you would be grepping through your millions of records as many times as you have addresses in your hostile systems list and this could take many hours. I find this method of using a Perl hash to count occurrences and running through the firewall log only once to be much easier and considerably faster.

Read more of Sandra Henry-Stocker's Unix as a Second Language blog and follow the latest IT news at ITworld, Twitter and Facebook.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: 10 new UI features coming to Windows 10