Data Visualization with Treemaps - A hands-on Tutorial

Have you tried using the Information Visualization Process with some of your data? Have you applied the process with some of the Tools that I introduced? Or any other visualization tool?

Today's topic is about generating treemaps. I use treemaps for a lot of different analysis purposes. Just a few examples: vulnerability data analysis, IDS signature tuning, or firewall log analysis. For today we are going to have a look at firewall log analysis with treemaps.

To start with, we need to convert our firewall log into a CSV file. I have an OpenBSD pf firewall. To parse the logs, I am using a parser that I wrote a while ago. If you don't have a pf log, still try to find a parser that someone wrote. If you can't find one, use awk to extract the necessary information from your logs. For my purposes I am extracting the source and destination IPs and ports, as well as the action, whether a packet was passed or blocked. The output looks something like this:

$ cat pflog.txt | pf2csv.pl "sip dip sport dport action" 
62.245.245.139,212.254.110.98,,echo request,pass
67.163.167.90,212.254.110.100,2424,25,pass
195.141.69.45,192.26.92.32,1030,53,pass
...

The next step is that we need to convert this CSV file into a TM3 file. The Treemap tool that I am going to use requires this data format. The following is the command you can use to convert the CSV output to a TM3 file:

cat file.csv | perl -pe 's/,/^I/g' | sort | uniq -c | perl -pe 's/^\s*//, s/ /^I/'

The other thing you will have to do then is adding a header row, such that the output looks as follows:

Count   SIP     DIP     SPORT   DPORT   ACTION
INTEGER	STRING	STRING	STRING	STRING	STRING
4	195.141.69.42	239.255.255.253	63700	427	block
4	195.141.69.42	239.255.255.253	63701	427	block
1	195.141.69.43	195.141.56.5	54218	53	pass

Note that in the command above, you need to use tabs instead of the ^I! Once we have this, we can open the Treemap tool:

java -jar treemap.jar

Open the TM3 file that you just generated. Then, on the right-hand side, choose Hierarchy. Click on DEFAULT HIERARCHY and then REMOVE. Then click on SIP and Add. Do the same for DIP. Now switch to Legend and change the Size to Count. Then change Color to ACTION. And finally set the Label to DPORT. Not bad, is it? You can also play around and change the color scheme, if you want. That's what I did in the treemap below.

What are we looking at here? For each source address (e.g., 212.251.89.126) you see what targets they connected to (e.g., 212.254.110.100). Basically, you see all the connections made. Then, inside of the target machine boxes, you can see all the services that these connections accessed (e.g., 135, or echo request). The color then indicates whether that connection was blocked (red) or passed (green). In all, this gives you a nice overview of your firewall activity. You should now be able to identify irregularities, mis-configurations, or attacks.

One of the interesting things in this log seems to be that there are a lot of echo requests floating around. Some got blocked (lower right and lower left red boxes). However, some were allowed through. We might want to confirm why some of these pings were needed. In addition, there seems to be a misconfiguration of 212.251.89.126. It tries to connect to a bunch of boxes on port 135 (Windows NetBIOS ). If it is blocked, it shouldn't try it. We should probably reconfigure that machine to stop doing this. The same is true for the port 427 traffic, which is associated with some MAC discovery protocol. There is a lot more that can be seen. You can, for example, identify your DNS server (port 53 traffic), etc. etc. Especially if you know more about your setup, you can identify all kinds of things with this treemap view.

If you go ahead and generate some interesting logs, upload them on secviz. If any of the steps above did not make sense, or you want me to elaborate on anything, please drop me a comment and I will happily do so.

Until next time when we are going to talk about the chart decision process.

Contact: Raffael Marty
Twitter: @zrlram

Raffy's Recent Blog Posts:

• Information Visualization Process
• The Tools I Use to Visualize Data
• There is no One Security Visualization Tool

• firewall
• firewall visualization
• security
• security visualization
• treemap
• visualization