Chapter 6: Incident Investigation and Forensics

Cisco Press

1 2 Page 2
Page 2 of 2

Caution - Be wary of simply clicking the Push button! MARS might have made a mistake in determining the switch port. Avoid potential network outages by double-checking your switch connectivity. The port that MARS has determined might be a trunk port connecting to a different switch that is not monitored by MARS. Disabling the port, in that case, can interrupt network access of all hosts on the other switch.

This is not a common occurrence, but you should always be careful.

If you would like to change the view, you can select among Layer 2, Layer 3, and Full Topology views. Additionally, if multiple mitigation devices appear on the left panel, you can select them to see different recommendations for mitigation. However, only the Layer 2 mitigation enables the Push button.

Viewing Raw Log Messages

In addition to viewing the incident with one or more of the visual tools, you often want to see the raw log messages, as they came from the monitored device. You can do this by clicking the icon that looks like a sheet of paper with 0s and 1s on it in the Reporting Device column, as displayed in Figure 6-4. Figure 6-12 displays the raw logs from our sample incident. You can see that the reporting device is a Cisco IPS sensor. The raw data you view includes both the packet that triggered the event notification and the alert from the IPS sensor.

Figure 6-12

View Raw Log Messages

Note - Notice that after you have opened a case, it remains open through all MARS screens. This allows you to add information or incidents whenever needed. You need to close the case when you are finished with it.

Tracking Other Attacker Activities

You might find it useful to run one or more reports to see what other activities the attacker has been up to. To do this, click the Query icon (the icon with a q in it) next to the attacker's IP address (in the Source IP/Port column) in MARS, as displayed in Figure 6-4. This prepopulates a query for you. The query, Event Types Ranked by Sessions, filtered to include only the attacker's source IP address, might be enlightening, as you can see from Figures 6-13 and 6-14. You might also want to run the query Destination IPs Ranked by Session to see which hosts that attacker has also been communicating with.

Figure 6-13

Results for a Query Filtered by Attacker Source IP Address: Part I

Each time you find a report or query that looks useful, click the Add This Report button that appears at the upper-right corner of this page (refer to Figure 6-13) to attach that report to the case you opened earlier. Each query or report that you attach to a case is easily accessible later. In addition, if you see more incidents that seem to be related, be sure to add them to the same case. At any time, you can click the case number at the top of the page to pull up a summary of everything that's attached to the case, as demonstrated in Figure 6-15.

Figure 6-14

Results for a Query Filtered by Attacker Source IP Address: Part II

Figure 6-15

Case Summary

Determining What an Event Means

If you aren't sure what an event means, click the event title in the Incident ID table. A description of the event will be shown, as shown in Figures 6-16, 6-17, and 6-18, explaining what the event means, which hosts are likely to be affected by it, and which security products supported by MARS can detect it.

Figure 6-16

Event Type Details

Figure 6-17

Event Type Details, Continued

Figure 6-18

Event Type Details, Continued

Finishing Your Investigation

Case notes are important to help you maintain your information. Whenever you find more useful information, click the More button (at the top of any page) to open a text area for adding notes. Additionally, if you ever lose your place and want to return to the incident, you can always click the case name that appears at the top of every page until the case is deselected.

When you click the case name, you'll see a page that's similar to Figure 6-15, showing all information gathered so far.

A full case history appears at the bottom of the Case History page. At the bottom of the page, you can click the View Case Document button. This opens a single printable document that shows all information gathered so far, including the incident, all reports, and the full case history.

If you need to work on a different case, be sure to deselect the current case or close it if it is completed.

False-Positive Tuning

You'll often find that what first appears to be malicious activity is normal traffic. When this happens, you need to tune either a monitored device or MARS itself to remove the unnecessary incidents.

Deciding Where to Tune

You have to decide where it makes the most sense to tune. Tuning typically means one of two things:

  • A certain network behavior that looks malicious is actually normal.

  • A network behavior isn't bad in certain circumstances.

When you need to tune, is it easier and more effective to configure the security or network device to not send events in a certain condition, or is it easier and more effective to configure MARS to ignore certain events? This decision varies according to your needs and the capabilities of your monitored devices and software, in addition to the way your organization is structured.

As an example, if your organization has only a single network IPS, it might be equally easy to tune the events on the sensor or on the MARS appliance. However, if you have more than one IPS sensor, and maybe even more than one vendor, it quickly becomes unworkable to tune at the sensor.

If separate teams within your organization manage devices or applications such as IPS, antivirus, firewalling, and security monitoring, it also might be unworkable to tune at the network device.

In general, most organizations decide to tune at the network device only if the event is a valid false positive—meaning that the security event is falsely identifying what it is supposed to identify. All other tuning is done on the MARS appliance.

A valid false positive is different from wanting to filter an event that violates your security policy only if it occurs on certain hosts.

Legitimate network traffic that is falsely identified as a network attack is a valid false positive. MARS uses the term false positive somewhat loosely. For example, if a legitimate attack is launched at a host on your network that is not vulnerable to the attack because it has been sufficiently patched, MARS considers the attack a false positive. While this intelligence is convenient and saves a lot of work on your part, it is not a false positive by the actual definition of the term.

Tuning False Positives in MARS

Within MARS, you have the following three ways to tune false positives:

  • False Positive Wizard—When you are looking at the Incident Details page, as shown in Figure 6-19, you can tune events by clicking the False Positive link at the far right of each line. This steps you through the process of either ignoring events or not creating incidents on them.

  • Create or edit a drop rule—This is a more flexible way of accomplishing the same thing that the False Positive Wizard does. However, instead of using a step-by-step wizard, you create it much like any rule.

  • Modify a system rule—The previous two methods of tuning allow you to create a rule that works like an exception to another rule. By modifying an existing system rule, you exclude some conditions without the need for an additional rule. In general, you are discouraged from tuning with this method, but some rules are best handled this way. For example, the built-in rules called Inactive CS-MARS Reporting Device and Client Exploit—Mass Mailing Worm are both intended to be modified to only include applicable hosts.

Figure 6-19

False Positive Wizard Link

Using the False Positive Wizard

Figure 6-19 showed the incidents created when a host that is connected to a monitored switch is rebooted or unplugged. IOS interface hardware has gone up or down. If you click the Raw Events icon next to the reporting device in the incident table, you see Figure 6-20. This incident was triggered because interface FastEthernet1/0/20 on the monitored switch changed state to Down. This occurs regularly anywhere people shut down or restart their computers.

Figure 6-20

Raw Event Data

Is this really a false positive? No, it is not. However, it is a log message that you might not care about, and it would be nice to remove the clutter of these incidents from the MARS console.

The False Positive Wizard makes it easy to filter out these messages. Click the False Positive link on the Incident Details screen shown in Figure 6-19. The wizard launches, looking much like Figure 6-21. Select the event type at the top of this screen, and select the most appropriate source and destination addresses. You do not have a great deal of flexibility in selecting addresses at this time, but you can edit the drop rule this creates later to make it more granular. Click the Tune button.

Figure 6-21

Launching the False Positive Wizard

The second step in the wizard is to choose whether to drop events or log them to the database only. If you decide to drop the events, be aware that they are deleted. You cannot run reports on the events that would have been created. If you decide to log the events to the database, they cannot be used in a rule, and therefore do not appear as an incident. However, they will still be accessible if you ever need to run a report on them.

If in doubt, log them to the database and click the Next button, as illustrated in Figure 6-22.

Figure 6-22

Select Tuning Option

You are nearly finished with the wizard. If you look at Figure 6-23, you can see that the completed drop rule is displayed. If you want to change any of the fields, click the desired field. Recommended practice is that you at least click the rule name and description to make it easier to understand later. When finished, click the Confirm button.

Figure 6-23

Completed Drop Rule from the Wizard

Creating or Editing a Drop Rule Without the False Positive Wizard

Although the False Positive Wizard is user friendly and simple, at times it is quicker and easier, or at least more powerful, to create or change a drop rule without the wizard. An example of this would be when you want to tune for multiple source or destination IP addresses, or when you want to use an entire range of addresses while excluding just a few.

From the main window, click the Rules button and then click the Drop Rules tab. All drop rules on the appliance appear as shown in Figure 6-24, whether they were created manually or with the False Positive Wizard.

Figure 6-24

Drop Rules

Editing a System Rule

Typically, you should not edit the built-in rules. However, you'll occasionally find it necessary to do this. One example that most companies run into is with the Client Exploit—Mass Mailing Worm rule, as illustrated in Figure 6-25.

Figure 6-25

Client Exploit—Mass Mailing Worm

This rule is triggered when a single host sends more than 20 e-mail messages in a single minute. In general, hosts should never send this many e-mails. However, your company's Simple Mail Transfer Protocol (SMTP) relay servers might send out many times more than this each minute, and this does not mean they are infected with a worm.

The best way to tune these incidents is to exclude the SMTP servers within the rule itself. If you have more than one SMTP server in your organization that should be exempt from this rule, it is easiest to first create a group. Creating a group is an optional step, but it can make your life easier in the future. Follow these steps to create a group:

Step 1   From the main window, click the MANAGEMENT button, and then click the IP Management tab. The resulting page should look like Figure 6-26.

Step 2   Click the Add Group button to show the next window, as illustrated in Figure 6-27.

Figure 6-26

IP Management

Figure 6-27

Add Group

Step 3   Provide a name for the group, such as "SMTP Servers." For the group type, select Generic Network Group. In the right panel, select All IP Addresses, and then scroll through the list to find the IP addresses of your SMTP servers. Select these servers and click the Add button to add them to the left panel. When finished, click the Submit button.

This group is now available to all rules, queries, and reports.

Step 4   Modify the system rule, Client Exploit—Mass Mailing Worm. Click the Rules button and locate the rule. You might need to broaden the number of rules that appear on a single page and search by pressing Ctrl-F. When you find the rule, place a check mark next to it and click the Edit button at the top or bottom of the page. Refer to Figure 6-25 to see the resulting page.

Step 5   For sources, select Network Groups in the right panel and select the SMTP Servers group. Then click the Not Equal icon, which looks like !=.

Step 6   Click the Apply button. You should see a rule summary that looks like Figure 6-28. In the Source IP column, notice that the SMTP Servers are now excluded from the rule. These hosts will now be ignored by this rule.

Figure 6-28

Rule Summary

Tip - When you're finished modifying rules or devices, always click the Activate button at the upper-right corner of the screen.


In this chapter, you learned valuable information regarding how to use CS-MARS during the investigation of a security incident. Remember the six steps to properly handle an incident:

Step 1   Preparation—This includes a policy that describes how you are to handle the incident, in addition to prior training in using the tools at your disposal, such as CS-MARS.

Step 2   Identification—Use CS-MARS and other tools at your disposal to determine what occurred.

Step 3   Containment—Isolate the incident.

Step 4   Eradication—Repair or replace compromised systems.

Step 5   Recovery—Put your systems back online.

Step 6   Lessons learned—Learn from the incident. Train others as well.

Additionally, this chapter taught you how to tune MARS for your environment.

Copyright © 2007 Pearson Education. All rights reserved.

Learn more about this topic

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2007 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
SD-WAN buyers guide: Key questions to ask vendors (and yourself)