Cisco Press
Rate your favorite Cisco Press books.
When a serious incident occurs, you need to know what to do. A serious incident will eventually occur with all organizations, and it could take many forms. For example, it might be any of the following:
Sensitive financial information about your company or employees is stolen and posted to a hacker blog.
An e-mail worm attacks your e-mail system, resulting in degraded network performance.
An employee is inadvertently sharing all his Word documents on Limewire, Kazaa, or some other peer-to-peer (P2P) file-sharing network.
You are notified by a motion picture association that someone on your network is downloading and distributing copyrighted material.
Your e-commerce website falls victim to a distributed denial of service (DDoS) attack. In a DDoS attack, tens or hundreds of thousands of hosts all simultaneously attack your site, rendering it inaccessible to legitimate users.
The following six steps can help you properly handle an incident:
Step 1 Preparation—It is always too late if an organization waits until an incident occurs to learn how a device can capture packet data or run debugging commands. The first step in incident handling is to learn about your equipment and tools and have a plan.
Step 2 Identification—You must identify the incident. This step highlights one of the strengths of the Cisco Security Monitoring, Analysis, and Response System (CS-MARS). Through its capability to use both built-in and user-defined rules, as well as the capability to detect anomalous traffic, MARS enables you to rapidly identify and respond to new incidents. This chapter can help you understand what you should do when you discover an incident.
Step 3 Containment—To contain an incident means to use a control, temporarily or permanently, to isolate or stop a security incident. Additionally, containment includes collection of available forensic data. This could mean unplugging a network cable or disabling a switch port. It could also mean changing a firewall rule to prevent a host from communicating across the firewall. Containment, or mitigation, as it is called within CS-MARS, is a task MARS can greatly assist you with and can contain a compromised device, giving you time to identify what occurred. This chapter describes the types of mitigation that you can accomplish with MARS and its importance to your incident-handling plan.
Step 4 Eradication—You must repair the compromised system to the state it was in prior to being attacked. It is often difficult, or even impossible, to fully repair the compromised system. Many organizations instead rebuild the system from scratch and restore only the necessary data. Eradication is a time-intensive step, and it should be performed only after all forensic data has been captured. In fact, you should consider replacing the compromised system with a new system, especially if you will be pursuing legal channels against the attacker. This chapter covers forensics and replay, but it is beyond the scope of this book to detail eradication in general.
Step 5 Recovery—In this step, the replacement or cleaned system is placed back into service. This should be done carefully, and controls should be in place to identify any further attempts to attack the system. MARS should have rules defined to watch for these attacks. Again, it is beyond the scope of this book to detail placing the system into service, but it is important to understand the process.
Step 6 Lessons learned—In this step, you look over the incident and make note of what you have learned. This is a time to reconsider your security controls, policies, and procedures and answer the following questions:
— Do you need to change anything to prevent a recurrence or similar compromise in the future?
— How much did it cost to troubleshoot and correct the incident?
— Did your organization sustain damage to its reputation?
Discuss the incident with your staff and management, if necessary, and then document the entire process. This can help you operate better the next time an incident occurs.
This book is not intended to be your guide to developing an incident-handling plan. However, it is useful for demonstrating how CS-MARS can give you better access to the information you need when an incident occurs. You can find excellent references to the preceding steps at the SysAdmin, Audit, Network, Security (SANS) Institute and the National Institute of Standards and Technology (NIST). Here are a few Internet links:
NIST’s Computer Security Incident Handling Guide—https://csrc.nist.gov/publications/nistpubs/800-61/sp800-61.pdf
U.S. Security Awareness—https://www.ussecurityawareness.org/highres/incident-response.html
The most important step in the preceding list on incident handling is the first step—preparation. Preparation allows rapid, orderly processes, and is a sign of the maturity of an organization’s view on security. Step 6—Lessons learned—helps your organization grow and prepare for the next incident.
MARS directly relates to both Step 2, Identification, and Step 3, Containment, in the incident-handling process.
Note – You can find more information about incident handling from the National Institute of Standards and Technology (NIST) in Special Publication 800-61, as previously referenced. These steps are considered by many to be the standard procedure for incident handling.
Incident Handling and Forensic Techniques
This section walks you through a sample incident using the Identification and Containment steps of incident handling.
ACME Widgets has had a suspicious incident appear on the MARS Dashboard. According to MARS, a Windows RPC DCOM Overflow attack has occurred (see Figure 6-1).
Figure 6-1
Suspicious Incident, as It Appears on MARS Dashboard
Initial Incident Investigation
It is often a good idea to create a case, especially when you are investigating what appears to be a serious security incident. After a case is opened, you can return to it and attach more incidents or change the status.
To create a case, drill down into the incident by clicking the incident ID. Then, click the New Case button, which appears in the upper-right corner of the screen, as illustrated in Figure 6-2.
Figure 6-2
Drilling into an Incident
As you open a case, you select a case severity level, or importance—Red, Yellow, or Green, as demonstrated in Figure 6-3—and can assign the case to an individual with comments.
If someone else should be assigned to this case, you can assign it to that person by selecting his or her name from the drop-down list. The names are automatically populated from the MARS user database.
Note – Shared logins should be avoided. Make sure that each MARS user has a unique login ID and password.
Try to be clear and concise in your notes. This can help you later when you review the incident, or if you need to work with law enforcement.
Create a Case
After you have created the case, it is time to begin investigating the incident. You might want to think about whether your actions will alert the attacker that he has been detected. For example, if the attack is in progress or the attacker is capturing keystrokes, pinging, or performing a host lookup, using common Domain Name System (DNS) tools such as nslookup or dig might signal the attacker that he is being investigated. If you have not yet isolated the compromised system, and the attacker learns he has been seen, he might attempt to erase his tracks or even destroy the compromised system.
Figure 6-4 shows the series of sessions and incidents that are related to the incident you’re looking at. It appears that this incident is isolated, but a couple other instances of this incident have occurred. This is apparent from the offset of 3 that appears next to the incident ID. From this screen, if you click an IP address, MARS automatically performs a DNS lookup on that IP address. If you’re dealing with a sophisticated attacker, this could signal him that he’s been seen. This shouldn’t prevent you from performing the lookup, but at least be aware of the possibilities and use discretion as it pertains to your circumstances.
Figure 6-4
Tracking Affected IP Addresses
When the incident occurs, MARS performs some additional forensic duties automatically. It attempts to determine where on the network this host resides, and which security or network devices sit between this host and the attacking host. Additionally, MARS attempts to determine what the operating system version and patch level are and which listening applications are running on the suspect TCP or User Datagram Protocol (UDP) port. All this information is used to determine whether an attack was successful, or even if the host is vulnerable to this type of attack. Log messages from other reporting hosts are correlated to determine whether the attack traffic was blocked within the path of the attack.
If you click the destination IP address of this incident, you see that the target host is a Cisco Security Agent (CSA) Management Center, as Figure 6-5 shows.
Figure 6-5
Host Information by Clicking IP Address
You can click the source IP address, as well, to get information on the attacker. Additionally, you can access useful information by clicking the port numbers. For example, if you click the destination port number of this incident, as shown in Figure 6-6, you can find out that TCP port 135 is commonly used for “Microsoft_RPC_DCE_Endpoint_ Resolution” and has also been associated with the MSBlaster worm.
Referring to Figure 6-4 again, you now know that there was at least one attempt to communicate with the CSA Management Center, with Microsoft’s endpoint resolution protocol, and your Cisco intrusion prevention system (IPS) sensor identified this traffic as malicious. The source address of the attacker was 192.168.2.5, which sits outside one of your firewalls.
Figure 6-6
Port Information
Viewing Incident Details
MARS offers several graphical views that can assist you in understanding, or even mitigating, the incident. On the Incident Details page, shown previously in Figure 6-4, if you look next to the incident ID, you can see two icons. The icon that looks like a star provides a logical view of the incident, and also provides the capability to step through the incident, one event at a time. The other icon provides the same information, displayed in a physical view. Figure 6-7 shows the logical view provided in the attack diagram.
Logical View Shown in Attack Diagram
Notice that you can click the Next and Previous buttons to step through an incident one session at a time. Hosts are color coded in all the graphical views. The host that is being attacked is shown in red, while the host that is attacking is shown in brown. The following list explains the color codes used:
Brown host—This host’s behavior indicates that it might be an attacker.
Red host—This host appears to be under attack. It is the victim.
Purple host—This host has performed as both the attacker and the victim. This might be a compromised host.
Figures 6-8 and 6-9 show the physical views of the incident. This is accessed by clicking the other icon next to the incident ID on the Incident Details page in Figure 6-4. Initially, you see the topology view in Figure 6-8, which shows only the hosts involved in the attack, as well as subnets and other devices within the path of the attack. However, by clicking the Toggle Topology button, you can see the attack overlaid with other devices near the incident, as illustrated in Figure 6-9.
Figure 6-8
Physical View Shown on Incident Graph
Figure 6-9
Incident Graph After Toggling Topology
The third way to graphically view the incident is to click the Path/Mitigate button near the right edge on each line of the Incident ID table, as illustrated in Figure 6-4.
Clicking this icon causes MARS to perform some behind-the-scenes investigation. If possible, MARS determines which monitored devices can block the attack in the future. For example:
A firewall might restrict access from the attacker to one or more hosts or networks.
A router might have an access control list applied to restrict access.
A switch can have a port disabled.
If the attacking host is internal to your network, and is connected to one of the switches that MARS is monitoring, MARS attempts to determine the attacker’s MAC address. MARS regularly queries switches and routers on your network to maintain this information, which exists in the content addressable memory (CAM) and Address Resolution Protocol (ARP) tables in your network devices. When MARS has this information, it can locate the host and the switch port to which the host is connected.
After MARS has evaluated the Layer 2 and Layer 3 paths involved in this attack, it presents options for mitigation. You should understand that MARS does not automatically mitigate an attack. Instead, it provides information to you. Figures 6-10 and 6-11 show the mitigation screen presented in this attack. MARS has determined that the best way to stop the attack, or prevent it from recurring, is to create an access control list entry on the PIX firewall.
Session Graph and Mitigation Options
Figure 6-11
Session Graph and Mitigation Options, Continued
Occasionally, you might also be presented with the option to disable the switch port that the attacker is connected to. In fact, you can click a red Push button to instantly disable the port.
Caution – Be wary of simply clicking the Push button! MARS might have made a mistake in determining the switch port. Avoid potential network outages by double-checking your switch connectivity. The port that MARS has determined might be a trunk port connecting to a different switch that is not monitored by MARS. Disabling the port, in that case, can interrupt network access of all hosts on the other switch.
This is not a common occurrence, but you should always be careful.
If you would like to change the view, you can select among Layer 2, Layer 3, and Full Topology views. Additionally, if multiple mitigation devices appear on the left panel, you can select them to see different recommendations for mitigation. However, only the Layer 2 mitigation enables the Push button.
Viewing Raw Log Messages
In addition to viewing the incident with one or more of the visual tools, you often want to see the raw log messages, as they came from the monitored device. You can do this by clicking the icon that looks like a sheet of paper with 0s and 1s on it in the Reporting Device column, as displayed in Figure 6-4. Figure 6-12 displays the raw logs from our sample incident. You can see that the reporting device is a Cisco IPS sensor. The raw data you view includes both the packet that triggered the event notification and the alert from the IPS sensor.
Figure 6-12
View Raw Log Messages
Note – Notice that after you have opened a case, it remains open through all MARS screens. This allows you to add information or incidents whenever needed. You need to close the case when you are finished with it.
Tracking Other Attacker Activities
You might find it useful to run one or more reports to see what other activities the attacker has been up to. To do this, click the Query icon (the icon with a q in it) next to the attacker’s IP address (in the Source IP/Port column) in MARS, as displayed in Figure 6-4. This prepopulates a query for you. The query, Event Types Ranked by Sessions, filtered to include only the attacker’s source IP address, might be enlightening, as you can see from Figures 6-13 and 6-14. You might also want to run the query Destination IPs Ranked by Session to see which hosts that attacker has also been communicating with.
Results for a Query Filtered by Attacker Source IP Address: Part I
Each time you find a report or query that looks useful, click the Add This Report button that appears at the upper-right corner of this page (refer to Figure 6-13) to attach that report to the case you opened earlier. Each query or report that you attach to a case is easily accessible later. In addition, if you see more incidents that seem to be related, be sure to add them to the same case. At any time, you can click the case number at the top of the page to pull up a summary of everything that’s attached to the case, as demonstrated in Figure 6-15.
Results for a Query Filtered by Attacker Source IP Address: Part II
Case Summary
Determining What an Event Means
If you aren’t sure what an event means, click the event title in the Incident ID table. A description of the event will be shown, as shown in Figures 6-16, 6-17, and 6-18, explaining what the event means, which hosts are likely to be affected by it, and which security products supported by MARS can detect it.
Figure 6-16
Event Type Details
Event Type Details, Continued
Figure 6-18
Event Type Details, Continued
Finishing Your Investigation
Case notes are important to help you maintain your information. Whenever you find more useful information, click the More button (at the top of any page) to open a text area for adding notes. Additionally, if you ever lose your place and want to return to the incident, you can always click the case name that appears at the top of every page until the case is deselected.
When you click the case name, you’ll see a page that’s similar to Figure 6-15, showing all information gathered so far.
A full case history appears at the bottom of the Case History page. At the bottom of the page, you can click the View Case Document button. This opens a single printable document that shows all information gathered so far, including the incident, all reports, and the full case history.
If you need to work on a different case, be sure to deselect the current case or close it if it is completed.
False-Positive Tuning
You’ll often find that what first appears to be malicious activity is normal traffic. When this happens, you need to tune either a monitored device or MARS itself to remove the unnecessary incidents.
Deciding Where to Tune
You have to decide where it makes the most sense to tune. Tuning typically means one of two things:
A certain network behavior that looks malicious is actually normal.
A network behavior isn’t bad in certain circumstances.
When you need to tune, is it easier and more effective to configure the security or network device to not send events in a certain condition, or is it easier and more effective to configure MARS to ignore certain events? This decision varies according to your needs and the capabilities of your monitored devices and software, in addition to the way your organization is structured.
As an example, if your organization has only a single network IPS, it might be equally easy to tune the events on the sensor or on the MARS appliance. However, if you have more than one IPS sensor, and maybe even more than one vendor, it quickly becomes unworkable to tune at the sensor.
If separate teams within your organization manage devices or applications such as IPS, antivirus, firewalling, and security monitoring, it also might be unworkable to tune at the network device.
In general, most organizations decide to tune at the network device only if the event is a valid false positive—meaning that the security event is falsely identifying what it is supposed to identify. All other tuning is done on the MARS appliance.
A valid false positive is different from wanting to filter an event that violates your security policy only if it occurs on certain hosts.
Legitimate network traffic that is falsely identified as a network attack is a valid false positive. MARS uses the term false positive somewhat loosely. For example, if a legitimate attack is launched at a host on your network that is not vulnerable to the attack because it has been sufficiently patched, MARS considers the attack a false positive. While this intelligence is convenient and saves a lot of work on your part, it is not a false positive by the actual definition of the term.
Tuning False Positives in MARS
Within MARS, you have the following three ways to tune false positives:
False Positive Wizard—When you are looking at the Incident Details page, as shown in Figure 6-19, you can tune events by clicking the False Positive link at the far right of each line. This steps you through the process of either ignoring events or not creating incidents on them.
Create or edit a drop rule—This is a more flexible way of accomplishing the same thing that the False Positive Wizard does. However, instead of using a step-by-step wizard, you create it much like any rule.
Modify a system rule—The previous two methods of tuning allow you to create a rule that works like an exception to another rule. By modifying an existing system rule, you exclude some conditions without the need for an additional rule. In general, you are discouraged from tuning with this method, but some rules are best handled this way. For example, the built-in rules called Inactive CS-MARS Reporting Device and Client Exploit—Mass Mailing Worm are both intended to be modified to only include applicable hosts.
Figure 6-19
False Positive Wizard Link
Using the False Positive Wizard
Figure 6-19 showed the incidents created when a host that is connected to a monitored switch is rebooted or unplugged. IOS interface hardware has gone up or down. If you click the Raw Events icon next to the reporting device in the incident table, you see Figure 6-20. This incident was triggered because interface FastEthernet1/0/20 on the monitored switch changed state to Down. This occurs regularly anywhere people shut down or restart their computers.
Figure 6-20
Raw Event Data
Is this really a false positive? No, it is not. However, it is a log message that you might not care about, and it would be nice to remove the clutter of these incidents from the MARS console.
The False Positive Wizard makes it easy to filter out these messages. Click the False Positive link on the Incident Details screen shown in Figure 6-19. The wizard launches, looking much like Figure 6-21. Select the event type at the top of this screen, and select the most appropriate source and destination addresses. You do not have a great deal of flexibility in selecting addresses at this time, but you can edit the drop rule this creates later to make it more granular. Click the Tune button.
Launching the False Positive Wizard
The second step in the wizard is to choose whether to drop events or log them to the database only. If you decide to drop the events, be aware that they are deleted. You cannot run reports on the events that would have been created. If you decide to log the events to the database, they cannot be used in a rule, and therefore do not appear as an incident. However, they will still be accessible if you ever need to run a report on them.
If in doubt, log them to the database and click the Next button, as illustrated in Figure 6-22.
Figure 6-22
Select Tuning Option
You are nearly finished with the wizard. If you look at Figure 6-23, you can see that the completed drop rule is displayed. If you want to change any of the fields, click the desired field. Recommended practice is that you at least click the rule name and description to make it easier to understand later. When finished, click the Confirm button.
Figure 6-23
Completed Drop Rule from the Wizard
Creating or Editing a Drop Rule Without the False Positive Wizard
Although the False Positive Wizard is user friendly and simple, at times it is quicker and easier, or at least more powerful, to create or change a drop rule without the wizard. An example of this would be when you want to tune for multiple source or destination IP addresses, or when you want to use an entire range of addresses while excluding just a few.
From the main window, click the Rules button and then click the Drop Rules tab. All drop rules on the appliance appear as shown in Figure 6-24, whether they were created manually or with the False Positive Wizard.
Figure 6-24
Drop Rules
Editing a System Rule
Typically, you should not edit the built-in rules. However, you’ll occasionally find it necessary to do this. One example that most companies run into is with the Client Exploit—Mass Mailing Worm rule, as illustrated in Figure 6-25.
Figure 6-25
Client Exploit—Mass Mailing Worm
This rule is triggered when a single host sends more than 20 e-mail messages in a single minute. In general, hosts should never send this many e-mails. However, your company’s Simple Mail Transfer Protocol (SMTP) relay servers might send out many times more than this each minute, and this does not mean they are infected with a worm.
The best way to tune these incidents is to exclude the SMTP servers within the rule itself. If you have more than one SMTP server in your organization that should be exempt from this rule, it is easiest to first create a group. Creating a group is an optional step, but it can make your life easier in the future. Follow these steps to create a group:
Step 1 From the main window, click the MANAGEMENT button, and then click the IP Management tab. The resulting page should look like Figure 6-26.
Step 2 Click the Add Group button to show the next window, as illustrated in Figure 6-27.
Figure 6-26
IP Management
Figure 6-27
Add Group
Step 3 Provide a name for the group, such as “SMTP Servers.” For the group type, select Generic Network Group. In the right panel, select All IP Addresses, and then scroll through the list to find the IP addresses of your SMTP servers. Select these servers and click the Add button to add them to the left panel. When finished, click the Submit button.
This group is now available to all rules, queries, and reports.
Step 4 Modify the system rule, Client Exploit—Mass Mailing Worm. Click the Rules button and locate the rule. You might need to broaden the number of rules that appear on a single page and search by pressing Ctrl-F. When you find the rule, place a check mark next to it and click the Edit button at the top or bottom of the page. Refer to Figure 6-25 to see the resulting page.
Step 5 For sources, select Network Groups in the right panel and select the SMTP Servers group. Then click the Not Equal icon, which looks like !=.
Step 6 Click the Apply button. You should see a rule summary that looks like Figure 6-28. In the Source IP column, notice that the SMTP Servers are now excluded from the rule. These hosts will now be ignored by this rule.
Figure 6-28
Rule Summary
Tip – When you’re finished modifying rules or devices, always click the Activate button at the upper-right corner of the screen.
Summary
In this chapter, you learned valuable information regarding how to use CS-MARS during the investigation of a security incident. Remember the six steps to properly handle an incident:
Step 1 Preparation—This includes a policy that describes how you are to handle the incident, in addition to prior training in using the tools at your disposal, such as CS-MARS.
Step 2 Identification—Use CS-MARS and other tools at your disposal to determine what occurred.
Step 3 Containment—Isolate the incident.
Step 4 Eradication—Repair or replace compromised systems.
Step 5 Recovery—Put your systems back online.
Step 6 Lessons learned—Learn from the incident. Train others as well.
Additionally, this chapter taught you how to tune MARS for your environment.
Copyright © 2007 Pearson Education. All rights reserved.
