Chapter 6: Incident Investigation and Forensics

Cisco Press

Rate your favorite Cisco Press books.

When a serious incident occurs, you need to know what to do. A serious incident will eventually occur with all organizations, and it could take many forms. For example, it might be any of the following:

  • Sensitive financial information about your company or employees is stolen and posted to a hacker blog.

  • An e-mail worm attacks your e-mail system, resulting in degraded network performance.

  • An employee is inadvertently sharing all his Word documents on Limewire, Kazaa, or some other peer-to-peer (P2P) file-sharing network.

  • You are notified by a motion picture association that someone on your network is downloading and distributing copyrighted material.

  • Your e-commerce website falls victim to a distributed denial of service (DDoS) attack. In a DDoS attack, tens or hundreds of thousands of hosts all simultaneously attack your site, rendering it inaccessible to legitimate users.

The following six steps can help you properly handle an incident:

Step 1   Preparation—It is always too late if an organization waits until an incident occurs to learn how a device can capture packet data or run debugging commands. The first step in incident handling is to learn about your equipment and tools and have a plan.

Step 2   Identification—You must identify the incident. This step highlights one of the strengths of the Cisco Security Monitoring, Analysis, and Response System (CS-MARS). Through its capability to use both built-in and user-defined rules, as well as the capability to detect anomalous traffic, MARS enables you to rapidly identify and respond to new incidents. This chapter can help you understand what you should do when you discover an incident.

Step 3   Containment—To contain an incident means to use a control, temporarily or permanently, to isolate or stop a security incident. Additionally, containment includes collection of available forensic data. This could mean unplugging a network cable or disabling a switch port. It could also mean changing a firewall rule to prevent a host from communicating across the firewall. Containment, or mitigation, as it is called within CS-MARS, is a task MARS can greatly assist you with and can contain a compromised device, giving you time to identify what occurred. This chapter describes the types of mitigation that you can accomplish with MARS and its importance to your incident-handling plan.

Step 4   Eradication—You must repair the compromised system to the state it was in prior to being attacked. It is often difficult, or even impossible, to fully repair the compromised system. Many organizations instead rebuild the system from scratch and restore only the necessary data. Eradication is a time-intensive step, and it should be performed only after all forensic data has been captured. In fact, you should consider replacing the compromised system with a new system, especially if you will be pursuing legal channels against the attacker. This chapter covers forensics and replay, but it is beyond the scope of this book to detail eradication in general.

Step 5   Recovery—In this step, the replacement or cleaned system is placed back into service. This should be done carefully, and controls should be in place to identify any further attempts to attack the system. MARS should have rules defined to watch for these attacks. Again, it is beyond the scope of this book to detail placing the system into service, but it is important to understand the process.

Step 6   Lessons learned—In this step, you look over the incident and make note of what you have learned. This is a time to reconsider your security controls, policies, and procedures and answer the following questions:

— Do you need to change anything to prevent a recurrence or similar compromise in the future?

— How much did it cost to troubleshoot and correct the incident?

— Did your organization sustain damage to its reputation?

Discuss the incident with your staff and management, if necessary, and then document the entire process. This can help you operate better the next time an incident occurs.

This book is not intended to be your guide to developing an incident-handling plan. However, it is useful for demonstrating how CS-MARS can give you better access to the information you need when an incident occurs. You can find excellent references to the preceding steps at the SysAdmin, Audit, Network, Security (SANS) Institute and the National Institute of Standards and Technology (NIST). Here are a few Internet links:

The most important step in the preceding list on incident handling is the first step—preparation. Preparation allows rapid, orderly processes, and is a sign of the maturity of an organization's view on security. Step 6—Lessons learned—helps your organization grow and prepare for the next incident.

MARS directly relates to both Step 2, Identification, and Step 3, Containment, in the incident-handling process.

Note - You can find more information about incident handling from the National Institute of Standards and Technology (NIST) in Special Publication 800-61, as previously referenced. These steps are considered by many to be the standard procedure for incident handling.

Incident Handling and Forensic Techniques

This section walks you through a sample incident using the Identification and Containment steps of incident handling.

ACME Widgets has had a suspicious incident appear on the MARS Dashboard. According to MARS, a Windows RPC DCOM Overflow attack has occurred (see Figure 6-1).

Figure 6-1

Suspicious Incident, as It Appears on MARS Dashboard

Initial Incident Investigation

It is often a good idea to create a case, especially when you are investigating what appears to be a serious security incident. After a case is opened, you can return to it and attach more incidents or change the status.

To create a case, drill down into the incident by clicking the incident ID. Then, click the New Case button, which appears in the upper-right corner of the screen, as illustrated in Figure 6-2.

Figure 6-2

Drilling into an Incident

As you open a case, you select a case severity level, or importance—Red, Yellow, or Green, as demonstrated in Figure 6-3—and can assign the case to an individual with comments.

If someone else should be assigned to this case, you can assign it to that person by selecting his or her name from the drop-down list. The names are automatically populated from the MARS user database.

Note - Shared logins should be avoided. Make sure that each MARS user has a unique login ID and password.

Try to be clear and concise in your notes. This can help you later when you review the incident, or if you need to work with law enforcement.

Figure 6-3

Create a Case

After you have created the case, it is time to begin investigating the incident. You might want to think about whether your actions will alert the attacker that he has been detected. For example, if the attack is in progress or the attacker is capturing keystrokes, pinging, or performing a host lookup, using common Domain Name System (DNS) tools such as nslookup or dig might signal the attacker that he is being investigated. If you have not yet isolated the compromised system, and the attacker learns he has been seen, he might attempt to erase his tracks or even destroy the compromised system.

Figure 6-4 shows the series of sessions and incidents that are related to the incident you're looking at. It appears that this incident is isolated, but a couple other instances of this incident have occurred. This is apparent from the offset of 3 that appears next to the incident ID. From this screen, if you click an IP address, MARS automatically performs a DNS lookup on that IP address. If you're dealing with a sophisticated attacker, this could signal him that he's been seen. This shouldn't prevent you from performing the lookup, but at least be aware of the possibilities and use discretion as it pertains to your circumstances.

Figure 6-4

Tracking Affected IP Addresses

When the incident occurs, MARS performs some additional forensic duties automatically. It attempts to determine where on the network this host resides, and which security or network devices sit between this host and the attacking host. Additionally, MARS attempts to determine what the operating system version and patch level are and which listening applications are running on the suspect TCP or User Datagram Protocol (UDP) port. All this information is used to determine whether an attack was successful, or even if the host is vulnerable to this type of attack. Log messages from other reporting hosts are correlated to determine whether the attack traffic was blocked within the path of the attack.

If you click the destination IP address of this incident, you see that the target host is a Cisco Security Agent (CSA) Management Center, as Figure 6-5 shows.

Figure 6-5

Host Information by Clicking IP Address

You can click the source IP address, as well, to get information on the attacker. Additionally, you can access useful information by clicking the port numbers. For example, if you click the destination port number of this incident, as shown in Figure 6-6, you can find out that TCP port 135 is commonly used for "Microsoft_RPC_DCE_Endpoint_ Resolution" and has also been associated with the MSBlaster worm.

Referring to Figure 6-4 again, you now know that there was at least one attempt to communicate with the CSA Management Center, with Microsoft's endpoint resolution protocol, and your Cisco intrusion prevention system (IPS) sensor identified this traffic as malicious. The source address of the attacker was, which sits outside one of your firewalls.

Figure 6-6

Port Information

Viewing Incident Details

MARS offers several graphical views that can assist you in understanding, or even mitigating, the incident. On the Incident Details page, shown previously in Figure 6-4, if you look next to the incident ID, you can see two icons. The icon that looks like a star provides a logical view of the incident, and also provides the capability to step through the incident, one event at a time. The other icon provides the same information, displayed in a physical view. Figure 6-7 shows the logical view provided in the attack diagram.

Figure 6-7

Logical View Shown in Attack Diagram

Notice that you can click the Next and Previous buttons to step through an incident one session at a time. Hosts are color coded in all the graphical views. The host that is being attacked is shown in red, while the host that is attacking is shown in brown. The following list explains the color codes used:

  • Brown host—This host's behavior indicates that it might be an attacker.

  • Red host—This host appears to be under attack. It is the victim.

  • Purple host—This host has performed as both the attacker and the victim. This might be a compromised host.

Figures 6-8 and 6-9 show the physical views of the incident. This is accessed by clicking the other icon next to the incident ID on the Incident Details page in Figure 6-4. Initially, you see the topology view in Figure 6-8, which shows only the hosts involved in the attack, as well as subnets and other devices within the path of the attack. However, by clicking the Toggle Topology button, you can see the attack overlaid with other devices near the incident, as illustrated in Figure 6-9.

Figure 6-8

Physical View Shown on Incident Graph

Figure 6-9

Incident Graph After Toggling Topology

The third way to graphically view the incident is to click the Path/Mitigate button near the right edge on each line of the Incident ID table, as illustrated in Figure 6-4.

Clicking this icon causes MARS to perform some behind-the-scenes investigation. If possible, MARS determines which monitored devices can block the attack in the future. For example:

  • A firewall might restrict access from the attacker to one or more hosts or networks.

  • A router might have an access control list applied to restrict access.

  • A switch can have a port disabled.

If the attacking host is internal to your network, and is connected to one of the switches that MARS is monitoring, MARS attempts to determine the attacker's MAC address. MARS regularly queries switches and routers on your network to maintain this information, which exists in the content addressable memory (CAM) and Address Resolution Protocol (ARP) tables in your network devices. When MARS has this information, it can locate the host and the switch port to which the host is connected.

After MARS has evaluated the Layer 2 and Layer 3 paths involved in this attack, it presents options for mitigation. You should understand that MARS does not automatically mitigate an attack. Instead, it provides information to you. Figures 6-10 and 6-11 show the mitigation screen presented in this attack. MARS has determined that the best way to stop the attack, or prevent it from recurring, is to create an access control list entry on the PIX firewall.

Figure 6-10

Session Graph and Mitigation Options

Figure 6-11

Session Graph and Mitigation Options, Continued

Occasionally, you might also be presented with the option to disable the switch port that the attacker is connected to. In fact, you can click a red Push button to instantly disable the port.

1 2 Page 1
Page 1 of 2
SD-WAN buyers guide: Key questions to ask vendors (and yourself)