Be Prepared for Incident Response & Forensics

By Andrew Plato, CISSP, CISM, QSA
President / Principal Consultant
Anitian Enterprise Security

It is a call I have taken many times: concerned executives, hushed voices, uncertainty, questions about privacy and discretion. There has been an incident. What to do? What’s the next step? Who do we call? And invariably the question nobody really wants to ask -- how much will this all cost?

I have witnessed, responded to, and participated in a wide range of security incidents. Mostly, I have watched people miss the whole purpose behind incident response. Incident response is about investigation and facts. And consequently, drawing conclusions from the facts and the facts alone. Do not become lost inside the drama of the event.

Suspicion, Speculation & Paranoia

These human factors can absolutely devastate the integrity of an investigation. People often jump to conclusions without a full understanding of the facts. And without facts, it's all too easy to twist theories and suspicions to suit the perceptions of those involved.

When (not "if") a security incident happens, as enticing and dramatic as it may seem, you must force yourself and your organization to abstain from speculative reasoning in the early stages. People can convince themselves, and others that they understand the situation based solely on their perceptions, but perceptions are notoriously unreliable, and subject to the expectations of the perceiver and the perceived.

Incident response must be under the guidance of a trained investigator who can rationally distinguish between human speculations and observable facts. Before you start imaging drives, turning off systems, or spreading panic about an incident -- identify an individual or organization who can conduct an investigation in a rational manner. It must be a person or group that can place facts above speculation or agenda.

Elementary, My Dear Watson

Investigations are not flashy, technical endeavors. A solid investigation begins and ends at the hands of a skilled investigator or team of investigators. While there are important technologies that can help speed the process of an investigation, the success of any incident response hinges on the analytical capabilities of the investigators.

There is a temptation to rely heavily on "e-discovery" tools. While such tools are important, and can greatly aid in gathering and corroborating evidence as well as reducing the cost of an investigation, those tools and their data are most efficiently used in the hands of a skilled analyst. Tools cannot determine intention, context and motive -- they simply produce data. A tool cannot draw a conclusion. Only a skilled investigator can interpret that data and make sense of it, considering not only the data, but the inputs that produced the data. An inexperienced investigator may input the wrong criteria, or allow a bias to influence the input criteria, thus leading to an invalid conclusion.

Inception

The first step in any investigation is the inception meeting or discussion, where an organization must make critical decisions that will affect the entire investigation.

The most important initial question in any incident response is: "What are your intentions?" It is easy to get lost in the details and excitement of an incident, particularly one that involves the separation of an employee, and where legal issues may be involved. Clarifying and understanding everybody's intentions in the investigation helps focus the effort. The investigator must understand how far you want to go. Criminal and civil litigation may seem reasonable at first, but less so when the expense of preparing a case is considered. Do you intend to prosecute? Do you intend to involve legal counsel? Are breach notification laws possibly applicable in this case? You must have a clear intention as to what you want the investigation to accomplish, and you must have the resources to do it.

Who to Call?

It is perfectly legitimate to keep an investigation internal and not involve law enforcement or legal counsel, but this decision must be made with a full appreciation of the ramifications of doing so. While a good security investigator understands the relevant laws and legal issues, most security analysts are not lawyers, and cannot provide legal advice. Only legal counsel can provide such advice.

Do not be surprised if local, state or federal law enforcement brushes you off or does not respond immediately to requests. Employees misusing assets or stealing data may seem like a serious crime, but in the eyes of the police, it may be viewed as a minor, non-violent crime. Law enforcement has limited resources and must triage their efforts on those investigations that present the largest danger or threat to people.

Even in a state with disclosure laws, it may be wise to not disclose any information to any third party until you have conducted a thorough investigation. An investigation may determine that there was no disclosure, in which case, any public disclosure would only cause unnecessary harm to your business.

As a general rule of thumb, refrain from contacting any external resources (except legal counsel) until the facts of the case are well documented.

Evidence Capture

The next key phase in an investigation is the gathering of evidence. This typically includes gathering forensic images of drives and other data.

Do not become consumed with the mechanical details of evidence gathering. Imaging drives, for example, is relatively easy. There are plenty of tools and trained handlers to do this.

There are, however, some key procedural aspects to evidence capture, which a skilled investigator will know. Chain of custody and maintaining evidence integrity ultimately depend on the investigator following widely accepted and repeatable practices.

Despite what some books and conventional wisdom might suggest, there is no "official" way to capture evidence. There are many widely accepted best practices, and a trained investigator will know these and be able to describe them. Be wary of amateur investigators who overcompensate with seemingly rigid or sensational response techniques.

One best practice that is widely known is the use of "golden" images. When evidence is captured, investigators will never conduct their analysis on the original image. They will make working copies of a "golden" image that is never directly altered. This allows another party, such as opposing counsel, to independently validate the integrity of the originally captured image. Some investigators will even go so far as to use a third party imaging company, thus eliminating any chance that the disk images could be altered.

Corroboration

Part of any investigation is to corroborate the data, placing it in context with the time of other pieces of information. For example, if a computer hard disk shows that a user downloaded a worm on 7.16.2009 at 11:45AM, then firewall, system, and network logs should corroborate this access.

Data, data, data -- all investigations demand data. Without data, specifically contextual data, it is very difficult to state, with certainty, what happened at any given moment. Lack of corroborating evidence can damage or destroy the integrity of an investigation. If an investigator cannot link events together with multiple points of data, then conclusions drawn from the data will be less reliable.

This is an area where companies can prepare. It is much easier, quicker and more reliable to investigate an incident when the data to corroborate that event is easily and reliably accessible. Security Information Management (SIM) technologies are particularly valuable in incident investigations, but must be configured correctly to gather all the relevant data and store it in a manner where it can be accessed with some degree of ease.

Intrusion prevention/detection systems are also very valuable, as can be Data Loss Prevention technologies. When building a security program, make implementation of IDS/IPS, DLP and SIM a priority.

Analysis

Analysis is where an investigation transforms from theory and speculation to facts. Working from copies of the "golden" images, analysts review the data. This is the single most important part of any incident response, and the success of the response ultimately hinges on the investigator's ability to accurately analyze the data and draw reasonable conclusions.

This is an area where you cannot discount experience. Amateur investigators often attempt to overcompensate for their lack of analytical ability through a variety of means. One common tactic is to push the analysis parameters back on the client, making the client define the set of search parameters and then merely conducting pattern searches for that data using investigation tools. This is not analysis. Anybody can feed criteria into a search engine and have it produce results. Only a trained analyst can put that data in context.

A skilled investigator can build a timeline of events using multiple, corroborating pieces of information. Software can't do this. It may provide some of the data the investigator uses, but only a person trained to analyze complex information systems has the ability to lay out an event on a timeline and derive conclusions from that data.

A good investigator must also understand the information systems being analyzed. There are a myriad of file systems, operating systems, applications, network connections, and so forth. It's easy to lose perspective and get caught up in the drama, but catching "bad guys" requires experience and knowledge of information systems. This is not a skill that can be picked up on a whim. Good investigators have 10 or more years of experience as system and network administrators, they know how an IT department is run, and they have a good understanding of IT operations.

Reporting & Briefing

The last part of any incident is the report and briefing. A good incident response report should include the following.

• Executive Summary: A summary of the incident, findings and conclusions.
• Incident Details: A detailed description of the events leading up to and during the incident. This may include the incident timeline as well.
• Findings: An objective presentation of what the analyst found. This should be presented in a matter of fact manner, with limited editorializing on the part of the analyst.
• Conclusions / Recommendations: Based solely on the findings (facts) the investigator should draw conclusions about the incident and offer suggestions for the correct ways to respond, repair any damage, and prevent future incidents.

It is important that the final report have recommendations, and that management carefully consider those recommendations. A competent and experienced investigator offers realistic and reasonable suggestions to remedy future incidents.

Incident Response Plan

Before there is an incident, plan for one. An incident response plan defines how an organization will identify, categorize, and respond to security incidents -- including those that would prompt an investigation. Also, such a plan should clearly define roles and responsibilities within the organization, to avoid confusion in the event of a serious security incident.

An incident response plan should also be closely aligned with a disaster recovery plan. The plan should include not just how to identify an incident, but also how any damages will be repaired.

Conclusion

Incidents happen and they can be very valuable experiences. With solid planning and a rational, scientific approach, you can ensure that your organization responds correctly and reasonably to an incident.

Whatever you do, do not fall victim to the sensational and dramatic elements of incident response. Many amateur security people become enamored with the excitement of investigations. Incident response and forensics is not an episode of CSI. It is a serious endeavor that is best performed by skilled, experienced investigators using rational methods to derive reasonable conclusions.

For more information, please call 888.ANITIAN, or email Anitian Enterprise Security.