Understanding the Market for Penetration Testing

By Andrew Plato, CISSP, CISM, QSA
President / Principal Consultant
Anitian Enterprise Security

One of the more frustrating aspects of a tightening economy is companies cutting corners on goods and services. Recently I took my car in for service and noticed how the dealership no longer gave out free car washes and chocolate bars as part of the service. Belt tightening is prevalent in technology departments as well: positions remain unfilled, capital projects are slashed and security vulnerabilities go uncorrected.

It comes as no surprise then that security services are under increasing scrutiny to offer more for less. Companies looking to contract security services need to be careful when hiring analysts to conduct security and vulnerability tests. There are some vendors offering “penetration testing services” that are not true penetration tests. Many of these low-cost tests are just simple scans with very little insight into the real risks that an organization faces.

To help make sense of this market, this article compares the differences between vulnerability scanning, penetration testing and application testing. It also defines some key elements you should look for when contracting with a firm to conduct a penetration test.

Vulnerability Scanning

Vulnerability scanning is predominantly an identification process. Scanning involves conducting automated scans of systems to locate security weaknesses. A scanning tool, such as Nessus, Qualys, or Rapid7, is used to conduct the scans and report vulnerabilities. Scans are entirely or partially automated, with little to no direct involvement from a security analyst.

The goals of a scanning project are:

• Identify hosts and fingerprint their operating system.
• Locate and identify services and/or open ports on responsive hosts.
• Identify, catalog and qualify security vulnerabilities.
• Assess general system and infrastructure risk.

Security scanning focuses on identifying weaknesses. Scanners attempt to connect to a host, read information from the system, and then make a determination as to what vulnerabilities exist on the system. This is a very valuable service. It can help you determine the general state of security on individual systems.

It is important to realize there are some notable constraints to security scanning.

• Scanning does not analyze the overall security of your infrastructure or organization. You can scan a network a thousand times a day and it will not tell you if the organization operates securely or is compliant with anything. Scanning gives you a glimpse into the general level of security and maintenance of hosts in the environment.
• Scans can often incorrectly identify vulnerabilities. While scanning technology has improved a lot in the past years, scanners still make mistakes. Many scanners identify vulnerabilities based solely on the version of software running. That is not always accurate.
• Scans can inflate the risk or existence of vulnerabilities. Scanners by and large lack the ability to consider the bigger picture of your network and systems. This can lead to vulnerabilities being reported as more (or less) serious than they actually are. Likewise, a scan may also identify a vulnerability that exists, but could not be reasonably exploited, therefore inflating its seriousness.
• Scans cannot detect fundamental weaknesses in web applications. Most scanners lack the ability to validate application constraints and functions. For example, if you have a web application that accepts an input value (like an address) most scanners cannot analyze that field and determine if the input for that field is constrained properly.
• Not all scanners are alike. There is a wide variety of commercial, open source and low-cost scanning software on the market. These products range from very good to very poor. The less expensive (or free) scanners are usually limited to a small subset of vulnerabilities they can detect and identify. Furthermore, low cost scanning providers are unlikely to spend the effort to optimize their scanners to provide comprehensive coverage of the ever-growing universe of vulnerabilities.

Vulnerability scanning has a place in your security program. However, that place needs to be properly adjusted to the requirements and constraints of scanning.

Penetration Testing / Application Testing

Penetration and application testing is predominantly a vulnerability exploitation process. Where as scanning merely identifies and possibly confirms the existence of vulnerabilities, penetration testing attempts to actually exploit vulnerabilities and gain access to a system. Furthermore, penetration testing conducts a more exhaustive set of tests, manually examining application inputs, functions and controls. Penetration testing is typically directed toward publically available applications or networks, such as web sites. Application testing is a more generic variant of penetration testing that focuses on specific applications which may or may not be publically available. Application tests may also be directed toward applications that are not web-based, such as a client-server type application. For the purposes of this article, penetration and application testing will be used interchangeably.

Vulnerability scanning is often a part of penetration testing. A penetration tester will conduct a series of vulnerability scans first to determine what well-know weaknesses exist. That information is used to direct the tester’s efforts toward the vulnerabilities and exploits that represent the largest risk to the organization.

The goals of a penetration or application testing are somewhat different from scanning:

• Identify insecure or unauthorized vectors of access to a network or application.
• Analyze the threat those vectors present.
• Attempt to exploit those vectors and confirm their existence.
• Analyze the overall security of an application and/or network to determine the risk present and likelihood of exploitation.

Penetration testing focuses on actually trying to exploit vulnerabilities. Running scans and printing our reports is, frankly, very easy to do. Actually validating those vulnerabilities and attempting to exploit them is much more difficult.

A comprehensive penetration or application test should look at the following vulnerabilities and issues:

• Un-validated sources of input and use of that input
• Un-validated output streams
• Flawed authorization, access control
• Flaws in session management
• Injection flaws, including SQL injection
• Cross-site scripting (XSS)
• Cross-site request forgery (CSRF)
• Information disclosure in source code comments
• Insecure direct object reference
• Weak encryption
• Application configuration flaws
• Denial of service vulnerabilities
• Infrastructure weaknesses or vulnerabilities
• Unsupported application interfaces
• Improper administrative and exception handling

Types of Tests

There are a set of common test types that a penetration tester can perform. These are often defined as:

• Black Box Testing
  Also called anonymous testing. The tester will attempt to break into the network/applications without any credentials or information about the environment.
• White Box Testing
  Armed with legitimate user credentials on exposed applications, the tester will attempt to use those credentials to escalate privileges and carry out more sophisticated attacks.
• Application Stress Testing
  Attempting to overflow or “stress” the application and/or hosting environment to reveal information or stop functioning
• Configuration & Code Review
  Analyze the configuration data and/or source code of the application(s), middleware and database for possible weaknesses.
• Environment Analysis
  Analyze the development and hosting environment of the application for security weaknesses or concerns.

What to Look for In a Penetration or Application Tester

Naturally, the market for penetration testing is as diverse as any. There are some key qualities of a good penetration testing group that you should consider.

• Experience
  Penetration testing is complex, technically rigorous work. It is important that the company and people doing the work have a proven background in security testing.
• Attitude / Personality
  Skilled security testers are not like the hackers seen on television and in movies. People who are enamored with the “hacker lifestyle” are usually a poor choice for penetration testing work for many reasons. “Hacker” types often use the “lifestyle” to hide their lack skill and reliable methods. Good penetration testers have formalized methods, respect business needs and view testing as a logical, analytical process.
• Tools
  Penetration testing requires some very unique tools. Good testers use a wide array of open source, commercial and custom made tools. If your provider relies on a single tool, that could be a sign that they do not understand how to conduct a comprehensive assessment.
• Application Layer Knowledge
  A quality penetration test is customized to the specific applications being tested. If one of your potential penetration testing vendors is not asking about the application environment, they probably are not going to conduct a comprehensive penetration test.
• Reports
  Always ask for sample reports. Good companies have established and detailed reporting capabilities. Pay close attention to the level of customization in the sample report. Sample reports that looked like canned reports from scanning products are a good sign that the company does not have the skill to conduct a comprehensive penetration test.
• Risk-Based Analysis
  Many inexperienced penetration testers practice “gotcha” testing. They focus intensely on exploiting vulnerabilities and not on whether those vulnerabilities pose a real threat to the company. A skilled tester not only can break in, but also can qualify that break in with solid risk analysis.
• Intensity of Effort
  Penetration tests are inherently constrained by time and financial resources. Theoretically, a dedicated hacker has all the time in the world to try and break into your network and applications. For a contracted test, such carte blanche is not feasible. Therefore, it is important to understand the “level of intensity” your penetration tester plans to invest in the project. There are three basic levels of testing intensity:
  • Level 1 "Script Kiddie"
    This penetration test would cover about 50-75% of the exploits in the wild. Testing would include vulnerability scanning and basic black-box application testing. This test would simulate what a novice to intermediately skilled hacker (top 50%) could detect and exploit. A solid, low-cost test that would likely miss a lot of the more sophisticate attack vectors.
  • Level 2 "Skilled Hacker"
    This penetration test would cover about 90-95% of the exploits in the wild. Testing would include vulnerability scanning as well as extensive black and white box application testing. The testing will validate all vulnerabilities discovered and conduct manual validation of application inputs. This test would simulate what an experienced and skilled hacker (top 10%) could detect and exploit.
  • Level 3 "Elite Hacker"
    A massive, nothing held back attack. The tester will use every resource possible to break into the network. This will include every possible test and attack that can be devised, including building custom attacks specifically targeted at the organization. This testing would cover 99.5% of the attacks in the wild (100% is impossible, since there are always some methods that are unknown.) This test would simulate what a truly elite hacker (top 1%) would attempt. These tests can be very expensive and take a long time to complete.

Why Do I Need a Penetration Test

There are many reasons to conduct a penetration test:

• Compliance
  Some security standards, like PCI, require a year penetration test.
• Diligence
  It is always best to “bake” secure coding practices into your software development life-cycle (SDLC). A penetration test can ensure your team followed good practices and wrote secure code.
• Measure Effectiveness
  If you have invested in security technologies and improvements, a penetration test can help determine the effectiveness of that investment.
• Insight
  An independent, third-party assessment can qualify and quantify risk. This can help management make informed decisions about the level of risk acceptable for the organization.

Conclusion

With shrinking budgets and limited resources, it is always tempting to take short cuts to compliance and security. While there are many cost-effective ways to reduce risk and improve the overall security of your organization, hiring fly-by-night penetration testers is rarely a good choice. Inexperienced testers and ramshackle methods lead to meaningless results and wasted efforts.

Moreover, it is important to know what type of security testing you require. A vulnerability scan is definitely not a replacement for a penetration test.

For more information, please call 888.ANITIAN, or email Anitian Enterprise Security.