Vulnerability Management for the Win

Attackers exploit weakness.  This age-old concept applies equally to castles and battleships, as it does to applications and networks.  The more vulnerabilities a network has, the easier it is to attack and compromise.  Nearly all of the recent high-profile attacks (like Target and Home Depot) targeted well-known vulnerabilities. As such, finding and fixing vulnerabilities is extremely important to protecting an organization’s digital assets.

Vulnerability management (VM) is the process of identifying, cataloging and fixing vulnerabilities.  There are plenty plans and technologies designed to make VM easier.  However, when we dig into the details of VM, it is not process or technology that is the weakest link.

Like so many other IT issues, people are the most daunting challenge in building a solid VM program.  Many organizations begin their VM efforts with a scanner and stacks of vulnerability scans.  Unfortunately, this typically devolves into frustration and finger-pointing.  Successful VM begins long before any scans are run.  Good VM comes from a holistic, systemic approach that involves the entire IT organization.

Our intention in this article is to define a set of concepts and practices you can embrace to make your VM program work.

Risky Business

The first step to building a VM program (which is the first step for most security efforts) is to understand the level of risk present in the organization.  Without some form of risk assessment, all your security efforts, particularly VM will at a minimum be unfocused and at worst be a waste of time.

In order for the risk assessment to be useful to a VM effort, it must be technical.  That means it cannot be a bunch of checked boxes on a compliance portal or some surveys that people filled out. Anitian has written extensively on this matter here.

Ultimately, any risk assessment that will aid in developing a VM program must meet the following criteria:

• Be completed in less than 30 days so the data is fresh and relevant
• Make sense to leadership and decision-makers, so they accept it and support the recommendations
• Provide a prioritized list of threats, based on risk rankings
• Include a comprehensive inventory of relevant assets
• Establish risk tolerances from executive leadership
• Include technical scans and tests that support risk analysis and decisions
• Focus on the probable and likely, not the sensational and fascinating
• Organize systems, applications and/or data into logical risk categories

Traditional risk assessments using NIST, FAIR, or OCTAVE rarely meet these needs.  They are too complex, too long-winded, and too focused on paperwork and process. This is a topic Anitian has presented on before.

The purpose of the risk assessment is to understand the threats that are relevant to the organization and the level of risk those threats pose based on the vulnerabilities and controls in place.  As such, a risk assessment facilitates the prioritization of vulnerabilities in high risk areas over those in low risk areas.  It also keeps the VM program grounded in the management of risk, rather than the chasing of phantoms and buying technologies.

Furthermore, the assessment needs to come from an unbiased source. Avoid using technology vendors or resellers as they are likely to skew the results to encourage buying more technologies from them.

We are All in this Together

VM is a team sport.  The entire IT department, as well as other groups with a strong stake in IT, must be involved in eliminating vulnerabilities.  If the IT staff in your organization are intractable and argumentative about patching or hardening systems, then you have larger organizational problems.

Cultural and organizational issues are the most serious vulnerability any enterprise face.  You must resolve this issue first if you ever expect to have a successful VM program.

How do you tackle this issue? There are no easy answers, however the place to begin is with leadership.  If executive leadership allows people to make excuses, rather than do their jobs, then there is a limit to what any technology or process can do.

If you are tasked with setting up a VM program, you need to ask yourself a very serious question: are people in this IT department going to patch systems or reconfigure them when the time comes, or are they going to argue, make excuses, complain, and try to avoid doing work?

If whining and complaining is the norm in your organization, then you must get your leadership to openly endorse the VM effort.  Ideally, you want to have your management communicate the following ideas to the entire organization:

• Protecting the business is everybody’s job
• Vulnerabilities jeopardize the business, its mission, and jobs
• Eliminating vulnerabilities is a priority and must be completed in a reasonable timeframe (like 30 days)
• Come to the table with solutions, not excuses

If you read that list and laughed thinking “yeah, right, that will never happen” then your VM efforts are probably going to fail. You might want to consider your next steps, as well as career choices, carefully.

However, if leadership can make these statements to the staff and make them stick, then you may be able to overcome these cultural problems.  It is very important that you address these cultural issues first.  Ignoring them will only lead to massive frustration in the future.

Security Fundamentalism

The next component that any VM program must have is the controls to actually manage and mitigate vulnerability.  It is pointless to be scanning an environment and trying to manage weaknesses when you have minimal (or no) ability to fix the vulnerabilities you find.  As such, your organization must first master the following security fundamentals:

• Patch management
• Seriously, we mean real patch management
• Did we say patching, yes we really mean patching, with no excuses
• Intrusion prevention, not detection, actual in-line, real-time prevention
• Endpoint anti-virus
• Configuration management
• Change management / detection
• System hardening standards and standardized builds
• Data, system, and application risk classification (which should have come from your risk assessment)
• Patch management, yes we are saying it again because we are really serious about it!

Those are the must-haves to make a VM program work.  Obviously, there are plenty of other controls that any organization should have, like email security and identity management, however those are only tangentially beneficial to a VM program.

In a Scanner, Darkly

Once you have a solid risk assessment and security fundamentals in place, scanning the environment for vulnerabilities is next.  Of course, you need a vulnerability scanner to do this.

The market is full of decent vulnerability scanners. Rapid7, Qualys, Nessus, etc. are all solid, reliable solutions.  The differences between these products is rather minor.  What is important is that you implement a product you can use effectively.

The scanner must be able to scan your entire network (or a large representational selection of your environment.)  Make sure you deploy enough scanners to cover the entire environment.  If you have a large network, disseminate scanners across the network to balance load.

In the grand scheme of VM, setting up a scanner and running scans is the easy part.  Just make sure you implement these features.

• Use authentication where possible. This provides more accurate results.
• Run discovery scans to locate new devices. This is especially useful for detecting rogue wireless access points.
• Update the scanner. Scanners need new checks and updates like any other device.
• Run scans at regular intervals (like weekly). Scanning must be comprehensive and routine in order for the data to be meaningful.
• White list the scanner. Your internal scanner should be able to “see through” any active defenses you have like an IPS (or NGFW/UTM). If you want to test the effectiveness of your IPS, do that separately and not part of the regular VM program.

Big Data in Little Scanner

Now that you have run some scans and have some data, the next step is to make sense of that data.  Vulnerability data can be overwhelming.  Rather than address each vulnerability individually, you should assess the entire data set systemically. In other words, ask yourself what story does this data tell?

A well patched, and well managed environment should have mostly low-risk, informational type vulnerabilities (which can usually be ignored).  A poorly patched, poorly managed environment will have lots of vulnerabilities.  Anything more than two or three high-risk vulnerabilities per system is a sign of a poorly managed environment. A few outliers here and there are normal.

Look for trends, patterns, and clusters of problems.  Are all your databases showing a ton of vulnerabilities?  Then the logical conclusion is that you have issues with your database management or managers (probably both). Is every workstation in the finance department running telnet? Why? What does that mean? Data tells a story. You have to find that story in the data.

It is very easy to become bogged down in the nitty gritty of VM.  Many VM companies frankly want that result because then they can sell you VM platforms with charts, and graphs, and all matter of pomp and circumstance.

Origin Story

With great data, comes great headaches. The way to slay the big data monster is to go after its origins.  In other words, you need get to the root causes of them vulnerabilities rather than trying to fix each one individually. In almost every case, the root cause is a lack of patching.  Patch management is the most effective way to remediate vulnerabilities.

If you absolutely cannot patch systems, then you must have some other control method to stop attacks against those vulnerabilities. In other words, you need an intrusion prevention system (not an intrusion detection system).  That means real-time, in-line, automated blocking of attacks against known vulnerabilities in the environment.  (Frankly, you should have a solid IPS regardless of your vulnerability situation.)

Configuration management and system hardening should also address numerous vulnerabilities as well. This is why your organization needs those controls in place before you execute vulnerability scans.

The asset categorization and risk tolerances from your risk assessment should serve as a guide at this point.  Prioritize patching and hardening on the most valuable systems (apps, networks, etc.)

Once you have patched systems, reconfigured them, and hardened them from attack, rerun the scans and watch the vulnerabilities drop from the list. Most scanning platforms have reports that can track vulnerabilities over time.  Naturally, you want the total number of higher risk vulnerabilities to decrease with time.

Manage the Outliers

Even after attacking the root causes of vulnerabilities, there may still be some remaining issues.  However, if you have done everything else, this list should be relatively small. The whole point of using a systemic approach to VM is to whittle the outliers down to a manageable number.  If you are individually addressing hundreds of vulnerabilities, then you are not taking a systemic approach.  Return to the big issues, like patching and configuration management.

Final Thoughts

At this point, we have outlined a basic process for establishing a VM program. There are a few other best practices worth mentioning.

• Do not let perfection become the enemy of good. This is good advice for any IT effort. Whittle down the vulnerability list over time.  It is unrealistic to think you can eliminate all vulnerabilities in a matter of a few weeks or months.
• Scan before you buy. Scan new software products before you purchase them to get insight into the vulnerabilities they may introduce into the environment.
• Nuke Java from orbit, it is the only way to be sure. Java is essentially a hacking kit at this point. Avoid Java whenever possible. It has a horrible track record as an application environment and it never seems to improve.
• Scan the cloud. Just because an application is deployed into the cloud does not mean it magically is immune from attack. You still need to scan cloud apps. Do not rely on scans from the cloud vendor as they are unlikely to scan fairly.
• Scan your own stuff. While you should scan your own apps in the cloud, you should not scan that which you do not own.
• No black box hosting environments. If cloud or hosting providers will not allow you to scan your environment, get a new hosting provider.
• Outsource it. VM is one of the more difficult security functions to outsource.  While you can outsource the scanning of systems, fixing them is an internal effort.  We recommend avoiding outsourcing VM.  Build a program internally.
• No scanner is perfect. Your scanner will drive you insane and fail to do a lot of things you want. The platform you have, that works, is infinitely better than the promises of a vendor to make it all better.
• Worried about the scanner taking down devices? If scans crash systems in your environment, then you have some problem systems, not a bad scanner. Regular vulnerability scans should never cause a host to crash or become unresponsive. Do not blame the scanner, blame the poorly configured host.
• Maintain independence. Avoid having the VAR that sells you gear define your VM program. Their advice is biased.
• GRC tools can be very helpful. Many of the governance, risk and compliance (GRC) tools on the market, like MetricStream, Archer, Lockpath, Allgress, etc. have excellent vulnerability management tools.  These products can ingest scan data and provide tools for assignment and management of vulnerabilities.  These tools can help a lot.  They can also provide additional tools for managing compliance data as well. While these tools are nice to have, they often require a lot of care and feeding.
• SIEM integration: Many Security Information and Event Management (SIEM) products also allow for importing vulnerability data to correlate log data with known vulnerabilities.  This can also be very valuable, particularly the reporting features. However, in the grand scheme of VM, SIEM integration is more of a nice-to-have, rather than a must-have.
• Make the pentester work for it: Ideally, penetration testing should merely validate that your VM program is working correctly. If your pentester is finding a lot of older, high-risk vulnerabilities in your environment, then whatever VM program you have is not working.

Conclusion

As mentioned in the beginning of this article, VM can be a very overwhelming task.  However, data alone is not the most problematic component.  Organizational resistance to change is a more prevalent problem that mires VM efforts.  Scanning hosts and identifying weaknesses is, frankly, the easy part of VM.  As with many technology issues, it is managing people that is the difficult part.

Before you charge off into building a VM program, make sure you address the people issues.  It does not matter how great your scanner is, if the IT people in your company will not fix the vulnerabilities your scanner discovers, then your VM efforts will never succeed.

Note: Robert Cooper, James Beukelman, Greg Ragland, and Chris Quevedo at Anitian all contributed to this blog entry. Thanks guys!