The curriculum of business schools are filled with case studies of companies that took short cuts to become competitive and accomplished the exact opposite. For the information security world, there is a similar “penny-wise, pound-foolish” behavior in the notorious “Checkbox Assessment.” These are security or compliance assessments that have little basis in reality and are merely done to check off requirements and “get them out of the way.” This is a profoundly dangerous and deceptive approach to information security, yet it is extremely common.
Why is something so dangerous, with ample examples of that danger, so common? The answer, like so many business problems is a toxic stew of deception, ignorance, and greed.
Regardless the reason, security practitioners have a responsibility to make the security or compliance assessment process have value and meaning to the company. However, this can be very difficult if a bad assessor is entrenched in your organization and has deep relationships with leadership?
As such, the intent of this article is to help educate security professionals on how to get the most out of their security assessments, and if necessary gracefully remove an assessor who is not providing value to the business.
How to Identify a Good Security Assessor
The first step, in this process is to establish criteria for what constitutes a good security assessor. The contrast between what you want and what you are currently getting can be emphasized to identify gaps in quality.
There are plenty of good security assessors in the business. Of course, you always want to evaluate an assessor for the basic qualities such experience, reputation, and approach. Those are important for any vendor relationship. However, for security assessors, there are some other unique attributes that you should also consider.
Hands-on Technology Skills
A skilled information security assessor should have a background in systems or network administration. They must understand the “nuts and bolts” of information technology such as firewalls, servers, routing protocols, encryption, virtualization, and intrusion prevention systems. This knowledge is absolutely critical to making reasonable recommendations for improvements.
In contrast, bad assessors lack hands-on IT skills and they will often attempt to compensate or hide this inexperience with an aggressive attitude or dismissal.
Ability to Embrace Complexity and Craft a Balanced Response
Experienced security professionals respect the inherent complexity of protecting information systems. They also can communicate that complexity and will balance reflective planning and strategy with decisive action.
Bad assessors struggle with complexity and will either dismiss it or exacerbate it. Subsequently, they will either react without planning (overbuying technologies), or become mired in analysis (analysis paralysis).
Fear is a powerful motivator for people. However, it can cloud judgment and lead to short-term, reactionary decision making. Good assessors emphatically reject fear as a persuasive tactic and embrace thoughtful, contemplative, strategic thinking that emphasizes positives rather than just stopping negatives.
Bad assessors are obsessed with criminal activity and sensationalist hacks because they are excellent distractions that lead to bad decision making (such as hiring bad assessors.)
Good assessors get to the point, quickly. They reject unnecessary formality when it serves no purpose. Reports from an accomplished assessor are clear, concise, and draw conclusions from evidence.
Bad assessors love paperwork and flowery, emotional language. Portals full of data are like catnip for them. They cannot get enough of the heatmaps, data flows, policy documents and other detritus that nobody will ever read. Worse, they will contrive conclusions to suit an agenda, which is too often to sell additional technologies or services.
Good assessors see themselves as an integral part of the organization’s security efforts and take that responsibility seriously. They are able to challenge people with probing questions that respectfully reveal how security controls are used and policies are implemented.
Bad assessors will push their job off on other people, with mountains of “checklists” other busy work for IT staff.
Good assessors do more than just tell you they are independent, they openly disclose their bias and intention. They also can fairly address alternative points of view or strategies without unqualified dismissal. They will encourage independent review of their work and are open to discussing it. They will defend their work with evidence and logical reasoning.
Bad assessors hide their intent and act as if they have no bias. They will discourage independent review and will defend their work with emotional threats and attacks.
Good assessors see regulations and standards as “baselines” that are open to intelligent interpretation and discussion. A good assessor does not just know a security standard, like HIPAA or PCI, but understands the intent of those standards and how they relate to the overall context of an information security program. Many good assessors will downplay the actual requirements preferring to focus on protecting the business and reducing risk first and then subsequently aligning those protections with relevant standards.
Bad assessors obsess over standards and view them as immutable and irrefutable. They will speak in absolutes and assume an “enforcement” approach to compliance.
Fascination with People
People are the single most important part of a security program. Good assessors are both technologists and students of human nature. They have developed an intuitive sense of how people, processes, and technology work within organization. They also listen to people carefully and know how respectfully persuade people to work collaboratively.
Bad assessors are self-absorbed and see themselves as “above” the organization. They feel it is their job to “enforce” security and have minimal regard for how those controls affect people, performance, or profitability.
Confidence without Arrogance
Lastly, good assessors are confident, accomplished professionals with a passion for service. They remain consistently focused on the client’s needs. They are demure and respectful of their client’s confidentiality. Moreover, they respect that people are always learning and need encouragement and inspiration to follow good security practices.
Bad assessors are arrogant and selfish. They may speak disrespectfully to people who are not security professionals. They will often brag about their skills and experiences to impress others. Many bad assessors will act indignant and flippant when people make mistakes.
How Bad Assessments Endanger an Organization
It is easy to point out the benefits of a good security assessment, but what about the dangers of a bad one? If you are trapped in a relationship with a weak assessor, the entire organization can suffer. Bad assessors produce bad assessments and that has bad consequences. Being able to explain the dangers of a bad assessment to management is difficult, but there are some key ideas you can work into conversations.
The most significant danger of a bad assessment is that they convince management of security state that does not exist. Bad assessments fail to address the real issues. They will either trade comfort and financial savings for actual measurable security improvements or hype up sensationalist issues that distract from reality.
Demand evidence of improvements. A good assessor can explain their conclusions in the context of threats and risk, and not merely what a compliance regulation requires. Ask assessors “why?” It is a powerful way to cut through shoddy work and uncover the truth.
Misdirected Investments & Waste
Bad assessors often lack sound IT skills, therefore the only way they know how to meet compliance requirements is to spend a lot of money on technologies specifically designed to meet compliance objectives. That tautology keeps the Sisyphean cycle of IT spending going and ensures ample commissions and kickbacks from technology vendors. It is a wasteful process that does little to improve security and merely creates more overhead of managing security controls.
Focus on shifting the conversations to value and capabilities. New technology investments should align with an overall IT strategy, not merely check off compliance requirements.
Breaches / Attacks
With misdirected and wasteful efforts, bad assessors setup their clients for eventual breach or attack. When this happens, the finger-pointing will be fast and furious. Those bad assessors will be quick to blame everybody but themselves and their bad advice.
While no assessment can insure against an attack, bad assessments may actual be precipitating it. Ignoring fundamental controls in favor of “high-effort” sensationalist technologies exposes the organization to more risk. Moreover, if compliance efforts do not improve security across the entire enterprise, then the chances of a attack, which could compromise protected data, will remain high.
Building a Sustainable Assessment Process
To ensure you have the best security assessors, you need to build a sustainable process to conduct those assessments and edge out the bad vendors. In addition to engaging skilled assessors, there are some key strategic efforts you can make that will help ensure a more meaningful assessment process.
Diversify & Rotate Vendors
Establish a company policy to use a pool of security vendors and rotate them regularly (annual is best.) Different assessors will have different perspectives and that helps encourage a more complete view of your security posture. Rotating vendors also will inherently avoid complacency with any one vendor.
It is important to keep bringing in new vendors and trying them out. Penetration testing and policy review work is an ideal way to assess the skill set of a new vendor.
Also, divvy up work across expertise. If you have a great penetration testing vendor, then keep them focused on that. Get a different partner to audit and test their work.
Lastly, be very transparent and honest with your vendors that you use multiple sources. Play fair and give them an opportunity to shine. Lastly, evaluate vendors on their value to the organization, rather than the quality of their paid lunches.
Promote Sharing & Build Trust
If you want results from an assessment, you must freely share information. However, that means you must trust them. Building trust means following trustworthy behaviors. In his book, Speed of Trust, Stephen MR Covey outlines 13 behaviors that define trust:
1. Talk Straight
2. Demonstrate Respect
3. Create Transparency
4. Right Wrongs
5. Show Loyalty
6. Deliver Results
7. Get Better
8. Confront Reality
9. Clarify Expectation
10. Practice Accountability
11. Listen First
12. Keep Commitments
13. Extend Trust
These are all behaviors you should see in your assessor, specifically, Listen First and Deliver Results. A good assessor is going to listen to you, not merely lecture about threats and problems. They are also able to deliver results that focus on helping you, rather than promoting their skills.
Require Proof & Context
Require hard evidence to support assessment conclusions. Vulnerability scans, penetration tests and direct observations are all evidence that can support a conclusion. Push your assessor to provide real examples of their recommendations and put them into the context of organizational security and compliance. If they say to fix a flaw in PHP, they should be able to define the risk that flaw presents to the business, and not just regurgitate CVE scores. They should also know how to make the fix.
Separate Assessment & Remediation
Wherever possible, use different resources for assessment and remediation work. The only exception to this is PCI compliance. There is benefit to having your PCI assessor help you with remediation efforts. This ensures you get a “consistency of interpretation” between the assessment and remediation efforts.
Look Beyond the Standard
Compliance standards, like PCI-DSS, NERC-CIP and HIPAA/HITECH are frankly horrible ways to secure your business. They are too focused on protecting one aspect of your business vs. the entire enterprise.
Security assessments should always look beyond the relevant compliance standard to address systemic issues. Simply meeting compliance standards is insufficient. If you are going to spend the time and effort to become compliant, you should leverage that effort to improve the entire business.
Stay in the Driver’s Seat
Do not let sales people to take control of your assessment or security efforts. Keep control of the process and drive expectations and requirements. Remember, sales people want to sell, not secure.
Security assessments are an integral part of any security program. There is a diverse collection of assessors in the industry, with a an equally diverse skill set. Security, unlike other technology efforts, has a deep emotional component: fear. Less scrupulous assessors can manipulate this fear to either lull you into a false sense of safety, or exaggerate unreal threats to inflate the value of their insights. Mature and responsible security practitioners do not let emotions rule their vendor selections.
The way to do this is with scientific methods of analysis. When assessors rely on rational, evidence-based analysis, then the conclusions and recommendations are plain for all to see and review. Moreover, that analysis must come from an experienced assessor who does not just know the theories and compliance rules, but also the practical application of security controls.
If only organizations were smart and mature enough to hire and retain goodassessors. In my experience there is no incentive for the assessors to provide objective assessments (as opposed to telling the client what they want to hear). Where are these clients you speak of?
In my experience, most infosec people (ISOs, CISOs, etc.) genuinely want to do the right things. However, executive management will bind them to assessors who were selected based on a relationship and not expertise or quality. As such, they become stuck with weak assessors who just keep telling them to buy more stuff or fret about ridiculous threats. In our practice, we find that empowering the infosec team to communicate the true nature of risk to executive management will often “right the ship” and refocus efforts. Sometimes that means echoing exactly what the security person already knows to be a problem. So, there are cases where you do just tell them what they want to hear (assuming it is the correct thing to say), but coming from an external party, it can carry more weight.
One more thing. You urge people to hire assessors with so-called “hands-on skills.” I am so sick and tired of HR twits telling me I have too many, or not enough, “hands-on skills,” whatever today’s definition of the term is.
Seems to me that what you are saying is, the more experience one obtains performing security control assessments, conversely, the same amount of time is not spent developing or maintaining “hands-on skills,” and therefore, by working as a security control assessor one is at the same time rendering oneself less qualified for future roles as a security control assessor.
I don’t think I meant to suggest that Catch-22 of the more you know the less you know. It comes down to basic tech skills. A good assessor needs a background in working with technology. This gives them respect for the complexities and realities of running systems.
I recall a particularly bad assessor I knew in the past who wrote a lengthy explanation for why an organization needed to implement this complex federated identity management solution to ensure compliance with something. It was all well and good, but the client was never going to do the implementation because it was a monumental amount of work, their budgets are already strained, and they lacked some of the fundamental technologies and controls to even consider doing it. As such, the advice was meaningless. When this assessor protested our analysis, it became clear that this person did not really know how an IT department worked, because they never had really worked in an IT department before. All their skills came from certifications and college classes. This person had no real-world experience, and therefore was not a very useful assessor.
So it is not so much that you are doing hands-on IT work, its that you have done it in the past. That experience is tremendously valuable because it shapes the advice you give out.