Recent events have caused people all over the information security community to question the efficacy of the PCI-DSS. The Target breach has become a lightning rod for debate as to how well the PCI-DSS protects organizations.

In a recent blog entry, Avivah Litan from Gartner said: “the PCI security standard has largely been a failure when you consider its initial purpose and history.” This entry goes on to discuss how the PCI-DSS has nothing in it that would have detected and stopped the latest malware in the Target breach. The full blog entry is here.

Yesterday, Bob Russo, head of the PCI Council issued a statement countering Litan’s assessment. In this statement, Russo claims that the majority of breaches have come from a breakdown in security basics such as poor implementation or maintenance of controls. Russo defends the standard as being an “excellent line of defense.”

They are both correct and they are both avoiding the elephant in the room: the failure of the PCI standard is not the standard itself; rather it is the deplorable state of Qualified Security Assessors (QSAs) who are passing companies that are nowhere near compliant. The PCI-DSS has ample controls that would catch malware like that used in the Target breach. What the PCI-DSS lacks are assessors who can effectively analyze an organization to determine if those controls actually work.


Appalling Assessments

The current state of QSAs is appalling. While there are plenty of capable and competent QSAs, there are also plenty of truly awful ones. Those awful ones seem to stay very busy. We have observed organizations that are profoundly non-compliant get passing grades from other QSAs. We have conducted assessments where clients were significantly out of compliance, which enraged the business leaders. These leaders then shopped around for another QSA who willingly passed the company in a non-compliant state. Furthermore, clients have outright solicited us to ignore their non-compliant configurations and let them pass, citing the fact that other QSAs were willing to pass them and therefore we should as well.

In one incident, about two years ago, I met with the compliance team at a large, national company. The director of this team said to me, in very clear terms, that they did not care what the PCI-DSS said, they were compliant, and if we wanted their business we would pass them without question. They told me they fired their previous QSA because they “took too long performing quality assurance.” When I told them this was not how Anitian worked, they dismissed us immediately and made it clear that our competitors had no problem with these assurances and that we clearly did not know how PCI worked. It was beyond demoralizing to hear this, especially coming from a company that has a very respectable name (which I will not mention for obvious legal reasons.)

Moreover, there are assessment firms who aggressively promote checkbox style assessments as a cost-effective way to compliance. They use euphemistic terms like “compliance as a service” to tip off business leaders that PCI is nothing more than checking off some boxes on a form. They also make it clear that they will pass a company regardless of their actual compliance state.

Furthermore, these are not small-time QSA companies operating out of dingy offices. These are widely recognized names with thousands of PCI clients. They have become extremely successful at selling a type of compliance that is appealing to business leaders who do not comprehend the scope of the risks, and just want PCI to go away.

Weak QSA companies ultimately breed weak assessors. Many of these firms hire people who only have basic audit skills and no technical acumen.  These people supposedly understand the PCI standard and process but lack the requisite experience to effectively evaluate if technical controls are implemented correctly on systems which they don’t understand. Their technical inexperience becomes an insecurity, and rather than admit they do not know something, they just ignore the complex problems. Compliant entities have seized upon this and purposefully created ridiculously complex descriptions of their security controls for the sole purpose of intimidating inexperienced auditors.

For example, Anitian has worked with a number of clients who had been compliant for years. Each year the previous QSA would come in, spend a few days locked in a conference room, talk with nobody, look at no systems, review no configurations, and then deliver a signed Report on Compliance. It was shocking for both us and the client to discover how little the QSA understood the environment. Yet, they took no time to actually look at anything or confirm that systems were configured in the manner the company had claimed.

Exacerbating this problem are companies who are both QSAs and product resellers (for full disclosure, Anitian does both as well.)  In these cases, there is an endemic bias in the assessment process since the assessor not only is approving the control, they also sold the control to the company. At Anitian, we keep these two teams entirely separate and enforce strict separation of duties. However, many QSA companies have no such separation and will use their role as the auditor to push whatever products they are selling.


The Council Responds

The PCI Standards Council has tried to clean up these weak QSAs, but with middling success. A few years ago, they introduced a quality assurance process for QSA companies. QSA companies are regularly audited to ensure they are following the assessment rules. Moreover, they introduced a process for removing companies that do not comply with quality assurance rules. However, in the years since this was introduced, a tiny fraction of companies have been removed from the list of QSA companies. Occasionally a QSA company will show up as “in remediation” indicating there has been a QA problem, but these are fleeting and quickly disappear.

The QA process was a step in the right direction, and frankly, the Council deserves credit for implementing it. It goes without saying that some QSA firms were unhappy to see those QA programs implemented since it meant more overhead for their assessments.

It would be logical to attribute our position in this blog entry to mere sour grapes for losing opportunities. There is truth to that assertion. We are frustrated that some people portray our honesty and integrity as a weakness. It is discouraging when the management of a respected brand-name company tells you that the only way you can earn their business is to be unethical. It is equally discouraging to hear about breaches, which were the result in basic lapses in security, while simultaneously QSA companies are promoting weak assessments as a competitive advantage.

Yes, we are frustrated, but it is because we agree with Russo that the PCI-DSS is a good standard for security – if companies actually followed it and QSAs actually enforced it.


Improve the QSA Certification Process

The PCI industry needs to grow up. This begins with some improvements to the PCI process of certifying and educating assessors. If we want to stop the breaches, the entire PCI process needs some improvements. As such, our recommendations for Mr. Russo and the PCI Council include:


1. Significantly Increase the Technical Expertise Necessary to be a QSA

The current QSA testing and certification process is focused exclusively on explaining the PCI standard itself. The training does almost no evaluation of an assessor’s technical skills. The certification process should have a more rigorous testing of technical concepts, particularly around network segmentation and system security. All potential QSAs should be required to pass a technical test, or attend a remedial security controls class.  A QSA should be able to expertly analyze complex network and system controls.


2. Reformulate Audits of QSA Companies

The PCI Council currently conducts audits of QSA companies on a quarterly basis. The audits are predominantly a paperwork exercise. The audit effort should focus on interviewing the actual QSAs themselves and less on paperwork. This should be an oral test of sorts, where a QSA has to respond to questions about scoping, technical controls, and the intent of the standard. QSAs that do not meet minimum qualifications should be required to attend remedial technical training or face removal from the QSA list.


3. Forbid QSAs from Performing Integration Work or Vice-Versa

There is too much opportunity for corruption and bias when a QSA is both the auditor and the integrator of compliance solutions. QSA companies should be limited to one or the other, but not both. A company that sells remediation technologies to an assessed entity should automatically be forbidden from also conducting their annual QSA audit. Likewise, a company that is conducting an annual assessment should be forbidden from selling security products to those same clients.


4. Forbid Long Term Assessment Agreements

Companies should be required to change their QSA assessor every two years. This will cut down on long, multi-year contracts which promote complacency.


It should be noted that many of these improvements would hurt our own business. However, we believe that if the PCI-DSS is to remain viable, it must improve the integrity of the assessment process.

The Council has already shown a willingness to start demanding higher quality work.  In the PCI-DSS 3.0, for example, the rules around penetration testing were significantly augmented. This was in direct response to companies hiring low-cost, low-effort, and low-quality “penetration testers” to fulfill the requirements. These tests were nothing more than glorified vulnerability scans and did not meet the purpose of the exercise. The new standard requires more rigorous testing process and validation of scope. Clearly, the Council is not afraid of going after weak testing. They just need to now extend that to the QSAs they certify.



The PCI-DSS is only as good as the people and companies that conduct assessments. Russo is correct; most breaches come from organizations missing basic security controls. If the PCI-DSS is going to remain relevant, it needs to be something that companies, consumers, and governments can trust to ensure those basic security controls are in place and functioning correctly. While weak QSAs might not be the reason for all breaches, they are eroding public trust in the PCI-DSS and facilitating conditions where breaches are more likely.

It is time to make the PCI-DSS be something that companies can really trust. This will means that the PCI Council is going to have to stand up to some of the largest names in compliance and demand they behave in a more responsible manner.