Sin #6 – Q/A Tunnel Vision

The Quality Assurance (Q/A) program is in full swing at the PCI Security Standards Council. After companies started taking PCI DSS seriously and retaining QSAs, merchants and service providers realized that not every QSA interpreted requirements the same. One of the biggest complaints about the QSA community is variance in interpretation on key items that could impact the cost of compliance—positive or negative. The Q/A program was announced at the 2008 PCI Community Meeting1 and began to take effect shortly thereafter. QSAs were put on the remediation list as early as 2009.

Myopic Assessment Views

The objective of the Q/A program was to decrease the variance in interpretation among QSAs and increase the overall quality of assessments. Each QSA company must submit redacted Reports on Compliance (ROCs) from a given time period, and each report is reviewed and scored on upwards of 700 different items.

GYpix1376_eyedoc, by gregor_y

The results so far? Inconclusive as far as I am concerned. The overall quality of the deliverable coming out of a PCI assessment is improving in direct relationship to this program, but what is missing is that on-site touch and feel that simply is not possible today. The Q/A process has effectively trained QSAs to produce better reports—potentially by pre-writing deliverables before the engagement starts. A solid deliverable is great, but if it does not accurately represent the environment to which is it written, it does no good to the company, their acquirer, and ultimately the QSA community.

One move by the Council that may help close this gap is the institution of the PCI Forensic Investigator (PFI) program, which replaced the Qualified Incident Response Assessor (QIRA) program from Visa, Inc. If the Council can see the forensic report from a breach in conjunction with the original ROC, it’s easier for them to take action against a QSA after a breach.

If a QSA becomes hyper-focused on the Q/A program, they will neglect to focus on the real issue—performing a thorough assessment and making sure the deliverable matches the assessed environment. I’ve seen QSAs armed with “known passable responses” to questions in massive spreadsheets ready to cut and paste away. If those chunks are edited to the environment, there is absolutely nothing wrong with this approach. Lawyers do it all the time2. But if pre-written comments are placed in the report without editing, you end up with a document that passes the Q/A process but fails the assessed entity.

How to Avoid Quality Myopia

The only way you can avoid this problem is to watch your QSA operate during the assessment and do your homework on what an assessment really takes before signing a contract. You can obtain the latter by becoming an Internal Security Assessor (ISA) which gives you the same training a QSA gets with a test at the end. Scoping is a big part of the QSA and ISA training, and this knowledge will help you budget for your next assessment. If your bid solicitation process produces quotes with a standard deviation greater than the value of one of the bids, someone doesn’t understand your requirements.

This post originally appeared on

  1. If you are a stakeholder in PCI DSS and are not going to these meetings, you are missing out. []
  2. In fact, I’ve wondered if the measure of a good contract lawyer is one that knows WHICH contract to plagiarize. []