After the last post, I thought I’d describe some of the challenges with measuring the effectiveness of PCI DSS. Some camps argue it is absolutely effective because there has not been a compromise to date of an entity that was fully compliant with PCI DSS at the time of their breach. Others suggest extremely low compliance rates in certain groups of merchants indicate it’s not effective in helping the little guy. A few pick up headlines and just scream that it’s broken.

An industry colleague of mine, Steve Levinson, is famous for a number of sayings. One he uses when faced with numbers that sometimes don’t make sense is:

“There are lies, damn lies, and statistics.”

While I know he didn’t create that sound byte, history is littered with examples of number manipulation to support a particular message. Sometimes it is unintentional as with confirmation bias, sometimes it is deliberate. Just remember, all data analysis has some bias in it.

As an example, I grabbed the current breach data set from PrivacyRights.org and did a count of the number of breaches that used descriptive words to indicate a breach of credit card numbers. The graph looks like this:

dateannounced

Data sourced from PrivacyRights.org.

Given that we are only 2/3 of the way through 2014, it appears that we are trending to have more breaches than 2013 (possibly even 2012). The bias in this data set comes from a limitation. The data must be reported in a way that PrivacyRights.org captures it. Other than that, it’s just a count of the number of reported incidents, irrespective of size. One of the problem with measures like this is that they do not account for the breaches that could be. Meaning, how many breaches did PCI DSS prevent? Given that most companies can’t detect a breach, there is little chance that they could tell you how many actual attacks (not drive-by script kiddies) they successfully thwarted.

Spoon, by felixtsao

Spoon, by felixtsao

If the measure of effectiveness is to markedly improve the information security posture in retail, then I would say the answer used to be an emphatic “YES”. PCI DSS, at its inception, caused retailers to radically improve their information security departments and overall security posture. Retailers went from weak security to improved programs that complied (or mostly complied) with PCI DSS. I watched a number of companies make herculean efforts to do just that. However, since its release in 2004, PCI DSS has not changed enough to cause the same radical improvements.

To use arbitrary numbers by way of explanation, retailers went from 0 to 100, and are generally maintaining somewhere in the 80-120 range since the end of the last decade. Meaning, most do something for PCI DSS, some go above and beyond, and some fall just short. Since there has not been a new target set at 200, as an example, there is no catalyst to reform information security postures again. Therefore, yes, PCI DSS markedly improved the information security posture in retail up to the end of the last decade, but has not had the same effect in this one.

Given the recent congressional testimony by leaders in the payments space, it appears that lawmakers are concerned enough to potentially take action. So what measure could you show them to illustrate effectiveness? Let’s brainstorm in the comments below (or reply via Facebook and Twitter).

This post originally appeared on BrandenWilliams.com.

Possibly Related Posts:


1