In 2005 - for the first time in history - all major payment brands collaborated together to create a unique set of requirements (PCI DSS) aimed at reducing credit card fraud. As a consequence, we have seen a demand for new security-related solutions and services emerging.

We didn't have to wait long to see the security industry respond to this demand, integrating the 3 letter acronym into their marketing plans. Suddenly every security company is a self-proclaimed PCI expert and is offering to help you become compliant. With so much noise, there was a need for some kind of regulation to guarantee the quality of all this "help". The PCIco partly addressed this need by establishing the thesholds for qualification of two major actors of the program: namely the Approved Scanning Vendors (ASV) and the Qualified Security Auditors (QSA).

I was working at MasterCard in 2005 when the requirements were put together and was personally charged with the creation and management of the certification program for ASVs. The PCIco does not certify products; this is not their core competency and never will be, so the aim of the ASV certification is to verify the ability of a scanning vendor to detect, report vulnerabilities and misconfiguration.

My team had to do something that had never been done: build an intentionally insecure network. While this sounds fairly easy - by definition isn't it insecure out of the box? - it's actually not straightforward to do it deliberately for a heterogeneous network of firewalls, routers, DNS, mail, application and database servers comprised of a diversity of services and applications. Furthermore, we had to know the exact list of flaws for each target. We did this to replicate the process ASVs go through when they scan a network.

Without much more information than a list of IPs, vendors have to scan 10-16 remote targets within a specific time window, which may be considered too short for some of them. Vendors are expected to treat the certification body (test lab) as a customer, using the same process and scanning technology they intend to use on the field.

Having led this program for about 5 years I can tell you how difficult it is to pass the test. Targets are regularly reconfigured and vulnerabilities frequently added or removed.

To pass the test, a vendor must report their results in the expected PCI format and reach a specific threshold (%) of findings for each target.

I saw hundreds of companies failing again and again while others were passing with our compliments each year. I came to the conclusion that the success resides in two areas:

  1. the scanning technology used, made of up of a scanning engine, vulnerability databases, and reporting systems.
  2. the skills and knowledge of individuals using the scanners. While not all scanners are adequate for the task, scanners that have been incorrectly configured are disasterous.

Since April 2011, the PCIco has been pushing their quest for quality further by requiring employees of ASVs to take training and pass a test on an annual basis, in addition to the existing requirement for the organization's ASV solution to be annually recertified. Furthermore, the PCIco is currently defining a quality program with the aim of controlling ASVs on the field.

Much more still needs to be done in the domain of quality and qualification though. One area were we could see the PCIco adopting a certification program in the future is penetration testing, though at present this is occupying a kind of no man's land for ambigueous reasons.

Didier Godart