I had an interesting conversation with a CISO colleague of mine the other day. We chatted about information security program development and maturation in general, but more specifically about his web application security program, which is a subset of his larger enterprise program. The conversation was interesting in that his program is quite mature, adequately staffed and funded, and is delivering the results and program benefits planned a few short years earlier. Still, he was having some nagging doubts about its efficacy going forward.
Over the previous few years, he had fully implemented a DevSecOps approach; his security architecture teams had created a comprehensive reference architecture, enlisted and “sold” to the dev teams and their leadership on the value of developer security training being provided. He identified developers who would function as security captains (mentors) within the various teams, such that at this point the dev teams are now performing a majority of their own security testing (hack thyself first) during development (using WhiteHat’s SAST solution and other tools).
Vulnerabilities were identified and remediated very early on in the SDLC, thus reducing remediation costs and the risk associated with late dev or production site defects. A combination of dynamic security testing (DAST) and manual penetration testing using various approaches (such as white/black box and Red Teaming) was implemented, and it was producing good results. His organization had fully embraced defense in depth (varying the use of tools and talents, vendors, testing strategies, and methodologies) in a layered approach. From my perspective, his program was fully meeting, if not exceeding, the expectations most organizations would have of its application security program.
As we chatted further, the issue became clearer to me. My colleague, like all good security executives/professionals, has a very astute and active “paranoid” gene. He was concerned that the game had evolved, that new or additional techniques might be required, and he was concerned about becoming complacent with his program. In short, he was asking himself, “what needed to change, and what should stay the same?”
His concerns boiled down to a couple of things:
#1: Defect/vulnerability metrics had changed over time.
As this CISO’s program matured, all of the easily identified vulnerabilities were identified and remediated, so the open vulnerability counts were lower overall and the severity of defects found tended to be more severe, e.g., injection defects, persistent XSS, etc. This is a “benefit” and a natural result of the right tools and practices being put to use in a mature program, which in this case included:
- Developer training, which resulted in cleaner code and fewer defects in the first place
- Early defect identification and remediation via static source code testing
- Pre-production dynamic and release-based manual penetration testing, which resulted in early identification of defects in the compiled app
- Production dynamic testing — always on, always testing
- Production point-in-time or release-based manual penetration testing (finding fewer, but perhaps more significant defects in the compiled app)
- Use of nontraditional programs, such as bug bounties
- Bug bounties are a “new” cost to the overall program, and a nontraditional approach
- Information Risk Management (IRM)/risk mitigation teams involved to track and manage risk/defect mitigation
- Threat and Vulnerability Management (TVM) teams and red teams involved in the hunt (catch me if you can!)
All of the above resulted in the flushing out of more significant vulnerabilities.
#2: A number of the “higher severity” vulnerabilities were found via manual testing.
Application security is a continuous evolution. It’s a process that incorporates many different resources, tools, and talents, all of which augment the others and none of which is individually capable of delivering the whole.
My friend shared several concerns with me and here is how I responded.
CISO: “My red team found a defect that my dynamic tester or other vendor didn’t.”
My response: “Great! The program is working as intended – that’s defense in depth. You should only be concerned if a vendor/tool is regularly not finding key/serious defects. Then it’s probably time to reevaluate the vendor.”
CISO: “What to change or not in my InfoSec program?”
My response: “Change is a reality, Mergers & Acquisitions occur, developers come and go, software/security reference architectures change, new software teams are on-boarded, new initiatives and technologies are introduced, and new programming languages are used.
“SAST/DAST dynamic testing and developer training, when used as part of a holistic security program, are very cost-effective ways to minimize the cost and risk that go with the ebb and flow of change, which is inevitable.”
CISO: “What about DAST, PCI and the insider threat?”
My response: “A DAST solution running continuously, when coupled with biannual or annual Business Logic Assessment (BLA), is very close in comparison to a full manual penetration test – and furthermore, when performed in this manner, dynamic testing meets requirements for web application penetration testing as set forth by PCI.
“Continuous dynamic testing can confirm unintended or surreptitious changes to the application.
“Dynamic testing that is federated via a robust identity and access management (IAM) solution makes it much more difficult for internal threat actors to deploy code, such that it looks as if the attack were initiated from the wild.”
So my colleague’s concerns were well-founded and worth investigating, as no security program should be allowed to slide into complacency. But his concerns about one team or vendor finding one defect over another is generally not a cause for alarm, but rather confirmation that the program is working as designed – good OpSec, good defense in depth.
The bottom line is this: No vendor, no employee, no solution, and no technology is ever 100% bullet proof. Defense in depth, coupled with DevSecOps and a solid OpSec program, is currently the best option for comprehensive information security coverage.