Editor’s note: Ari Elias-Bachrach is the sole proprietor of Defensium LLC. Ari is an application security expert. Having spent significant time breaking into web and mobile applications of all sorts as a penetration tester, he now works to try and improve application security. As a former developer who has experience with both static and dynamic analysis he can work closely with developers to try and remediate vulnerabilities. He has also developed and taught secure development classes, and can help make security part of the SDLC. He is a regular speaker on the field of application security at conferences. He can be found on Twitter @angelofsecurity. Given his experience and expertise, we asked Ari to review our 2014 Website Security Statistics Report which was announced yesterday to get his thoughts which he has shared as a guest blog post.
The most interesting and telling chart in my opinion is the Vulnerability class by language chart. I decided to start by asking myself a simple question: can vulnerabilities be dependent on the language used, and if so which vulnerabilities? I did a standard deviation on all vulnerability classes to see which ones had a high degree of variance across the different languages. XSS (13.2) and information leakage (16.4) were the two highest. In other words, those are the two vulnerabilities which can be most affected by the choice of programming language. In retrospect info disclosure isn’t surprising at all, but XSS is a little interesting. The third one is SQLi, which had a standard deviation of 3.8, and everything else is lower than that.
Conclusion 1: The presence or absence of Cross-site scripting and information disclosure vulnerabilities is very dependent on the environment used, and SQLi is a little bit dependent on the environment. Everything else isn’t affected that much.
Now while it seems that frameworks can do great things with respect to security, if you live by the framework, then you die by the framework. Looking at the “Days vulnerability open by language” chart, you can see some clear outliers where it looks like certain vulnerabilities simply cannot be fixed. If the developer can’t fix a problem in code, and you have to wait for an update to the framework, then you end up with those few really high mean times to fix. This brings us to the negative consequences of relying on the framework to take care of security for us – it can limit our ability to make security fixes as well. In this case the HTTP response splitting issue with ASP are both problems that cannot be fixed in the code, but require waiting for the vendor to make a change, which they may or may not judge necessary.
Conclusion 2: Live by the framework, die by the framework.
Also interesting is that XSS, which has the highest variance in occurrence, has the least variance in terms of time to fix. I guess once it occurs, fixing an XSS issue is always about the same level of effort regardless of language. Honestly I have no idea why this would be, I just find it very interesting.
Conclusion 3: Once it occurs, fixing an XSS issue is always about the same level of effort regardless of language. I can’t fathom the reason why, but my gut tells me it might be important.
I found the “Remediation rate by vulnerability class” chart to be perhaps the most surprising (at least to me). I would have assumed that the remediation rates per vulnerability would have been more closely correlated to the risk posed by each vulnerability, however that does not appear to be the case. Even more surprisingly, the remediation rates do not seem to be correlated to the ease of fixing the vulnerability, as measured by the previous chart on the number of days each vulnerability stayed open. Looking at SQLi for example, the remediation rate is high in asp, ColdFusion, .NET, and Java, and incredibly low in PHP and Perl. However PHP and Perl were the two languages where SQLi vulnerabilities were fixed the fastest! Why would they be getting fixed less often than other environments? XSS likewise seems to be easiest to fix in PHP, yet that’s the least likely place for it to be fixed. Perhaps some of this can be explained by a single phenomena – in some environments, it’s not worth fixing a vulnerability unless it can be done quickly and cheaply. If it’s a complex fix, it is simply not a priority. This would lead to low remediation rates and low days to patch at the same time. In my personal (and purely empirical non-scientific) experience, perl and php websites tend to be put up by smaller organizations, with less mature processes and a lesser emphasis on security and a greater focus on continuing to create new features. That may explain why many Perl and PHP vulnerability are either fixed fast or not at all. Without knowing more, my best guess is that many of the relationships here, while correlated, do not appear to be causal. In other words, some other force, like organizational culture, is driving both the choice of language and the remediation rate.
Conclusion 4: Remediation rates do vary across language, but the reasons seem to be unclear.
I started off with a very basic question “does choice of programming language matter”, and the answer does seem to be yes. While we all know that in theory there is no vulnerability that can’t exist in a given environment, and there’s no vulnerability that can’t be fixed in any given environment, the real world rarely works as neatly as it should “in theory”. Certain vulnerabilities are more likely in certain environments, and fixes may be easier or harder to apply, which impacts their likelihood of ever being applied. There has been a lot of talk lately about moving security into the framework, and this does provide evidence that this approach can be very successful. However it also shows the risks of this approach if the framework does not implement the right security controls and in the right way.