Web Application Security

201x: The Year of the Security Industry Breach

jeremiahWhat if right now, people and companies are spending billions of dollars, each and every year, to be less secure? What if security products, those things indoctrinated by best-practices and mandated by compliance obligations, are actually the weakest link in the security chain? What if defense-in-depth is really nothing more than a nice idea. I’m confident this reality is coming, soon even. An inescapable and ironic situation where the security industry is going to be publicly embarrassed and must eat its own dog food for a change. That yucky, yucky stuff we’ve been trying to force down the throats of everyone else for decades.

Wait. Stop. Let me back up.

Back in the day, malicious hackers commonly targeted unpatched FTP, mail, and DNS servers — others brute-forced telnet ports. From a defense perspective, patch and configuration management in an enterprise environment is often difficult and expensive, particularly when there’s a lot of hosts to protect. This was is a leading reason why network firewalls are pervasively deployed across basically all Internet-connected organizations, to hide away insecure software from the hostile wilds of the Internet. This was the classic network response to an inherent software security problem, a problem no one from that industry ever bothered to address. What did happen was 65,536 ports became just two, 80 and 443 (Web), and developers largely just recreated the software that already existed to run on those ports, and within a Web browser.

In response, the bad guys SHIFTED.

The bad guys began focusing their attacks at the Web-layer, which is where today we see the majority of the breaches taking place and almost all of the data being lost.

During the same period of time, other digital miscreants preferred hacking operating systems, such Windows, which for a long while was fairly trivial. The recommendation from the Information Security (InfoSec) industry, from their RSA keynote stages, was to spend more on firewalls and anti-virus software. Oh yeah, and patch, patch, patch… if you can.

Microsoft grew tired of being the security industry’s laughing stock and hacking’s path of least resistance, so they kicked off an initiative called Trustworthy Computing and invested heavily in their Software Development Lifecycle (SDL). It took some time, but their efforts in software security paid off.

How do we know? Well, the bad guys SHIFTED.

We saw that instead of predominantly targeting Windows, our cyber adversaries began exploiting the applications installed on top of the desktop. Applications like Web browsers, Microsoft Office, PDF processors, and email clients — but mostly the Web browsers. Of course the InfoSec industry said, buy more firewalls! Buy more [email and browser] AV! And sadly, a lot of people listened. Patching, yeah, that‘s good idea. Do that too! Some listened.

Then, as we saw starting around 2007, to exploit thousands, nay, millions of PCs, all one had to do was SQL Inject a vulnerable website, take your pick of millions because remember that pesky 80/443 software security problem was never solved by firewalls or AV, and lace those websites with browser-based exploits. Well-known exploits, zero-days, it didn’t matter which, few people ever kept up on browser patches anyway. Yet, more money spent on firewalls and AV just the same.

Browsers and browser vendors, Google, Microsoft, and Mozilla, then took their turn in the exploitation crosshairs, feeling the pain of a lack of adequate software security in their respective products. Browser exploitation became a leading cause of malware propagation. As we could expect, they didn’t much like that position, that reputation, even when offering a $0 application. Users expected the software to be secure. Browser vendors had to get serious about software security, and you know what, they did!

How do we know? Wait for it… the bad guys SHIFTED!

If you notice, the bad guys next began targeting browser plug-ins, namely Flash and Java. Yep, that software already installed in just about everyones browser. That software riddled with vulnerabilities. That software rarely patched by the end-user. Adobe (Flash) and Oracle (Java) are working through their own software security nightmare right now.

The browser vendors, not content to wait for Adobe and Oracle, are taking their own steps to protect their platforms, like Microsoft did by offering up ASLR and DEP. They are alerting, disabling or unbundling outdated plugins, if not actively uninstalling them altogether, and making sure technology exists so as not to need them at all. See HTML5.

Give the situation 6 months. Give it a year. It’s difficult to say when exactly, but eventually browser hacking, including the plugins, will get sufficiently difficult to warrant another SHIFT. So where will the bad guys focus next? That’s the billion dollar question. My bet is not “mobile,” while that day may come too, I think it’ll be “security products” targeted first.

Think about it. Think about all the security products out there, such as IPS, DLP, WAFs, various deployment forms of AV software, and so on are pervasive across enterprise networks and end-user PCs. These products are designed to parse and analyze data from unknown origins, making them ripe for haxoring. Products whose makers, the InfoSec Industry, never really had an emphasis on “software security,” if they even know what that concept is, and notoriously bad at handling vulnerability disclosure — a sure sign of immaturity. Their only training being how to sell more firewalls and AV.

Imagine an email specifically designed to exploit a system, but only one protected by an anti-virus email gateway. A piece of Web page code that exploits a browser, but only those protected by anti-virus software. Incoming Web traffic whose goal is to compromise an IPS or WAF itself, not necessarily the website behind it.

None of this is far fetched. In fact, the writing is already on the wall. Just look at what Tavis Ormandy did recently to Sophos’s products in his spare time. No one should be naive enough to believe this is an anomaly. How many zero-days do you think are yet to be found in that software? What about other AV products? What about all the other security products out there? Juicy untouched zero-day heaven, that’s what it is. Oh right right, we know the answer we’ll be given. Buy more firewalls and AV! And of course people will listen, but what for? To protect insecure firewalls, insecure AV, and the other insecure security products? Please.

I’m sure the industry apologists will also predictably say, “there is no silver bullet,” as if that somehow absolves responsibility for shipping risk increasing products.

Hacktivists, cyber-criminals, nation-state sponsored APT, however we label them, we’ve witnessed how our adversaries select their targets, and especially the method of attack, typically by the path of least resistance. One vulnerability is all a bad guy really needs, and the first and easiest one to identify and exploit will do just fine. So when one path of attack doesn’t work or becomes too difficult, the bad guys will shift. Reporters and PR agencies, get your digital ink ready, we’re in for a bumpy ride.

  • josh

    I know for a FACT that a number of security appliances (think IDS/IPS/DLP sensors and similar products) are terrible about patch management. Most of them are just running on top of an out of date Linux build, and I have personally been on several calls with a couple of the largest security vendors trying to get them to patch their shit (because maintenance contracts prohibit us from manually patch the appliances we paid millions to own). It reminds me very much of the book Cuckoo’s Egg, where Cliff Stroll kept finding default passwords and other inexcusable problems on the systems of defense contractors whose specific market was making secure system builds. Inexcusable.

    probably a black hat talk in publically shaming some of these companies for anyone interested

  • https://twitter.com/arhn7 arhn7

    Sorry being a killjoy but there is a typo —

    “but only those protected by anti-virus softwre” <– Spell check

    • http://www.whitehatsec.com/ Jeremiah Grossman

      @arhn7 kill away! fixed :)

  • Brian Dead

    I don’t doubt that you’re right about the weaknesses in security software products. However, there is one big factor that your prediction overlooks, even though you mention the cause of it.

    The targets of the previous waves of infection – DOS, Floppies, Office, Windows, Browsers, Java, Flash, etc – have all been very dominant platforms providing an almost ubiquitous vector for attack. This ubiquity also reduces the breadth of research for finding targetable vulnerabilities. These ubiquitous technologies are therefore very deeply probed for weaknesses, and exploiting those weaknesses can have a very wide effect. There’s a high return on a limited investment.

    Targeting security products, for the most part, is not as economically viable. It’s unlikely that a file crafted to exploit one vendor’s products would work against another’s, so attackers are going to have to study more products in order to launch attacks that will impact a smaller population.

    In short – past experience suggests that the cost/return ratio needs to be heavily stacked by the ubiquity and monopoly of an attack platform for it to become a persistent, widespread, general threat vector. So although there’s undoubtedly some glaring errors in some products that could provide a short burst of low-hanging fruit, I doubt security products will become the next Flash or Java. But unless they improve they will provide a range of holes that could be abused for targeted attacks.

    • Ben

      I can see several likely targets, on the desktop there is wide deployment of very few AV and Endpoint/DLP solutions. Target these in the way the article describes and you potentially have a widespread compromise that would be hard to detect and/or remove.

      The other end of the picture is the in-line appliances, AV, IPS, even proxies, compromise a SSL inspecting proxy that has a trusted re-signing cert on it and you have man-in-the-middle on an entire organisation that may go undetected for at least hours and potentially months depending on the entities monitoring capabilities and expertise of the engineers managing the devices.

      When you look at the vendor landscape there are vendors in each segment that have massive market penetration and an exploit to compromise a single model could potentially expose millions of people. Conversely, even small (potentially immature) players in markets, if they win a few key deployments could potentially put a handful of their devices in front of thousands or even tens of thousands of users. If someone then targets their appliance and is able to compromise it easily the potential payout (by either harvesting information or using the devices as an platforms for secondary vectors launched from inside the private network) is very large for a relatively small amount of effort.

      The other thing to consider is that many of the big players are currently overhauling their software and adding new feature on top of new feature to push the trend of making management easier and more integrated with this like Microsoft AD to provide identity awareness and identity based policy. I would put money on the fact that all of this new code and functionality is going to open these historically tight and secure platforms to all kinds of new attack vectors either through errors or oversights in the code or through exploitation of the devices they depend on to deliver this new functionality.

  • http://attrition.org/ Jericho

    The repetition of “the bad guys SHIFTED!” is excellent, and straight to the point.

    The only thing that would have enhanced this blog, is an equally repetitive “.. and the good guys DID NOT!”

    • http://www.whitehatsec.com/ Jeremiah Grossman

      Thanks! Yah, I should have done something like that.

  • Carl

    I think we may already be seeing the shift towards what private investigators, flimflammers, check kiters, and con men have know for decades: social engineering can get you pretty far. Phishing, impersonation, etc. can open a lot of doors for a relatively low price. True multi-factor authentication and authorization schemes can mitigate this by limiting the time-scope opened by a single exploitation, but the cost and inconvenience of adoption (in most instances, it’s gross overkill) means that it will remain a viable stratagy for the forseeable future.

    Some say “You can’t patch the carbon layer,” but I’m not entirely sure that we’d WANT to: Social engineering exploits the human disposition to help others, and I’m not sure that the security gains of universal suspicion and paranoia are worth the social cost.

  • http://blog.diniscruz.com Dinis Cruz

    I agree, and this will also include Application Security Tools and Services, like for example Sentinel :)

    Think about it, how valuable (from an attacker’s point of view) are the list of vulnerabilities that current exist on your database, Veracode’s of Fortify on Demand?

    What about creating special payloads that trigger DAST (or even SAST) engines?

    • http://www.whitehatsec.com/ Jeremiah Grossman

      @Dinis Everything we find is technically a 0-day, a one-off, and not terribly useful for a mass scale attack. Secondly, while some might be interested in obtaining our vuln data to exploit whomever their target is, it would be far easier just to go after the target directly. I mean its not like webapp flaws are hard to find. 😉

  • Pingback: Water Polo Blog()

  • http://www.mypoptrack.com www.mypoptrack.Com

    I’m really inspired together with your writing skills as well as with the format on your blog. Is this a paid topic or did you customize it yourself? Either way keep up the excellent high quality writing, it’s

    uncommon to peer a nice weblog like this one these days.


  • Pingback: Security in 2013: Intelligence, Coordination and Integration (and Will We Get There?) « EMA Blog Community()

  • Pingback: The Leverage Attack: Do We Really Get It? « EMA Blog Community()

  • Pingback: Security in 2013: Intelligence, Coordination and Integration (and Will We Get There?) « EMA Blog Community()

  • Pingback: The Leverage Attack: Do We Really Get It? « EMA Blog Community()