Technical Insight

Do We Know How to Make Software?

During the first half of the 20th century, the United States ruled the automotive industry — an industry we created from a machine we invented. For decades we built cars domestically and exported them throughout the world. Over time the auto manufacturing process was modernized and commoditized, which meant basically any company could make a product just as good, perhaps even better, but without needing uniquely skilled factory workers. The innovation in the manufacturing process opened the door to off-shoring to countries such as Japan and China to compete where a labor force was available and several times less expensive.

The trend is that we (in the U.S.) invent something new, create an industry around it, and manufacture it domestically for a while. Then someone really figures out how to make it efficiently, and the manufacturing jobs go overseas. This pattern has repeated itself many times in many industries. Electronics being another prime example. Unless a country has oodles of cheap labor, you must innovate or die — and survival is, of course, optional.

Here’s the thing, I don’t work in the automotive industry, nor electronics, or in anything having to do with physical goods for that matter. I’ve no idea where my toaster or microwave was built, though I doubt it was in the US. My business is software, software security if you want to be precise, and I’m wondering why, why after 50 years since the invention of software, is so much development, especially the high-end stuff, still performed in the United States?

This question was recently kindled after a ridiculously good article by, general partner of VC firm Andreessen-Horowitz and Netscape co-founder, Marc Andreessen. He called it, “Why Software Is Eating The World.” In his words…

“This is a tragedy since every company I work with is absolutely starved for talent. Qualified software engineers, managers, marketers and salespeople in Silicon Valley can rack up dozens of high-paying, high-upside job offers any time they want, while national unemployment and underemployment is sky high.”

If you in work in technology, are an entrepreneur or investor of any kind, or a student soon to be entering the labor force, this is a must read. Note the key words “talent,” “qualified,” and “high-paying.” These are not terms used to describing a commoditized industry, or a commoditized manufacturing process. We know our friends in asia are verifiable pros at tearing apart the inventions of others, breaking down its manufacturing process and streamlining the hell out of it. But for some reason not they, or anyone else for that matter, has the ability to stamp out quality software with the same ruthless proficiency as a car, computer, or lunch box. If they had, there would be a declining amount of programming jobs in the states, but that’s just not the case. Quite the opposite is true.

Software, the ability to automate intelligence, is one of mankind’s greatest inventions. The utility of software has already globally connected two billion people via the Internet, billions more if you count cell phones. Software is quickly becoming embedded in nearly everything, from power supplies to power grids, from medical devices to the management of all medical records. The power of software, and fact that it’s not a “solved” problem, signals amazing opportunity — world changing opportunities. Here are two more quotes from Mr. Andreessen’s article that speak to this.

“My own theory is that we are in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy.”

“More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense.”

All you have to do is look at the trends to see the truth of it, and perhaps one might even say, Marc is just predicting the present.

Today’s largest bookseller is a software company: Amazon

Today’s largest video service by subscribers is a software company: Netflix

Today’s dominant music companies are software companies: Apple’s iTunes, Pandora, and Spotify

Today’s fastest growing entertainment companies are video game makers (i.e. software companies): Zynga (maker of FarmVille) Rovio (maker of Angry Birds)

Today’s largest direct marketing platform is a software company: Google

Today’s fastest growing telecom company is a software company: Skype

Today’s fastest growing recruiting company: LinkedIn

The best new movie production company in decades was a software company: Pixar

Back to the original question. Do we, in a global sense, know how to make software? No, well yes, err sort of. At least not to the degree of efficiency and predictability of building cars, buildings, and anything with blinking lights. Today, software must be built by highly skilled people, whose skills are not trained up quickly or easily. Even those who profit by the billions from creating software, like Microsoft, Oracle, and Adobe seem unable to ship multi-million line software projects on a deadline. No one has yet discovered the secret formula to software development. Our modern day equivalent of the philosopher’s stone.

Perhaps the reason is as Wendy Nather, Research Director for The 451 Group, says, “the current approach to software development resembles more of a literary art rather than an engineering discipline.” Though no one is quite sure if the latter approach will lead to the promised land. What we do know is the future will be fueled by software.

  • http://www.internetsecuritydb.com dodgy_coder

    Nice article, the thing about software development is that in many ways it is an art and attempts to treat it like a physical engineering discipline, like building a suspension bridge for example, are doomed to fail. With the advent of new techniques such as test first development, continuous delivery, refactoring, and design patterns, things are improving.

    • http://greensecure.blogspot.com German Arduino

      dodgy_coder I couldn’t agree more with you, software development is an art and the attempts to industrialize this creative labor are going to fail.

      Also agree with the need of use Design Patterns and TDD as the valid ways to improve the software development process. These are techniques we are using (in Smalltalk) from long long time.

  • jmdorrian

    Terrific post. I think it is worth noting that the modern automobile (with petrol internal combustion engine) came into existence in the 1880s. It took foreign markets until the 1970s to begin assuming the role of manufacturer from American companies. So your observation that software has been around for 50 years and still remains largely American is correct, but give it 20 years and we’ll see the tides change. Probably sooner.

    • http://www.whitehatsec.com/ Jeremiah Grossman

      So the guess is ~20 more years and the rest of the world will likely figure out how to stamp out software efficiently. I’m hoping they do, figure it out at least, and than that I’ll be alive to see it!

  • PhilW

    I think that maybe “we” did at one time, but not any more. Mainframes built 1960 onwards were proprietary, both hardware and software integrated, and designed for banks, insurance companies, airlines etc with a lot of built-in security. There was an industry reboot around 1990 where smaller desktop or C language based systems started taking off, and they had no designed-in security at the user access level or buffer overflow level. The industry winners were scaled up desktops instead of scaled down mainframes. There’s more to it than that of course, but it does seem to me that even today you’d be hard-pressed to penetrate proprietary mainframes, and that’s not because there’s no juicy data on them – it’s because they are by nature very difficult to exploit.

  • http://www.flyingpenguin.com/ Davi Ottenheimer

    Well, maybe you knew I couldn’t resist commenting on your automobile engine analogy. I’m still laughing from the time last year you told me “when gas prices went up, prices on Suburbans went way down, so I bought one to drive my five miles to work”. Clearly we still don’t see eye-to-eye on managing risk.

    You say “the United States ruled the automotive industry — an industry we created from a machine we invented”. For brevity sake I’ll concede the industry was largely built by the US (not created) but I can’t let you assert that the machine was invented in the US. The engines of steam, electric, internal combustion, diesel…all were invented outside the US in the 1800s. I mean by comparison the US at that time was stuck in a rut over whether slavery was a viable engine to power its industrial production!

    Yeah, ok, I know Ford gets lots of credit for ramping up his assembly line and blowing a whistle at his workers, but even that was an application of British automation developed and built 100 years earlier to support the quality and speed necessary for their military during the Napoleonic wars. Imagine watching a steam engine-driven system in 1808 that produced over 100 thousand blocks (pulleys) for the Navy. The Block Mills of Portsmouth proved that with an assembly line and machines just 10 men were made able to work as quickly as 100.

    More to the point you say “The trend is that we (in the U.S.) invent something new, create an industry around it….” That seems to skip right past the fact that most industries in the US were started by European immigrants based on European ideas in place for many years before the US copied them. From Budweiser to Champagne, Cheddar Cheese to Chandeliers, what the US has really done well is bring down the price of goods and make them more accessible. In fact, that was an obsessive element to the Nixon administration that success would be determined entirely by the availability of goods. A steak on every table. And it’s true our shelves were stocked our pantries full while others in the world were still paying more for fewer goods, but somewhere in that heady explosion of prosperity out of the 1900s the US lost its sight of quality as a measure of success in “efficiency”.

    You bought that Suburban, you said, because you perceived value, right? Did you feel like you were buying innovation? Quality? Maybe a trip to a car show to look at the latest models (all outside the US now) will change your perspective:

    http://www.nytimes.com/2011/09/11/automobiles/as-frankfurt-show-opens-industrys-balance-shifts.html

    “If it seems as though German manufacturers are on the leading edge of new, gas-free urban transportation solutions, it is due in no small part to the European Union’s strict pollution controls. ‘Today, all the innovation in the auto industry is coming from the German manufacturers…A little from Japan. None from the U.S.'”

    NONE from the US. Our amazing ideas of “efficiency” apparently were not so.

    I mean a four-door all-wheel-drive station wagon made by Volvo is expected to be available next year that delivers better horsepower than a Ferrari 308 and a Camaro Z28, yet will also provide 100 mpg. That should have been an American made vehicle. No reason that it could not have been built and sold here. We have the weather, the open roads, the crap to haul around. Oh, no reason except people were for some reason still buying Suburbans. You know I could go on about this forever and someday I MAY convert you to a highly resilient low-risk source of energy for transportation, even if I have to do it on the mat…but I’ll try to get back to the point of your post.

    I think your definition of software may be too narrow. You say “software must be built by highly skilled people, whose skills are not trained up quickly or easily.” But isn’t that the very opposite of what is causing so many problems in code? Code is being written by many more people less trained and using toolkits. It is based on a massive rise in the amount of shared/borrowed/stolen code available. I see this most in recent cases of malware mutations — so many more people developing (or at least modifying) more code more rapidly than ever. The mobile app stores are another example. Anyone with a cheap personal computer and a few online tutorials now is in place to build and release software to hundreds of millions of users. Compare that to the training, samples and platforms of twenty years ago. Software is just flying off the wires now and it’s going to get even faster as more remote areas are connected.

    You say “those who profit by the billions from creating software, like Microsoft, Oracle, and Adobe seem unable to ship multi-million line software projects on a deadline”. You’re looking at the wrong sources of innovation. That’s like criticizing the British Navy for deploying ships late (a critique as old as the British Navy — special note to the Falkland’s War deployment, which led to the development of ITIL). While the Navy isn’t going away and will continue to find ways to automate production, they are solving massively complex problems. The future of software build efficiency is less about the big guys just like ship building an ocean-going vessel for the masses is at a much smaller scale today. The lessons learned from the big expensive mistakes are applied faster, better and at smaller scales of automation.

    So, I’d be one to argue yes, we know how not only to make software but hundreds of millions of people know how to save time by learning from the innovation of others — sharing knowledge and tools to reduce build times. I’d be happy to go more into the myths of commodity and innovation. I also would like to clarify trends and real numbers but I’ll leave those for another day (e.g. Today’s fastest growing telecom company? Skype is barely over 500 mil while India mobile is soon expected to have 1.2 billion subscribers). Alas, it’s time now to go make some more fuel for my engine.

  • Sumit Gupta

    This is really an answer to topic “the current approach to software development resembles more of a literary art rather than an engineering discipline.”

    Well said.