The state of the software development crisis: status quo

I found this comment on Reddit and it’s worth repeating. I don’t usually post someone else’s words as the entire content of my post. But this really sums up my feelings about the term “software development crisis”.
http://reddit.com/r/programming/info/6j1mf/comments/c03zf02

I’m sorry, but this is just nonsense.

People are forever coming along and proclaiming that software engineering is in a state of crisis. And always their reasoning is that (other) programmers aren’t smart enough.

They, of course, are special magical code gunslingers with superhuman intelligence, members of the top 5%. (Surely software engineering is blessed to have 90% of its practitioners located in the top 5%!)

But the truth is that there is no crisis, and there never has been. The only problem, and the reason software projects keep failing, is that of unrealistic expectations.

Software is hard. Really hard. This should not be surprising to anyone who understands that it is really the field of assembling instructions for doing… anything. Anything that people want to get done. It’s sort of a meta-field that encompasses almost all other fields, with more being added every day.

Of course it’s hard. The only real question is why people consistently underestimate its difficulty, especially why they underestimate the difficulty of any particular software engineering task.

I think there are a number of reasons for this:

1.

“It fits in the little box, how big can it be?” Humans, particularly those who aren’t technical, have a tendency to judge difficulty with their eyes.
2.

“It’s just a word processor.” Everyone understands what they are building, what the set of instructions is supposed to do, and they probably know how to do it by hand, albeit very slowly. They tend to assume that writing the software is just a matter of telling the computer to do the same thing, but faster. What they do not realize is that they don’t really know how their brain works at all, and that all the details which they can just leave to their giant neural net when doing it by hand, have to be figured out and brought consciously to the software.
3.

The Cult of Smart. Programmers, on the other hand, have figured out that software is complicated, and that being smart really helps; in fact, nothing is a substitute for it. This causes them to emphasize it, convince themselves that they have an abundant amount of it, and to convince themselves that results are not the result of them learning about problem domains, and building better and better versions iteratively, but just the inevitable consequence of bringing their enormous brain to bear. This leads to things like the “Agile Methodology”, as in “I’m so Agile I don’t need a Methodology.” Instead of realizing that you have throw one away (usually more like ten), they think they can be so magically smart they don’t have to.
4.

Expectations based on hardware. Chips are square. A linear decrease in CMOS transistor size results in a quadratic increase in the number that can be packed on a chip. Code is linear. A linear decrease in the amount of time it takes to produce X amount of code is merely a linear increase in the amount of code that can be produced. This results in an ever-widening design productivity gap, where capacity forever recedes away from our ability to exploit it. We can waste some of that excess capacity to save programmer time (this is why high-level languages have an expanding role in the field), but this is never going to go away. It isn’t that software is inherently blighted. It’s that hardware is inherently blessed.

  • http://www.lazycoder.com/weblog/ http://www.lazycoder.com/weblog/

    test openID

  • Pingback: Software Developers Never Change - Nick Berardi’s Coder Journal

  • Pingback: nothing happens :: echo chamber 14 may 2008

  • http://www.onagile.com Ryan Cooper

    This is for the most part a great post by the reddit commenter, but I did a double take at this part:

    “This causes them to… convince themselves that results are not the result of them… building better and better versions iteratively, but just the inevitable consequence of bringing their enormous brain to bear. This leads to things like the “Agile Methodology”, as in “I’m so Agile I don’t need a Methodology.” Instead of realizing that you have throw one away (usually more like ten), they think they can be so magically smart they don’t have to.”

    It saddens me that so many consultants, vendors, etc. out there hawking “Agile” solutions as a meaningless branding message, and disillusioning smart folks who share principles with us true agilists (but who don’t share our terminology).

    Real agile development is all about learning and iteration, and in my experience real agilists (as opposed to corporate sales people using ‘Agile’ as a buzzword) are a modest bunch, fully aware of their imperfection. Kent Beck, father of Test-Driven Development, has said he’s not particularly smart; he just has really good habits. True agile development embraces the fact that we screw up and can’t depend on our smarts alone to be successful. The premise of agile development is that we need rapid feedback mechanisms in every level and facet of software development, so we can learn and improve as fast as possible, because we *know* we won’t get it right the first time.

  • Pingback: Software Developers Never Change - Nick Berardi's Coder Journal