What's the point of software engineering?

Dec 11, 2009

Tom DeMarco is one of the founding fathers of modern software engineering. His 1987 book PeopleWare is a highly influential text in the field, and his ideas anticipated the Agile methodology which is now commonplace (common enough that it is now the dominant paradigm that new ideas must refine or rebel against). So when someone of his stature asks if software engineering is “An idea whose time has come and gone?”, something interesting must be happening.

That’s exactly what he asked in the August edition of IEEE Software magazine (PDF). DeMarco’s article takes the form of a critical look back at some of his most influential ideas, in particular the importance of measurement in assessing the performance of a software project. He takes issue with his earlier statement that “You can’t control what you can’t measure”, pointing out that some very successful projects have not been held back by a lack of precise measurement of their progress. This suggests that the focus on measurement and predictability that has been a basic assumption of software engineering may have been misguided.

Of course, Agile development has already moved us away from the idea of creating detailed delivery plans very early in the project lifecycle, but DeMarco’s conclusions go much further. He gives the example of two projects, both of which cost $1m, but project A will eventually yield value of $1.1m and project B will yield value of $50m. Project A must be tightly controlled, its scope narrowly defined and delivery must proceed according to a strict schedule. Even minor budget overruns will cut deeply into the return on investment to be gained. Project B, however, has no real need for such tight control - if the budget doubles, it matters relatively little so long as the project is delivered.

DeMarco’s epiphany is that if the project you’re working on is subject to tight control, or if as a project manager you are having to enforce tight controls, this strongly suggests that the project is of only marginal value to begin with. If a small budget overrun will obliterate the profit generated from your project, maybe the project isn’t such a good idea at all. And given the amount of attention that measurement and control generates within software engineering circles, it may be the case that most of the projects being worked on are of such little value that it is necessary to tightly control every aspect of their development, a difficult and costly task in itself.

DeMarco is not proposing anarchy in place of order, but is proposing that we - developers, project managers, analysts, CTOs and ultimately business decision makers - need to think differently about software projects. There remains vast unexploited potential for software to improve our lives and our businesses, and the challenge is to be bolder and more creative in identifying this, creating products that deliver sufficient return on investment to justify their existence even when allowing for the inevitable budget overrun. The current conservative culture of tight control is effectively a lie anyway, since budget estimates given at the start of a large project are unlikely to survive the collision with reality. It would, he argues, be better to admit this and to focus efforts on ensuring that the results make this worthwhile.

This struck a number of chords with me. It reminded me a little of my recent post Why winning is easer than finishing second in that DeMarco argues that if we find ourselves having to work very hard at something, it may be a sign that it’s just not meant to happen. In my post I talked about the huge effort that goes into failed attempts to pitch for projects compared to the relatively small effort that goes into winning projects where the client already has a preference for you; this mirrors the point about how the least valuable projects are the ones that require the most work.

It also reminded me a lot of user-centered design, at least in the way that we apply it at PRWD. When designing a user-centric system, one of our main considerations is to ensure that what we’re designing or developing delivers genuine value to users. If a feature of an application or website is unlikely to be used often, or is unlikely to add anything to the user experience, the cost of development is unlikely to be recouped. We prefer, wherever possible, to focus on projects where the return on investment is significant enough to justify doing things properly, rather than trying to squeeze schedules and budgets to make a marginal feature viable.