How should software companies be run?

Nov 08, 2014

Yesterday Nic Ferrier linked to a video of a conference talk by Erik Meijer, in which Erik claimed (amongst other things) that agile development is harmful and that software engineers should want to leave companies that practice it. I felt that Erik made some good points, some bad points and some points that I’m not sure I understood fully, but what’s clear is that he has divided opinion. On my timeline were people who variously agreed or disagreed with him, sometimes conditional on a re-statement of his position to remove some of the bolder claims for his preferred alternative. Nic is in the “more disagree than agree” camp and his response is here.

Me? I’m a precious snowflake who wants to be able to agree with some bits without accepting any of the bits that I didn’t like. Since you can’t really be a precious snowflake on Twitter and live to tell the tale, I’m going to have to explain what I mean here.

Most of the argument has been around the question of whether Erik was right or wrong in his criticism of “agile”. It seemed to me that he was just the latest in a long line of people complaining about specific agile practices, in situations where they have been applied thoughtlessly. This is a well-known problem and Nic’s post does a good job of explaining why this does not serve as a general critique of agile principles, though there are good reasons to engage critical thought when applying these practices in your own environment.

To me, all of this is much less interesting than the rest of the talk.

The agile rant was really just a preamble to establish Erik’s premise that software projects and companies are frequently mismanaged. In particular, he made several bold claims:

  • “Self-managing teams” are not self-managing, and they exist to fool people into thinking that they have more autonomy than they do.
  • Programmers should do their best work between 22 and 32 and should train and work like professional athletes during that time.
  • Management, correctly understood, consists largely of strategic direction-setting, and this is a problem that can be addressed using mathematical models (contrasted with fuzzier notions of corporate “leadership”).
  • Hierarchical management works best.
  • We should expect and maintain higher standards of knowledge for people who wish to be considered professional programmers.
  • Programmers should be more willing to spend money on tools to help them do their jobs.

If you read Nic’s post, you’ll see that he also summarised Erik’s arguments. I have summarised them more positively than he did, and I’ve left out some that I don’t feel I understand (if you think that Erik was just talking nonsense then you might think that I have done him a favour). I’ve also taken some information from Erik’s recent ACM paper, which is heavier on the organisational economics and lighter on the agile-bashing.

Of the above arguments, I would actually find relatively little to disagree with, the irony being that I come at this from an “agile” perspective.

Let’s just get the worst one out of the way - the notion that programmers should do their best work between 22 and 32, rather like professional soccer or NBA players. This strikes me as silly, because we could just as well make novelists or lawyers our reference class, and then talk about how programmers should do their best work in the latter half of their careers. I suspect Erik’s argument seems a bit more reasonable if you think, as he does, that the software invasion of other industries is set to intensify, increasing the flow of wealth to software companies.

He also contends that software companies could be run for the benefit of software developers to a greater extent than at present, in much the same way that European soccer clubs are (they differ from American sports franchises in their lack of maximum wage restrictions; almost nobody makes money from owning a European soccer club because all of the money ends up flowing to the players and their agents). In this light, he’s suggesting that software developers should able to retire at 32 because they’re paid enough money that they’re now rich and don’t need to work. As a 32-year-old software developer I approve of this message; in practice, I don’t think that the idea makes any sense without that somewhat improbable economic transformation.

On self-managing teams, meh. I think he may have a point, that autonomy works best when there’s true alignment of risk and reward, and true power to make one’s own decisions. Most self-managing teams, in my experience, are allowed to manage themselves within fairly narrow parameters. It’s better than micro-management, but that’s not saying much. But he’s speaking with the zeal of the convert here - as someone who worked for Microsoft for 14 years before setting up his own company, he’s really just saying that it’s a lot more fun to work for yourself. This is pretty well-known in the social science literature, but not everyone can work for themselves and agile notions like self-managing teams can help to make people both happier and more effective when working inside larger organisations.

His argument about professionalism - that developers should spend money on acquiring high quality tools, and that we should push back on the idea that professional-grade programming skills can be picked up in a few weeks with a book you bought for £20 - makes sense to me. Programming well is hard, and takes years of practice, although the basics of programming can be learned fairly easily by anyone who cares to try. Programming can also be fun, and doesn’t need to be considered solely for its economic value, but insofar as programming is being carried out professionally it makes sense to maintain high standards.

At long last, this brings me to the interesting bit - how software companies (and that’s really most companies) should be managed.

Erik’s view is that software companies don’t really need to produce much apart from software, citing the examples of Uber and Airbnb, which own software platforms but not taxis or hotels, despite competing in those industries. There are other more obvious examples - Microsoft, Google, Oracle - which are largely pure software companies, but Uber/Airbnb are interesting precisely because they’re software companies that aren’t selling software. This makes it plausible - let’s just go with this here - that the next decade might see many similar companies, formed around core teams of professional software developers, that build software platforms that can perform the functions of the internal bureaucracy of larger companies for a fraction of the cost, enabling them to be competitively successful and generate large returns. By “large returns”, we’re talking about enough for that core engineering team’s members to retire by 32. Erik’s view appears to be that the scope for such companies is wide enough that substantial numbers of software engineers could be employed in this way.

Well, partly he’s telling an audience of software developers that they deserve to be made rich for doing their jobs, which always goes down pretty well. What interests me is whether or not it’s feasible. And this is where he throws in an idea that doesn’t seem to fit: hierarchical management, modeled on the Army.

Literally, he suggests the Army as a role model for how these new software companies should be run. How can someone who complains that self-managing teams aren’t really self-managing enough suggest that the Army, a by-word for top-down command-and-control, as a role model?

Maybe I am being overly charitable, but this makes sense to me, based on some ideas I already had before listening to the talk. Firstly, Erik is suggesting that management is mostly about strategic goal-setting, and he believes that this can be modelled mathematically. The job of senior management would therefore be to develop and maintain these models, and the systems for feeding input to the models, not to simply use their authority to order people around.

Secondly, it turns out that 21st-century armies delegate a lot more than their predecessors did. The top-down-tactics of World War I have given way to the notion of “Commander’s intent”, where the senior officer will outline a strategic objective, but will delegate decisions about how to achieve it to small teams on the ground. In a practical example, we might imagine that the strategic model for an Airbnb-like company could identify the need to reduce booking cancellations, but the decision of what software to build (or even if software is necessary at all) in response to this problem is delegated to an engineering team. Management ensures alignment around common intent, but not much more than that.

As an idealised notion of how a company could operate, it sounds feasible to me. I’ve even seen this approach work on an individual project level; the approach I have seen which fits this picture is Impact Mapping, which I would consider to be an “agile”, or at least “agile-friendly”, technique. In Impact Mapping, we identify the Goal of a project, the Actors we want to affect, the Impacts we want to have on them, and finally some Features that might actually achieve them. The engineering team is responsible for achieving the impacts, and the features simply comprise a set of options for how to do so (not all of which will work, and not all of which will actually be needed).

This feels not dissimilar to Erik’s hierarchy, where the strategic goal-setting and market feedback-gathering happens at one level, and the detailed problem-solving happens at another level. In my view, seeing this as a “top-down” hierarchy is actually misleading; it should be regarded as a conversation between strategy and implementation. If we ditch the notion of seniority and rank and simply think in terms of a separation of concerns between strategy and implementation then Erik’s idea isn’t totally nuts.

To have all of this make sense to me, I’ve had to do a fair bit of editing of Erik’s ideas. I think that he’s seeing something similar to what I am seeing, albeit in a different vocubulary and with different emphases. I also think that he’s flat wrong on several of his specific prescriptions. But I’ve enjoyed writing this and so I’m glad that he gave the talk and stirred these ideas up.