Fork me on GitHub

robknight.org.uk

Blog

Why software developers aren't just engineers

Posted on 16 January 2010 - 10:25am

Recently I wrote about Tom DeMarco's claim that software engineering is, in some sense, a 'dead' discipline. In many respects, I think he is right, especially in pointing out that a narrow focus on 'engineering' concerns is no longer appropriate. Engineering is about making something that carries out its stated purpose in an efficient and reliable manner, but it cannot tell us whether or not that purpose is a valuable one.

I think it's worth expanding on that insight a little further, particularly as it affects me in my job. I am, fundamentally, a coder by trade and my job still requires me to be able to write good code. Given that I've been writing code since primary school, I like to think that I'm now reasonably good at it. But when I was 18 and thinking ahead 10 years, I thought that being an ace coder was the only thing, professionally, that mattered, and it isn't.

Don't get me wrong, I still think that it matters a lot. Being able to write good code in one's chosen language(s) is a pre-requisite for anyone who takes him/herself seriously as a software developer. But the narrow conception of the software developer as a code monkey or savant hacker finding self-expression in the depths of a C++ program just doesn't seem to exist in reality.

Software development in 2010 is a much more multidisciplinary subject than the public perception of software development recognises. To be a good developer, you need to understand -

Psychology

If you want people to use your software, you need to understand how they relate to it. What makes software usable, intuitive, comfortable and reassuring for people, and what makes software confusing, irritating and unfriendly? What problem is a person trying to solve with your software and how can you make it easier for them?

Economics

Why should people buy or choose your software? What's the payoff? How to structure pricing and incentives is very important: obvious examples include mobile apps and subscription-based web2.0 services, but there are plenty more. Of course, economics goes a lot further than "how to sell your product" and a good grasp of economics will teach you much more: incentives, coordination problems, economies (and diseconomies) of scale and the benefits of specialisation and trade, for starters.

Business and organisations

There has never been such a thing as a self-contained "computer system", but it's now harder than ever to pretend that there is. Networks are everywhere, and software - your software - forms a small part of a chain that will include people, networks and other software. Where does it fit in and what problem does it solve? How does it co-exist with the rest of the system, human and machine?

Aesthetics

This is a somewhat controversial point, but I think that software needs to be beautiful to be truly successful. And I don't just mean that the code should be elegant (because often it isn't). A great work of art can be appreciated even by people who cannot comprehend how it was made, or by those who do not typically see beauty in such works. This doesn't mean that software needs to have eye candy, it just means that software needs to feel 'right' to the people using it in the way that a good painting or piece of music feels 'right' to the viewer or listener.

All of this can be summarised by saying that software must be useful. But there's a lot more that goes into making software useful than most people would commonly imagine, including most people whose job it is to make software. I once thought that it would be all about being able to optimise subroutines or create perfect data structures or choose the right design patterns, but software development now is all of those things and more.

This flies in the face of the stereotype of the software developer as someone who doesn't understand people and doesn't particularly want to. If such people really exist, then there's always going to be a place for the ultra-focused hacker who can code anything 10x faster and more reliably than his or her colleagues, but to progress from designing code to designing systems requires a different skillset. The challenge for my generation of software developers is to take what we know about code and computers and combine it with new knowledge and understanding of the world around us to make software that delivers benefits great enough to justify itself.

Footnote: In the title of this post I said that software developers are not "just" engineers. I do not mean this in the sense of "mere" engineers; I mean that engineering is one specialism within software development and not the only one. Software development is a team game and teams need good engineers as much as they need good architects and project managers.

Clouds and commodities

Posted on 3 January 2010 - 9:58pm

I've been following Sean Park's blog for about a year now, and have been finding his insights to be very interesting. His latest post, on Amazon Web Services, is typical in that he connects several different innovations to speculate about the future. What follows is my response to the ideas and issues he raises.

The latest development is Amazon's launch of 'spot' pricing for EC2 instances. In short, this means that the price for computing time can vary over time and customers can bid for that time. If their bid is equal or higher to the current spot price, they get their time. If it's lower, they don't - their currently running instances are shut down until they increase their bid or the spot price falls below their current bid. The customer's bid is only a maximum that they would be willing to pay, so if the spot price is lower than the bid then the customer only pays the spot price. This should enable cost savings for AWS customers who don't care when their processing happens, possibly at the cost of those who need it now.

As Sean points out, this is only the first baby-step towards something that we might recognise as a market. The spot price is apparently arbitrarily determined by Amazon, presumably based on their own algorithms used to monitor the capacity available on EC2. I would argue that there is a competitive market in cloud computing capacity at present, but it is a considerably less fluid market than it will be in years to come. What I think Sean and others have in mind for the future is a fluid market in which prices shift constantly in response to supply and demand across many different cloud providers, with real-time decisions being made - probably automatically - to re-allocate computing tasks in response.

The current market in computing power is impeded by technological costs - it's not always easy to move a computing task from one cloud provider to another. The different cloud providers have sufficiently different products - in terms of SLAs, connectivity, bandwidth, processing power and other available resources - that it's not trivial to determine which is offering the best value. For example, some computing tasks depend on having low-latency network access, which makes geographical and network location important. This makes commodification non-trivial (and can mean that sometimes, "a compute cycle is a compute cycle is a compute cycle" is not true).

I suspect that the market in computing power in the future will not move towards the idea of a single commodity or even a small number of commodity products. There will be many products, differentiated by a range of technical factors. As cloud computing matures, the needs of customers will become more varied, and this will mean that some products may be unsuitable for certain needs. It's not hard to imagine how some customers may place a premium on security, others on network latency, others on available system RAM and others on energy efficiency - and, of course, on price. Although any of the available products could 'get the job done', there will be big differences in suitability.

So, that's my guess. There are, I suppose, some reasons why I might be wrong. Perhaps the infrastructure required to discover and access the most suitable computing resource will be too expensive. We might be better off with a simple market with less choice, where computing time is packaged in standard units: at present, Amazon offers 21 products - seven sizes of EC2 instance multiplied by three available locations. We might imagine that competitors will have to mirror this structure and will offer a similar geographical breakdown in order to compete on a like-for-like basis with EC2. Perhaps customer needs will be less varied than I imagine, and the majority of demand can be met by the supply of a small number of commodity products delivered with high efficiency.

But if I'm right, then we're going to have a very complex market. Each customer will have unique requirements and will need to be able to explore the available products based on their suitability according to mutiple criteria. Provisioning across a wide variety of platforms will be a considerable technological challenge (though CohesiveFT seem to be doing good work here) and there is definitely scope for the development of a complex exchange (or multiple exchanges?) where bidding for computing time can take place, with appropriate agencies in place to handle provisioning once bids are accepted.

The ultimate aim should be to make the workings of this system entirely transparent to the people using it. Say I'm looking at pictures I've taken on an iPhone and I want to stitch these together with pictures taken at similar locations by other people to generate a 3D walkthrough of a particular locale - a fairly intensive, though technically feasible, task. For this I will need a certain amount of processing time and RAM, and a certain amount of bandwidth and data transfer. Assuming (with some magic involved, I admit) that my software can create an accurate estimate of my needs, it should be able to automatically go to the exchange and locate the cheapest provider that satisfies the requirements (a VRM system!) and automatically provision the required computing time. A short while later, I'm browsing a 3D landscape on my phone.

The uses for corporate customers are probably where the real value lies, but when individual consumers have seamless access to computing power in this way then we will know that we have a fully functioning and fluid market.

What's the point of software engineering?

Posted on 11 December 2009 - 10:51pm

Tom DeMarco is one of the founding fathers of modern software engineering. His 1987 book PeopleWare is a highly influential text in the field, and his ideas anticipated the Agile methodology which is now commonplace (common enough that it is now the dominant paradigm that new ideas must refine or rebel against). So when someone of his stature asks if software engineering is "An idea whose time has come and gone?", something interesting must be happening.

That's exactly what he asked in the August edition of IEEE Software magazine (PDF). DeMarco's article takes the form of a critical look back at some of his most influential ideas, in particular the importance of measurement in assessing the performance of a software project. He takes issue with his earlier statement that "You can’t control what you can’t measure", pointing out that some very successful projects have not been held back by a lack of precise measurement of their progress. This suggests that the focus on measurement and predictability that has been a basic assumption of software engineering may have been misguided.

Of course, Agile development has already moved us away from the idea of creating detailed delivery plans very early in the project lifecycle, but DeMarco's conclusions go much further. He gives the example of two projects, both of which cost $1m, but project A will eventually yield value of $1.1m and project B will yield value of $50m. Project A must be tightly controlled, its scope narrowly defined and delivery must proceed according to a strict schedule. Even minor budget overruns will cut deeply into the return on investment to be gained. Project B, however, has no real need for such tight control - if the budget doubles, it matters relatively little so long as the project is delivered.

DeMarco's epiphany is that if the project you're working on is subject to tight control, or if as a project manager you are having to enforce tight controls, this strongly suggests that the project is of only marginal value to begin with. If a small budget overrun will obliterate the profit generated from your project, maybe the project isn't such a good idea at all. And given the amount of attention that measurement and control generates within software engineering circles, it may be the case that most of the projects being worked on are of such little value that it is necessary to tightly control every aspect of their development, a difficult and costly task in itself.

DeMarco is not proposing anarchy in place of order, but is proposing that we - developers, project managers, analysts, CTOs and ultimately business decision makers - need to think differently about software projects. There remains vast unexploited potential for software to improve our lives and our businesses, and the challenge is to be bolder and more creative in identifying this, creating products that deliver sufficient return on investment to justify their existence even when allowing for the inevitable budget overrun. The current conservative culture of tight control is effectively a lie anyway, since budget estimates given at the start of a large project are unlikely to survive the collision with reality. It would, he argues, be better to admit this and to focus efforts on ensuring that the results make this worthwhile.

This struck a number of chords with me. It reminded me a little of my recent post Why winning is easer than finishing second in that DeMarco argues that if we find ourselves having to work very hard at something, it may be a sign that it's just not meant to happen. In my post I talked about the huge effort that goes into failed attempts to pitch for projects compared to the relatively small effort that goes into winning projects where the client already has a preference for you; this mirrors the point about how the least valuable projects are the ones that require the most work.

It also reminded me a lot of user-centered design, at least in the way that we apply it at PRWD. When designing a user-centric system, one of our main considerations is to ensure that what we're designing or developing delivers genuine value to users. If a feature of an application or website is unlikely to be used often, or is unlikely to add anything to the user experience, the cost of development is unlikely to be recouped. We prefer, wherever possible, to focus on projects where the return on investment is significant enough to justify doing things properly, rather than trying to squeeze schedules and budgets to make a marginal feature viable.

Ah, so that's why I hardly ever get comments...

Posted on 11 December 2009 - 8:55pm

If one person complains that my spam filter rejected their comment, I will mumble some vague apology and hope that they go away and stop bugging me about it. But when several people point out that my spam filter is repeatedly rejecting their perfectly normal comments, it's time to do something about it.

I had high hopes for Mollom, mostly because I really like its founder Dries Buytaert (who founded the Drupal project that powers this site and many others), but whilst it does an admirable job of keeping spam out, it does a less than admirable job of allowing genuine commenters in. I've now replaced it with TypePad AntiSpam, at least for a trial period of a week or two. So if you have been itching to comment but haven't been able to, now is the time!

Peer-to-peer, private and utility

Posted on 10 December 2009 - 9:01pm

This is going to be a slightly more speculative post than is usual. I'd like to explore three distinct models of delivery for services and goods. This is all pretty uninformed, as I don't really know anything about economics bar what a nasty Google Reader habit will leave you with. With that caveat in place, here goes...

It strikes me that many of the important goods we utilise as the building blocks of modern civilisation can, broadly, be managed in three different ways: as private concerns, as utilities, and as peer-to-peer goods. To sketch a rough example, electricity began with private power generation, progressed to utility provision by massive-scale entities (often state-owned or regulated) and may now be transitioning to small-scale and, in part, peer-to-peer generation using solar panels and wind farms (Nick Carr writes interestingly about this; I really must buy his book(s)). I am, of course, grossly over-simplifying, but I hope that I am catching some essential characteristic of the different approaches.

Money is another example. At times, money has been a private thing; access to it was limited largely to those who already had it. Money - as coins and notes - evolved as a representation of pre-existing private wealth. Over time, the system of goldsmiths issuing promissory notes in return for deposits of gold gave way to currency and fractional reserve banking, which eventually mutated into the utility model of mass banking we have today. Now, barely anyone deals with the local goldsmith or issues their own promissory notes and the stock of gold in the Bank of England or Fort Knox is a relatively small quantity of the wealth denominated in pounds or dollars. We interact with financial utilities, massive organisations that are intended, through their sheer size and scale, to deliver efficient supplies of capital to those who can make the best use of it. Of course, the rise of utility banking has not totally eclipsed what went before it, and this somewhat schizophrenic quality of banks remains part of the problem with them and explains why some want to split investment banking off from retail utility banking. Again, I have simplified here but I hope that I have captured some essential truth.

Computing power has actually undergone several transitions, but the major trend of recent years has been away from the idea of the private reserve of computing power - having your own servers and desktop PCs - to utility computing, where servers are provisioned dynamically from the cloud, and the computing power on the desktop is only really used to power the web browser. Seti@Home provides a glimpse at what a future peer-to-peer computing network might look like, and the forward-looking vision of those who created the internet on peer-to-peer principles should smooth the way.

It's important to note here that I do regard this as being a general trend of private to utility to peer-to-peer, but it doesn't happen all at once and each subsequent stage will not completely displace the former. There are still plenty of, say, private electricity generators in operation, they're just not a very big slice of the pie any more.

Might marketing data also follow a similar trajectory? Since time immemorial, vendors have held 'marketing data' about their customers simply by virtue of personal relationships. Shopowners would know the needs of their customers. In the 20th century, with the increasing ubiquity of large corporations, private data-gathering operations took hold. "Market research" was invented and the efficient management of information about customers and their observed needs became an important part of business. This process is still continuing, as the decrease in technology costs and the increase in opportunities to capture data about customer behaviour have led to the creation of vast private data repositories in our largest corporations. Many of the the most successful companies in business today claim that it is this deep understanding of customer needs that has driven their success.

Of course, there is a cost attached. To amass this data, store it, process it and analyse it is expensive. What's more, the benefit of gathering data increases as you gather more of it. Data about two million people's shopping habits is more than twice as valuable as data about one million people's. Once you reach the scale of Tesco (whose ClubCard loyalty scheme gives them personalised shopping history for millions of people; £1 in every £8 spent in British shops is spent there) the data is of huge importance. The investment required in gathering that amount of data is so immense that it represents a barrier to entry far bigger than the cost of acquiring premises or setting up supply lines, and control of the data is vital to the defence of a strong commercial position.

Data, as it is held now, is a classic example of a private good. It is hoarded at great cost by the few who guard it jealously, with the cost of acquiring data of their own sufficing to hold down the competition. There are, however, alternatives. The most obvious is Google, which holds a similarly huge amount of information about individuals, but has no desire to sell products to them directly. Google is happy enough to allow other retailers to piggyback on their database via advertising. Google is, essentially, a utility provider of data about customer buying intentions. For retailers, it's not as good as having your own data-mining operation, but it is dramatically cheaper. As a retailer, you only pay for what you use - on a per-click basis - and Google takes care of figuring out whether or not a person might want to buy your product. Google is not without competitors though: Facebook has a different kind of personal data about its users, and will also provide access to this data to others, for a price. For smaller businesses, this could be a boon as it erodes the advantage that mega-retailers have from their superior data. But, of course, the real winners are the utility data providers - Google and Facebook for now, though I suspect that credit card companies could get in on the act, and maybe even banks looking for a new business model could appear on the scene.

But what about peer-to-peer data? This is where VRM comes in. VRM offers the possibility that individuals will control their own personal data - their identity, their observable characteristics and attributes, and the data trail they leave from their interactions with others - and will be able to choose to share it with whomever they like. They can share it with other people, with small companies, large companies or nobody at all. In this sense, it represents something close to peer-to-peer exchange, in the sense that everyone 'creates' their own data and nobody has privileged access to it.

If you're interested in reading more of my thoughts on VRM, here are a few old posts of mine to get started with: This is my first post on VRM, when I was just figuring things out. VRM: Rent don't buy explores some of the concepts I mentioned in this post and What's a consumer? considers how a VRM world might change the relationship between 'consumers' and vendors.

It would be interesting to consider other goods which have undergone similar shifts. The written word, perhaps - from narrow, private creation by the few, to mass consumption (the printing press) to mass creation (blogs, wikis etc.). Is this a pattern that is far more fundamental than I realise?

A peer-to-peer future for money and markets?

Posted on 9 December 2009 - 8:50pm

As the recession grinds on, the question of "what to do about the banks" remains on the agenda. Today the government announced their latest maneouvre, a 'windfall' tax on bonuses paid to banking staff based in Britain. The BBC's Robert Peston looks at the possible effects of this move, including the possibility that by discouraging bonuses, the government will cause the banks to instead build up their capital and, as a result, be in a better position to lend and kick-start economic growth. I don't know enough about economics to know if this is right, but I'm skeptical.

As the "great financial crisis" has worn on, I've become ever less interested in how to 'fix' the banks, at least by using any of the measures that our politicians will entertain. Taxing bonuses does nothing to address the underlying problems and it does nothing to deal with the fact that banks, despite banking being a mature and, in theory, competitive industry, are still generating huge (though very volatile) profits, against what economic theory tells us should happen. If the banks can be fixed then I'll have to hope that one day, the people with the right ideas to do so take over, but I'm not holding my breath waiting.

So, what's an ordinary person to do? For me, a far more attractive proposition than hand-wringing over someone else's bonus payments is to look at alternatives to traditional banking. One such option is peer-to-peer lending, which operates on the simple basis of people lending their savings to other people who need to borrow. The interest rates are generally profitable enough to make it worth the lender's risk, but also generally cheaper than borrowing an equivalent amount on the same terms from a bank. The lack of middlemen taking bonus payments helps!

I suspect that the future will involve a lot more of this kind of thing. The continuing reduction in the costs of communication and the increasing ease of financial transfers means that we don't really need banks for many of the things that they have traditionally monopolised. I'm reminded of this post by Sean Park which suggests that the business models that some banks have relied upon simply don't work any more. Banks as great central clearing-houses of money, able to control who gets it, when, and for what price, may soon be a thing of the past.

Another interesting contribution to this debate came from Melody Hildebrant, in her O'Reilly Ignite talk entitled "Web capitalism doesn't need a bailout", which covers peer-to-peer ecommerce and lending, prediction markets and crowdsourcing (all some of my favourite things; if she had covered VRM too then it would have been perfect!). It's worth watching as an introduction to the idea that the future world might involve fewer "too big to fail" institutions and corporations, and more peer-to-peer interaction. On a more personal note, having just bought a house and with a need to spend some cash on home improvements, I'll probably be looking at Zopa for my borrowing needs rather than a bank.

Recent work - Drupal, Drupal, Drupal

Posted on 8 December 2009 - 5:33pm

I've been pretty busy lately with several projects, most of which are developed on Drupal. One, in particular, has been really exciting to work on. The great thing about working in the web industry is that your clients are often varied and interesting people, and this is a good example.

The most recent major project was for a new public policy think tank called ResPublica. I won't pass judgement on their politics, except to say that the people involved are all committed, intelligent and genuinely hoping to make a positive difference to the world. I've already blogged about the purpose of the site and written a case study on Drupal.org, but I'd like to go into a bit more detail about the process of building and deploying sites like this.

Drupal's new essential modules

There has always been a class of Drupal modules that were 'essential' for most sites despite not being part of Drupal core. In Drupal 5, the essential modules included Views, CCK and often an image-handling module like ImageCache. Google Analytics, XML Sitemap and others were useful, but for me Views + CCK were the two modules that I would always use on every project. Drupal 6 suffered greatly from the fact that these modules were not 100% ready with upgrades when Drupal 6 launched - a problem that has been largely solved for the upcoming Drupal 7.

Since Drupal 6 was released, a few new modules have made it on to the 'essential' list. The first is 'Features', which integrates with CCK, Views, Context and other modules to make it easy to package your settings into code in a module. This means that you can create Views and Content Types on your development server, save them as a 'Feature' (which is a Drupal module with some extra hooks) and then install that module on your production server. Voila! Your settings appear on the production server instantly. No need to manually export/import anything. This radically simplifies deployment and development scenarios.

Even better, because Features are stored in code and not in the database, there are far fewer occasions when you might need to take copies of database tables. If you wipe your database, you still have all of your Views/Content Types/etc. And to add icing to the cake, storing these settings in code means that they can be committed to source code control (I use git, but any version control system would be suitable). With this, you can have a full version history of your major Drupal configuration options, able to roll back to old versions or compare old with new.

Features was created by the guys at Development Seed, and they're still developing new, ahem, features which make this even more exciting. One of these is the 'Feature Server' which provides an easy way to distribute Features that you have created. For example, if you have created an image gallery using CCK, ImageField and Views, you can package the settings up as a Feature (which will automatically have the correct dependencies) and distribute it via your Feature Server. Sites using your Feature will check the Feature Server for updates to the Feature, notifying admins of new versions just like new versions of modules on Drupal.org.

One of the modules that I mentioned integrating with Features earlier was the Context module. Context solves the frustrating problem of getting Drupal to display the correct blocks of content in sidebars, footers and headers around particular pages. In the traditional block configuration page, which hasn't changed much since I first started using Drupal four years ago, you can optionally enable/disable the block based on user permissions and on a match with the URL path. This works well if you have a simple URL structure on your site, but can quickly become troublesome on sites with complex URL structures. And heaven help you if you change your URL structure - all of your blocks will need to be updated too. If you want to use any other information to control block visibility, your only other option is to write a PHP snippet to control it, or install one or more custom modules.

Context solves this by creating a simple UI for selecting from a wide range of criteria to control block visibility. You can select criteria such as node content type, location in the menu trail and the traditional URL path. Context can also be used to disable whole regions (for example, disabling a sidebar on the home page) and to set custom theme variables. It's a far superior way to control aspects of your site's appearance based on context, and I expect that it will develop further to allow for more fine-grained control.

My other choice of essential module is, in fact, not a module at all. Drush is a command-line script which can be used to perform many common administrative tasks from a shell prompt, rather than via the web interface. It can also do a few things that the web interface can't (without custom modules), such as downloading modules, clearing the cache and taking a backup of your database. For a developer, this speeds up common tasks considerably and, in particular, makes setting up new sites incredibly straightforward.

There's lots more that I would like to mention, including drush_make and the Aegir hosting platform, but I'll have to return to them another time (if you want to know why these are a big deal, read this post).

Why winning is easier than finishing second

Posted on 21 November 2009 - 12:00am

Over the last 18 months at work, I've taken on certain duties that, as a software developer, were new to me. Whilst it's hard to work as a software developer without having at least some exposure to client meetings, specifications and project scoping, my more recent experience has included writing proposals, giving presentations and generally taking a more active role in sales. Some of the things that I have learned have surprised me.

Looking back over a range of projects, some that we won and others that we lost, my most striking conclusion is that the amount of effort expended on the ones we lost, on average, substantially exceeded the effort expended on the projects that we won. Finishing second somehow takes a lot more work than finishing first.

I have some ideas about why this might be. For any given project, you have starting odds of winning; these might be based on your portfolio, existing relationships, ability to charge low rates or your sales acumen. Before a single word is written in a proposal, there's already some competitors better placed than others. And in many cases, the company that's best placed at the beginning goes on to win. Sometimes the favourite is so far ahead that there's no point in trying to beat them.

But it can be quite difficult to know that this is the case. It's very tempting to think that if you create the best proposal possible, give a great presentation to the client and generally bend over backwards to satisfy their every whim, you'll win the contract. In truth, if you're having to make that kind of effort it's probably a sign that you're not going to win. If you feel that the only way to be in contention for a contract is to spend large amounts of (unpaid) time trying to persuade the client to award you that contract, you're almost certainly better off doing something else.

Now, this might sound a bit defeatist and depressing. But it really isn't. The solution isn't simply to give up on any hope of making sales. The solution is to sell via your reputation, such that the potential clients you deal with already know who you are, what your strengths are, and have pre-screened you for inclusion in pitching processes. Instead of being one of the "also ran" companies there to make up the numbers when someone else is the favourite, the goal should be to *be* the favourite. Some of our most interesting and most rewarding work has come from sales where there wasn't even any other agency in the picture, where the client knew what they wanted and knew that they wanted us to deliver it.

This is where, at least in a web development context, open source software really helps. Open source projects generally have good communities around them, making it easy to talk about your projects and promote yourself through your work. Publishing modules or case studies, or giving talks to user groups, are great ways of establishing credibility. Being known for being great at something means that clients come to you when they know that that's what they want.

Drupal and the usability question

Posted on 15 November 2009 - 7:08pm

A few things recently have prompted me to think about the difference between how we pursue the goals of usability and the goals of good software development. In particular, how do we deal with the fact that the mindsets of professionals in both disciplines are often different?

The main prompt was this post by Larry Garfield. He's a long-time Drupal contributor and his lengthy post covers various topics, including the Drupal 'smallcore' movement and the D7UX project designed to improve usability in the upcoming Drupal 7 release.

His first major point is a criticism of the way 'smallcore' if often discussed, arguing that the focus on 'smallness' is misleading as the true goal should be flexibility. I agree with this, but this post isn't about smallcore, so I'll leave it at that. What's important about this observation is that, amongst software developers, there's a great concern to avoid making software that is 'bloated' or 'inflexible'. There's barely any disagreement about this topic: given that a piece of software like Drupal must serve as a platform for many other developments, it's necessary that it maintains the capability to be customised for many different purposes.

What follows in Larry's post is a discussion of the different philosophical approaches one might take to build a flexible CMS/framework. This is the sort of debate that occurs very often in software development. There are hundreds of books, each of which running into hundreds of pages, detailing different philosophical approaches to building software. Each concept for software design is expanded upon until, eventually, we have so many competing concepts that it becomes necessary to create yet another conceptual model that attempts to go 'back to basics' and simplify things (I'd argue that Domain Driven Design is an attempt to take Object-Oriented Programming back to its roots, for example).

My point is that software developers are always seeking for a way of building software that is simple, elegant and definitive. We never quite find one that we're happy with, but what we're looking for is a system that reduces - or eliminates - the need for repetitive work. Software developers are lazy - and that's why they're good at their jobs. Software should be as complex as necessary and as simple as possible. Keep it simple, stupid, and don't repeat yourself. Simple, reusable software can be tested for bugs once and re-used many times without that expensive effort having to be repeated. When we want to change something, we change it in one place and the same change automatically applies elsewhere. A lot of effort goes into making things that simple.

This brings me on to the second part of Larry's post, about the Drupal 7 User Experience project (D7UX). This is intended to improve the usability of Drupal, a very necessary aim as Drupal is now competing on many fronts with both corporate enterprise software and lightweight, single-purpose content management software.

Tellingly, Larry's criticism of the D7UX project is this:

The D7UX project was focused on a particular class of CMS, and most of the code that came out of it was in the form of large single-purpose Lego blocks.

In other words, the usability improvements made were insufficiently re-usable, too focused on solving narrow problems. Given how developers think, these conclusions are not surprising. Larry links to Leisa Reichelt, the usability lead on the D7UX project, who concludes that:

From the very outset our goal with D7UX was to make Drupal more accessible to people outside of the Drupal community and less technical people – people who didn’t even know what PHP was let alone how to code it.
...
This approach is representative of the goal to make Drupal more of a ‘product’ – an out of the box CMS solution that non-technical users can drive.
...
The changes that are required to the interface to really achieve the goal that we were tasked with ... has the consequence of making Drupal a less efficient and enjoyable place for Drupal developers to build cool stuff.
...
And so we have this tension. Drupal as a ‘Consumer Product’ and Drupal as a ‘Developer Framework’. Currently, the official direction is that the project is going to attempt to be both. I think this is a serious problem.

I think that this is the classic technology vs. usability problem. Developers want to create things that solve a wide range of problems at once, including problems we don't even know that we have yet. Usability professionals think more in terms of concrete use cases and user personas. Getting the two sides to agree about how a piece of software like Drupal should work is very difficult. Larry quotes a discussion:

Bojhan, one of our two new UX leads (woohoo!), commented:

Yet in most discussions, developers assume that design proposals are ill-informed – even if the design proposals truly are informed (by user observations and knowledge about Drupal inner workings)

To which Jeff Eaton replied:

And designers, in most discussions, assume that developers are visually illiterate and uneducated about UX matters. It’s a double-edged sword.

In January this year I gave a presentation to Northern User Experience about the different approaches that developers and UX people bring to their roles. For me, the difference boils down to the fact that UX people think in specifics, and developers think in the abstract. The D7UX project seems to have suffered from some of the problems I outlined, with misunderstanding and suspicion creeping in as both sides struggle to understand the other's aims.

So what's the solution? Personally, I think that Leisa Reichelt is correct, that it is likely to be impossible to reconcile the needs of all of Drupal's user groups in a single project. My perspective on this comes from working at PRWD, as a software developer at a usability-led company. I've learned a tremendous respect for usability, but it's important to remember that usability means thinking in specifics. It's impossible for a single centrally-run project to consider all of the specific scenarios in which Drupal might be deployed, or even to identify a very common use case.

What would be better would be to accept that Drupal is never going to be an off-the-shelf product. This does not have to mean the end of the ambition for Drupal to be everywhere on the web. Larry makes an analogy with Firefox, but my analogy is with Linux: there is a core of software called the 'Linux kernel' which is used in thousands of high-powered web servers, data-crunching applications and supercomputers, but is also used on Netbooks and even mobile phones. It would be wrong to call Linux a 'product', because nobody uses 'pure' Linux. Ubuntu, Fedora, CentOS and Slackware are products that people use. Some of these Linux distributions are built on top of each other - Ubuntu is based on Debian, for example. Ubuntu is friendlier and easier to use, but it still wouldn't exist without Debian. Likewise, we're going to have to build new distributions of Drupal and *that* is the correct level on which to consider usability. Development Seed have produced a great example of this with their OpenAtrium intranet software - it's built on Drupal, but the user experience is far better than an out-of-the-box Drupal installation. And the reason that it is better is that it has been designed for intranet use. Those same usability improvements wouldn't work for a Drupal e-commerce site, or a Drupal social network.

As strong as the Drupal brand is, I think that the future might consist of separately-branded products targeting Drupal at specific use cases. The underlying framework will be the same, and that framework will ensure module compatibility across different distributions. But usability, apart from the most generic of usability considerations, will be the responsibility of these different products to consider.

Magento shipping module update

Posted on 14 November 2009 - 4:09pm

Aside from working with Drupal, another recent development at PRWD has been an update to our Auto-shipping Module for Magento. This module works by automatically applying the most suitable delivery cost to an order before the checkout process begins. This ensures that the customer sees the full cost of the order on the shopping basket page. This is important because nobody likes surprises during a checkout process, and unexpected costs can be a big cause of checkout abandonment.

The new updates to the module enable multi-store functionality, so you can set different default shipping destinations for different stores (e.g. where you have a different store for different languages). This makes the module considerably more flexible.

If you've used the module or have any suggestions for how it could be improved, let us know on the PRWD blog.