robknight.org.uk

I've been pretty busy lately with several projects, most of which are developed on Drupal. One, in particular, has been really exciting to work on. The great thing about working in the web industry is that your clients are often varied and interesting people, and this is a good example.

The most recent major project was for a new public policy think tank called ResPublica. I won't pass judgement on their politics, except to say that the people involved are all committed, intelligent and genuinely hoping to make a positive difference to the world. I've already blogged about the purpose of the site and written a case study on Drupal.org, but I'd like to go into a bit more detail about the process of building and deploying sites like this.

Drupal's new essential modules

There has always been a class of Drupal modules that were 'essential' for most sites despite not being part of Drupal core. In Drupal 5, the essential modules included Views, CCK and often an image-handling module like ImageCache. Google Analytics, XML Sitemap and others were useful, but for me Views + CCK were the two modules that I would always use on every project. Drupal 6 suffered greatly from the fact that these modules were not 100% ready with upgrades when Drupal 6 launched - a problem that has been largely solved for the upcoming Drupal 7.

Since Drupal 6 was released, a few new modules have made it on to the 'essential' list. The first is 'Features', which integrates with CCK, Views, Context and other modules to make it easy to package your settings into code in a module. This means that you can create Views and Content Types on your development server, save them as a 'Feature' (which is a Drupal module with some extra hooks) and then install that module on your production server. Voila! Your settings appear on the production server instantly. No need to manually export/import anything. This radically simplifies deployment and development scenarios.

Even better, because Features are stored in code and not in the database, there are far fewer occasions when you might need to take copies of database tables. If you wipe your database, you still have all of your Views/Content Types/etc. And to add icing to the cake, storing these settings in code means that they can be committed to source code control (I use git, but any version control system would be suitable). With this, you can have a full version history of your major Drupal configuration options, able to roll back to old versions or compare old with new.

Features was created by the guys at Development Seed, and they're still developing new, ahem, features which make this even more exciting. One of these is the 'Feature Server' which provides an easy way to distribute Features that you have created. For example, if you have created an image gallery using CCK, ImageField and Views, you can package the settings up as a Feature (which will automatically have the correct dependencies) and distribute it via your Feature Server. Sites using your Feature will check the Feature Server for updates to the Feature, notifying admins of new versions just like new versions of modules on Drupal.org.

One of the modules that I mentioned integrating with Features earlier was the Context module. Context solves the frustrating problem of getting Drupal to display the correct blocks of content in sidebars, footers and headers around particular pages. In the traditional block configuration page, which hasn't changed much since I first started using Drupal four years ago, you can optionally enable/disable the block based on user permissions and on a match with the URL path. This works well if you have a simple URL structure on your site, but can quickly become troublesome on sites with complex URL structures. And heaven help you if you change your URL structure - all of your blocks will need to be updated too. If you want to use any other information to control block visibility, your only other option is to write a PHP snippet to control it, or install one or more custom modules.

Context solves this by creating a simple UI for selecting from a wide range of criteria to control block visibility. You can select criteria such as node content type, location in the menu trail and the traditional URL path. Context can also be used to disable whole regions (for example, disabling a sidebar on the home page) and to set custom theme variables. It's a far superior way to control aspects of your site's appearance based on context, and I expect that it will develop further to allow for more fine-grained control.

My other choice of essential module is, in fact, not a module at all. Drush is a command-line script which can be used to perform many common administrative tasks from a shell prompt, rather than via the web interface. It can also do a few things that the web interface can't (without custom modules), such as downloading modules, clearing the cache and taking a backup of your database. For a developer, this speeds up common tasks considerably and, in particular, makes setting up new sites incredibly straightforward.

There's lots more that I would like to mention, including drush_make and the Aegir hosting platform, but I'll have to return to them another time (if you want to know why these are a big deal, read this post).

A few things recently have prompted me to think about the difference between how we pursue the goals of usability and the goals of good software development. In particular, how do we deal with the fact that the mindsets of professionals in both disciplines are often different?

The main prompt was this post by Larry Garfield. He's a long-time Drupal contributor and his lengthy post covers various topics, including the Drupal 'smallcore' movement and the D7UX project designed to improve usability in the upcoming Drupal 7 release.

His first major point is a criticism of the way 'smallcore' if often discussed, arguing that the focus on 'smallness' is misleading as the true goal should be flexibility. I agree with this, but this post isn't about smallcore, so I'll leave it at that. What's important about this observation is that, amongst software developers, there's a great concern to avoid making software that is 'bloated' or 'inflexible'. There's barely any disagreement about this topic: given that a piece of software like Drupal must serve as a platform for many other developments, it's necessary that it maintains the capability to be customised for many different purposes.

What follows in Larry's post is a discussion of the different philosophical approaches one might take to build a flexible CMS/framework. This is the sort of debate that occurs very often in software development. There are hundreds of books, each of which running into hundreds of pages, detailing different philosophical approaches to building software. Each concept for software design is expanded upon until, eventually, we have so many competing concepts that it becomes necessary to create yet another conceptual model that attempts to go 'back to basics' and simplify things (I'd argue that Domain Driven Design is an attempt to take Object-Oriented Programming back to its roots, for example).

My point is that software developers are always seeking for a way of building software that is simple, elegant and definitive. We never quite find one that we're happy with, but what we're looking for is a system that reduces - or eliminates - the need for repetitive work. Software developers are lazy - and that's why they're good at their jobs. Software should be as complex as necessary and as simple as possible. Keep it simple, stupid, and don't repeat yourself. Simple, reusable software can be tested for bugs once and re-used many times without that expensive effort having to be repeated. When we want to change something, we change it in one place and the same change automatically applies elsewhere. A lot of effort goes into making things that simple.

This brings me on to the second part of Larry's post, about the Drupal 7 User Experience project (D7UX). This is intended to improve the usability of Drupal, a very necessary aim as Drupal is now competing on many fronts with both corporate enterprise software and lightweight, single-purpose content management software.

Tellingly, Larry's criticism of the D7UX project is this:

The D7UX project was focused on a particular class of CMS, and most of the code that came out of it was in the form of large single-purpose Lego blocks.

In other words, the usability improvements made were insufficiently re-usable, too focused on solving narrow problems. Given how developers think, these conclusions are not surprising. Larry links to Leisa Reichelt, the usability lead on the D7UX project, who concludes that:

From the very outset our goal with D7UX was to make Drupal more accessible to people outside of the Drupal community and less technical people – people who didn’t even know what PHP was let alone how to code it.
...
This approach is representative of the goal to make Drupal more of a ‘product’ – an out of the box CMS solution that non-technical users can drive.
...
The changes that are required to the interface to really achieve the goal that we were tasked with ... has the consequence of making Drupal a less efficient and enjoyable place for Drupal developers to build cool stuff.
...
And so we have this tension. Drupal as a ‘Consumer Product’ and Drupal as a ‘Developer Framework’. Currently, the official direction is that the project is going to attempt to be both. I think this is a serious problem.

I think that this is the classic technology vs. usability problem. Developers want to create things that solve a wide range of problems at once, including problems we don't even know that we have yet. Usability professionals think more in terms of concrete use cases and user personas. Getting the two sides to agree about how a piece of software like Drupal should work is very difficult. Larry quotes a discussion:

Bojhan, one of our two new UX leads (woohoo!), commented:

Yet in most discussions, developers assume that design proposals are ill-informed – even if the design proposals truly are informed (by user observations and knowledge about Drupal inner workings)

To which Jeff Eaton replied:

And designers, in most discussions, assume that developers are visually illiterate and uneducated about UX matters. It’s a double-edged sword.

In January this year I gave a presentation to Northern User Experience about the different approaches that developers and UX people bring to their roles. For me, the difference boils down to the fact that UX people think in specifics, and developers think in the abstract. The D7UX project seems to have suffered from some of the problems I outlined, with misunderstanding and suspicion creeping in as both sides struggle to understand the other's aims.

So what's the solution? Personally, I think that Leisa Reichelt is correct, that it is likely to be impossible to reconcile the needs of all of Drupal's user groups in a single project. My perspective on this comes from working at PRWD, as a software developer at a usability-led company. I've learned a tremendous respect for usability, but it's important to remember that usability means thinking in specifics. It's impossible for a single centrally-run project to consider all of the specific scenarios in which Drupal might be deployed, or even to identify a very common use case.

What would be better would be to accept that Drupal is never going to be an off-the-shelf product. This does not have to mean the end of the ambition for Drupal to be everywhere on the web. Larry makes an analogy with Firefox, but my analogy is with Linux: there is a core of software called the 'Linux kernel' which is used in thousands of high-powered web servers, data-crunching applications and supercomputers, but is also used on Netbooks and even mobile phones. It would be wrong to call Linux a 'product', because nobody uses 'pure' Linux. Ubuntu, Fedora, CentOS and Slackware are products that people use. Some of these Linux distributions are built on top of each other - Ubuntu is based on Debian, for example. Ubuntu is friendlier and easier to use, but it still wouldn't exist without Debian. Likewise, we're going to have to build new distributions of Drupal and *that* is the correct level on which to consider usability. Development Seed have produced a great example of this with their OpenAtrium intranet software - it's built on Drupal, but the user experience is far better than an out-of-the-box Drupal installation. And the reason that it is better is that it has been designed for intranet use. Those same usability improvements wouldn't work for a Drupal e-commerce site, or a Drupal social network.

As strong as the Drupal brand is, I think that the future might consist of separately-branded products targeting Drupal at specific use cases. The underlying framework will be the same, and that framework will ensure module compatibility across different distributions. But usability, apart from the most generic of usability considerations, will be the responsibility of these different products to consider.

At PRWD we've just completed our latest project, a website for the new public policy think-tank ResPublica. The site was built using Drupal, and you can read the drupal.org case study here.

This has been the first full Drupal site I've delivered for a while, and I've been getting to grips with some of the great new contributed modules that have appeared recently. In particular, the Features module has been a revelation - what it does is incredibly simple and yet also incredibly powerful. It's something that I can't imagine not using on every project.

The Features module solves one of the biggest problems with Drupal development and deployment: the fact that lots of structural information - in particular, content types, views and imagecache presets - gets stored in the database. To deploy a site from development to staging and then to live can involve either A) duplicating (in part) the database or B) manually exporting each view, content type etc. each time they are altered, and importing them into the relevant sites. There's no version control and mistakes are very easy to make.

Features solves all of this by packaging your stuff into a module. This automatically-generated module contains definitions of your views, content types etc., ready for installation on another site. So you might create a view on your development site, add it to a Feature, then just install that Feature on the staging server. Voila! Your view appears on the staging site. What's more, your Feature module can be kept under source code control, enabling you to manage it in precisely the same way that you would manage custom modules you've created yourself. If you change the view on your development server, just update your Feature module, check it in to your version control system, and check it out again on the staging server and you instantly get your new view.

This is a massive step forward for professional Drupal development. Once again I'm in awe of how great the Drupal community is at solving each other's problems. The fact that I'm able to take advantage of the fantastic work of other people - and hopefully I'll be contributing something back - is a testament to the power of open source and, in particular, the Drupal project. There's a post that I want to write at some point investigating why certain projects - the Linux kernel, Apache web server, Rails and of course Drupal - seem to hit that sweet spot of community engagement where the efforts of all of the people involved combine well to create something truly useful. Why do these projects succeed and others with similar aims and ambitions fail? For now, the only thing worth mentioning is that Drupal is definitely a project that is succeeding, perhaps better than any other open source CMS or any other PHP development framework.

This is the first post in a series on the subject of Domain-Driven Design and its use in PHP development.

If you're reading this, I'll assume that you already have a reasonable knowledge of PHP, although it's not essential. Before going any further, I'll explain the basic notion of what DDD is, and why you might want to care about it. Fortunately, I've already written a basic introduction to DDD in a response to a question on Stack Overflow, which I'll reproduce here:

Firstly, if you don't know that you need it then it's possible that you don't need it. If you don't recognize the problems that DDD solves then maybe you don't have those problems. Even DDD advocates will frequently point out that DDD is only intended for large (>6 month) projects.

Assuming that you're still reading at this point, my take on DDD is this:

DDD is about trying to make your software a model of a real-world system or process. In using DDD, you are meant to work closely with a domain expert who can explain how the real-world system works. For example, if you're developing a system that handles the placing of bets on horse races, your domain expert might be an experienced bookmaker.

Between yourself and the domain expert, you build a ubiquitous language, which is basically a conceptual description of the system. The idea is that you should be able to write down what the system does in a way that the domain expert can read it and verify that it is correct. In our betting example, the ubiquitous language would include the definition of words such as 'race', 'bet', 'odds' and so on.

The concepts described by the UL will form the basis of your object-oriented design. DDD provides some clear guidance on how your objects should interact, and helps you divide your objects into the following categories:

• Value objects, which represent a value that might have sub-parts (for example, a date may have a day, month and year)
• Entities, which are objects with identity. For example, each Customer object has its own identity, so we know that two customers with the same name are not the same customer
• Aggregate roots are objects that own other objects. This is a complex concept and works on the basis that there are some objects that don't make sense unless they have an owner. For example, an 'Order Line' object doesn't make sense without an 'Order' to belong to, so we say that the Order is the aggregate root, and Order Line objects can only be manipulated via methods in the Order object

DDD also recommends several patterns:

• Repository, a pattern for persistence (saving and loading your data, typically to/from a database)
• Factory, a pattern for object creation
• Service, a pattern for creating objects that manipulate your main domain objects without being a part of the domain themselves

Now, at this point I have to say that if you haven't heard of any of these things before, you shouldn't be trying to use DDD on any project that you have a deadline for. Before attempting DDD, you should be familiar with design patterns and enterprise design patterns. Knowing these makes DDD a lot easier to grasp. And, as mentioned above, there is a free introduction to DDD available from InfoQ (where you can also find talks about DDD).

So, DDD is an approach to system design that is useful on large projects that involve complex systems. DDD is great when you're dealing with a well-defined system and you have experts on hand to explain how that system needs to function. If you're writing software to automate factory processes, you need access to a shop floor manager who can tell you everything about how those processes operate. Not every project is like this. Sometimes you're creating software that's quite open-ended, where you're not sure exactly what the outcome is going to be at the end of the project. Whilst some DDD principles can help here, not all of it will be applicable. For small projects, the overhead of DDD is simply unnecessary - pragmatism may dictate that it's better to just get the job done rather than worry about the best possible design method.

As of now, DDD has gained a widespread following amongst 'enterprise' developers using .NET or Java, but isn't widely known in the PHP world. This is because the large-scale projects for which DDD is best suited are still relatively rare in PHP - they exist, but they exist alongside many more smaller projects that do not require DDD.

Recently, there have been some huge developments in the PHP world, which mean that large-scale projects are much more achievable than in the past, and PHP has found a new role in large-scale web projects that might previously have been done using Java or .NET. In his post on the subject, Cal Evans talks about how PHP is now being used by web innovators such as Digg and Facebook, as well as blue-chip corporates. PHP 5.3 has added some powerful new language features and has addressed issues of performance, stability and reliability. The Zend Framework provides implementations of many key design patterns which serve as the building blocks of both code and the design of code.

In summary, I think that the PHP community has a great opportunity to learn from other communities who have been solving enterprise problems for years. Concepts such as Domain-Driven Design can help us to make better software. In this series of posts, I'll cover DDD from a PHP perspective, explaining how it can be applied from the perspectives of design and coding.

The next post in this series will cover the groundwork, introducing the basic design patterns that are present in all DDD projects.