Announcing: Wisdom Hive

Rob Knight — Thu, 23 Oct 2008 10:48:48 -0400

Just a brief note to announce the public beginning of the Wisdom Hive project. It's a project to create a next-generation open source prediction market platform (and if that meant absolutely nothing to you, you might want to read the introductory post explaining some of the details).

I'll be posting a bit on the blog there and will probably be re-posting some of that material here. In the meantime, you can read my first 'proper' post (apart from the intro) on the Wisdom Hive blog: A bad time to believe in markets?

ThoughtWorks event on usability and agile tonight

Rob Knight — Thu, 23 Oct 2008 10:44:51 -0400

Just a quick note for anyone in the Manchester area - tonight there is an event on at the Kilburn Building featuring Luke Barrett of ThoughtWorks. It's a talk on usability in software design, with a focus on agile development techniques, something I mentioned in a recent post on the PRWD blog.

UCD and Agile

Rob Knight — Sun, 12 Oct 2008 04:33:32 -0400

I've got a new post up on the PRWD Blog: User-Centered Design and Agile Development.

IT Recruitment: The spam approach

Rob Knight — Tue, 24 Jun 2008 13:47:30 -0400

Most IT recruitment (the area I mention simply because it's the one that I have the most direct experience of) resembles spam. Recruitment agencies try to get as many CVs out as possible to as many companies as possible, and candidates also try to get their CV out to as many agencies as possible and, in turn, ensure that their CV is seen by as many prospective employers as possible. All sides want to have maximum choice, though arguably employers are in a better position to be picky than some employees are, although this is always dependent on situation.

The problem here is that employees will tend to try to cram as many 'keywords' as possible into their CVs, in an effort to match as many jobs as possible. After all, it's better to match too many jobs than too few, right? And it's always possible to turn down a job if it isn't quite right, so there's a strong incentive to list some skills that you don't really have in the hope that it gets you a foot in the door. In the best case this might enable some positive chance encounters to occur – perhaps a PHP developer listing Java amongst his skills might actually persuade his (new) employer to take up PHP – but the worst case is that people are put forward for jobs which they are totally unsuited for. The recruitment agency needs only to care about whether they can get someone – anyone! - into the job, which means trying to make candidates look as good as possible by burnishing their CVs.

Employers have a slightly different aim. They do want to see a wide field of potential recruits in many cases, but they are fundamentally more concerned with quality than quantity, and see quantity as only a means to the end of finding the right employee. However, determining quality can be hard, and employers might be tempted to do this by specifying a very long list of required skills, some of which will be of only tiny relevance to the job as a whole. We've all seen the gigantic laundry lists of IT/development skills which are so commonplace and so exhaustive that they easily could be (and probably are) simply copy-and-pasted by both parties in the recruitment process.

The problem here is a lack of weighting. Employers putting down a list of required skills often do not prioritise. They want a Java developer, so they certainly put 'Java' down. They might list a few other related technologies, like J2EE, Spring, Hibernate and so forth, or Linux, MySQL, XML and so forth. But are any of these really important? Will the developer even really be using Spring? Sure, knowing what Spring is is a good sign in a Java developer, but it's not a core skill. If the developer is merely aware of it, he might not put it down as a skill, even if he could learn it in a very short space of time. What the company really wants is a kick-ass Java developer who can work without too much oversight, but what they've ended up asking for is a walking software engineering encyclopedia. This process forces the potential recruits to pretend to be just that – this is doubly unfortunate as such a pretence is quite easy to maintain. It's not hard to blind people with science when you know a bit of technical jargon (I think I'm doing pretty well in this example for someone who has never written a line of Java).

And why not do this? There's no cost to simply listing as many buzz-words as you can think of, for either side. And therein lies the problem. A better system would force both sides to rank the skills they require – perhaps allocating a value to each, from a limited pool of available value, e.g. a percentage. That way, candidates can make claims about what their strongest skills are, and companies can place emphasis on the skills that they really need. There may even be circumstances where it's better to list one skill with a rating of 100% if that's what you really want to be doing in your job (or what you really want from your recruit). This would more accurately represent a 'marketplace' of skills, because it would be clear what skills are in demand and which are in supply.

To genuinely place the right people with the right companies, recruitment agencies need to think about the needs of both parties, and consider a wide range of variables in their assessment. When I was recruited for my current job, the process was handled by Psycuity, who did an excellent job in doing both aptitude and personality testing. Once a company is in a position where they have identified their candidates, this approach is perfect, and I'd highly recommend it. But where companies are at an earlier stage, just wanting to ensure that the right candidates know about the job, and the candidates want to make sure that they're considered for the right roles, most ordinary recruitment companies fall down. As a whole, the IT recruitment industry is barely functional, and is in need of some shaking up. Psycuity's approach certainly goes a long way, compared with the practices I have observed before, but perhaps there's potential for things to be even better? Having heard plenty of IT recruitment horror stories, I'd say that there's certainly a need for it.

Massive free educational resource from the OU

Rob Knight — Tue, 03 Jun 2008 18:08:47 -0400

I've recently been considering taking some extra qualifications, and one of my options is the Open University, mostly because it's something that I could do alongside work.

One thing that has been invaluable in examining the OU's courses has been the new(ish) OpenLearn site - essentially, this contains a selection of the OU's course materials for free. Given that this includes some post-graduate material, the courses for which can cost thousands of pounds, this is not to be underestimated. Even if you just want to brush up on a subject by reading some course material, this could be a very good place to start. The site also appears to have some rudimentary social networking features which, whilst they don't quite seem to have widely adopted yet, could form the basis of a 'learning community'.

It's hard for me to look at something like this and not see this as being a big part of the future of education. Knowledge workers such as IT professionals are already used to the idea of being largely self-taught, and the availability of high-quality materials in a variety of subjects can only promote this idea.

History of a programmer

Rob Knight — Mon, 07 Apr 2008 08:15:34 -0400

Reading a recent piece on Slashdot about the BBC Micro set off a wave of nostalgia which got me thinking about just how I got involved in computers in the first place. The BBC Micro was not the first computer I had used – we had a ZX Spectrum at home – but it was the first computer that I made a proper effort at programming.

My primary school had several BBC Micros. The intended purpose of the machines was never clear, though we did occasionally make use of the famous LOGO turtle. Whatever other uses we made of them have faded into the recesses of my memory; the one thing that remains crystal clear to me is the undirected time I had in which to play about with creating my own programs. I learned how to do this from a school friend, who knew some rudimentary BASIC. (And there you have it – I learned more about how to program computers in school from a nine-year-old boy than I ever did from a teacher). He taught me enough that I could create some very simple programs, most of which never did very much at all. I had a few attempts at creating what I would now describe as text-based adventure games but which were really just chunks of narrative separated by some IF statements.

I was hooked. The idea that I could control this box of electronic magic by giving it a set of commands fascinated me. The fact that, once running, a computer program essentially had a life of its own was a source of wonder and amazement to me. I spent far more time than I was supposed to at school inputting endless lines of BASIC code, just to see what I could cause to happen.

As I got older, I moved on to DOS and Qbasic, which was similar enough to the old BBC Micro BASIC that I could do all of the same things with it. But this was around 1995, and DOS was finally on the way out. I tried some Visual Basic but didn't particularly enjoy it. The sense (an illusion, but not a too gratuitous one) of having total control over the computer's operations was not present with Visual Basic's event-driven model. And I had picked up from somewhere that real programmers didn't use Visual Basic – real programmers used C. So, I picked up a C++ (I figured that I might as well learn C++) manual and began reading it.

I was still reading it a year later and wasn't any closer to understanding it. I played around a bit with Borland Turbo C++ - a retrograde step in that I was going back to DOS, but this was probably necessary; having to learn C++ and the Win32 API at the same time was probably too much for my 14-year-old brain. I soldiered on, but advanced little; I was now using printf() instead of PRINT, but that was it.

Then, one day, it all clicked into place. I suddenly got it and discovered just how amazing it feels to get it. I was now, irretrievably, a computer geek, but I was now also proud of it. I had taken on a proper programming language and I had won! The mysteries of pointers, references, operator overloading, memory allocation, polymorphism, virtual functions and static typing had been revealed to me. (I admit that it took a bit longer before pure virtuals, templates and the STL made any sense).

After this, the Win32 API was next – the concept of the message pump, window handles, the Win32 threading system, socket programming and, ultimately, COM and its associated technologies. I wasn't even sure what I wanted to do with any of these except understand how they worked. And I can say with complete confidence that that is exactly what I did. To this day, I can still remember the definition of the IUnknown and IDispatch interfaces, or explain what IConnectionPoint was for, even if that makes me one of the few people outside of Redmond who bother to remember.

After I finally did some assembly programming (for an alpha-blending function using those new-fangled MMX instructions), my quest to learn the geekiest stuff available to me was complete. I realised that sometimes knowing stuff isn't as important as what you can achieve with what you do know, and diverted my attentions toward more practical technologies. Linux and the internet were changing what it meant to be a developer, and the new age of web applications meant learning SQL and web scripting languages such as PHP and Python. Nowadays, it's not uncommon for me to be working on projects that have frameworks on top of these scripting languages. In many ways, I'm working at a much higher 'altitude', a long way from the C code which is at the heart of the OS and web server that my applications sit on top of. But merely knowing that these things are there, and understanding how they are built, and how the minds of the people who built them work, provides an incredible advantage in developing higher-level applications.

For new programmers starting out now, the experience must be greatly different to mine. I could sit down at a BBC Micro and input a set of commands which would produce an immediate output. I'm not even sure how one would go about this using Windows XP or Vista – sure, both operating systems have scripting systems built in, and I recall that the Windows Scripting Host can do some command-line I/O, which is all that you really need to get started. But it is surely more complicated than commands like PRINT and INPUT and this complexity might be enough that my equivalent, some 15 years behind me, might not bother to learn how to take those first steps.

So, perhaps there is something that the BBC Micro can still teach us. In many ways, it would still be a good platform for learning to program on. If I wanted to teach kids programming, I can think of worse systems to use.

Affiliates and crowd marketing

Rob Knight — Sat, 22 Mar 2008 10:12:31 -0400

In this post I hope to explore an idea which has formed a major part of the sales strategy for some of the internet's most successful retailers: affiliate marketing.

Affiliate marketing is the practice of allowing and encouraging third parties to participate in marketing a retailer's products in return for a share of the profit generated by the sale. A frequently-seen example of this would be Amazon's system, which allows third parties – often bloggers – to embed a specially-crafted link to a book on amazon.com. If someone follows the link and buys the book, the third party receives a small commission.

For bloggers, this often dovetails well with the blogger's own desire to promote or recommend a book that they have personally enjoyed, and the recommendation carries more weight because of this. Obviously, the precise rate of commission offered will be an important calculation for the retailer, since they will not want to erode their profit margins. But the commission need not be extravagant; to some extent, it is simply the principle of a reward that matters.

Such incentives can, however, motivate more than individual bloggers. Often, charities, voluntary organisations and even political parties can make use of these affiliate deals, promoting products to their members or customers on the basis that a purchase made through such a deal will benefit the organisation. That it benefits the retailer too is not considered, but it most certainly is to the retailer's benefit! The retailer gains substantially from these sales, made without any costs other than the commission paid to the affiliate.

These days, such affiliate programmes are used extensively by websites that exist to promote or recommend products based on reviews, feedback or some other mechanism. Sites such as BargainSpy recommend products which have been reduced in price or are otherwise available on special offer, taking a commission from those retailers that make affiliate schemes available. There is now a whole infrastructure of product recommendation and marketing, driving sales in large numbers, none of which is operated by the retailers themselves.

This can be seen as the beginning of a new model of doing business, in which the marketplace is mediated by the 'crowd', an amorphous collection of bloggers, recommendation site operators and other as-yet-unimagined affiliates. Retailers can gain substantial marketing clout from the crowd if they can create the right economic incentives. Typically, this means paying a commission. The question for retailers is whether the percentage cut in their profit margin on a given product is a more cost-effective use of their money than more traditional advertising. Amazon provides a compelling example of where that undoubtedly is the case, and countless other major internet retailers have followed their example.

Smaller retailers may have found it difficult to pursue this avenue. There is undoubtedly some administrative overhead in managing an affiliate system, and the software must be robust enough to ensure that the system operates reliably, only rewarding those who genuine drive business to the site and, equally, ensuring that all purchases which are made at the instigation of an affiliate are rewarded promptly.
But these are largely questions of software and infrastructure. As time progresses, e-commerce software must incorporate the management of affiliate schemes as a standard feature. The economic imperatives are clear – a properly-structured affiliate system can grow a business and improve profits. And in a time of economic uncertainty, businesses will need to rely on such innovations to maintain growth.

Semantic web: is it the big thing yet?

Rob Knight — Wed, 12 Mar 2008 19:17:24 -0400

The 'semantic web' is not a new idea. In fact, the idea has been around almost as long as the web itself, and its relevance is based on the very simple notion that much of the information that we make publicly accessible via the web is not yet organised according to any particular scheme. Resultingly, searching it can be very difficult, hence the great rewards that flow to companies like Google; their search engine does the best job of mapping the chaos, turning a vast jumble of information into something that is usable to ordinary humans.

Traditionally, search engines work by looking for the frequency and location of words within a document. When you search for 'chicken pasta recipe', the search engine attempts to find pages which contain all of those words. The more often these words occur, the better. If these words appear in the page title, or within page header tags, or in links pointing to the page (the more of these the better) then the ranking of the page improves. Google's great innovation was counting the incoming links from other sites, but this probably stands as the last truly great innovation in search.

Until now. Yahoo! is set to make a major step towards semantic searching in an effort to recover market share against Google (as an aside: am I the only one who has noticed the flurry of activity from Yahoo! since Microsoft began their takeover attempt?). For the first time, a major search company is openly talking about making use of the growing amount of semantic data now available on the web. It has been a long wait, but the age of the semantic web is finally here.

What is the semantic web anyway?

Here it is necessary to briefly recap just what the semantic web is. Semantic data is, essentially, data with meaning. In the semantic web, 'Rob Knight' isn't just a combination of letters, 'Rob Knight' is a person and the search engine knows it. The search engine may have some contact details, social networking profiles or other pieces of relevant data to display to someone searching for me. This is only possible if I - either directly or via third-party sites that I am a member of (e.g. LinkedIn) - make this data available in a format that explains what the data means. So, instead of a jumble of informated scattered all over a page (or several pages), data is made available in machine-readable formats. These are much simpler than HTML pages and are therefore comprehensible to computer systems like search engines. Many of the things that we describe in ordinary language in HTML pages can be re-described in some new semantic format. Microformats provide an easy stepping-stone for converting existing web page data into a semantic format.

Perhaps the biggest application will be in product search. Retailers will be able to provide far more detailed information about their products, information that means something to the computer systems responsible for searching and categorising this information. Consider a clothing retailer; each product can be described not just by name, but by colour, size, price, availability and more. When searching for 'blue jeans', the search engine is no longer looking for the words 'blue' and 'jeans' somewhere on a web page; it can recognise that 'jeans' are a type of product and that 'blue' is an attribute of the product, and can then list all blue jeans that it knows about. And if the user wants to sort by price, they can. And since the search engine will also have semantic information about the retailer - their geographical location, for example - the search engine may be able to find the nearest retailer with the lowest shipping costs.

The collection of data about a 'thing' (be it a person, company, product, place or any identifiable object) into discrete, searchable data files will also make it easier to detect connections between things. For example, a review of a product could link directly to the definition of that product on a manufacturer's website, rather than linking to a simple HTML page. This enables the search engine to have considerable confidence that the review is about the product rather than, say, about the manufacturer, or the manufacturer's website. This will enable much better dialogue between consumers, retailers and manufacturers, in a way which will, in the long run, produce changes in the way we do business that are at least as big as the changes made in the last decade.

The immediate impact

Having observed the technology industry for some time, I'm not convinced, despite the advantages of the semantic web, that there is an instant payoff. We're only at the beginning of the change. And with the global economy looking shaky, investors and companies would be right to be wary of promises of technological salvation. The real work will happen over the next few years, just as the real work of building web 2.0 happened in the shadows of the dot-com crash. Those who innovate sensibly, delivering lower costs and better return on investment will find themselves well-placed for the upswing, just as those lonely pioneers who carried on inventing after the bubble burst found that they had struck gold once more as the web 2.0 trend appeared.

I will be following this up with a series of posts about semantic web technologies, going into more detail about the specific technologies and their implementations.

Move along people, nothing to see here

Rob Knight — Sun, 06 Jan 2008 11:55:52 -0500

At some unspecified future date, this site will have actual content on it. Until then, this is all that you get.