There’s a reporting bias in the press, too, of course: Man Bites Dog syndrome. If the project works and saves money, it won’t make headlines. For example, the Ontario Telemedicine Network connects physicians, nurses and patients online every single day, with high uptime rates. Every patient that doesn’t have to travel to Toronto from Sudbury saves the government — and the patient — mucho dinero.

There’s another comparison I rarely see, as well: how many projects of any kind are successful? If we look at the recent decision by the Canadian government to dole out ‘stimulus’ money for infrastructure projects (which invariably mean building new roads, for some reason), we see failure rates which are comparable to those in IT. For the 2 year, $4 billion fund, only 25% of projects were finished before the March 2011 deadline, with the most likely scenario being that some 900 projects will not meet the deadline. Manitoba is apparently hoping for favorable weather to meet the target.

Perhaps the difference is that when it is something physical, like a road, it is much harder to leave it half-finished then a piece of software.

I challenged his conclusions on the basis of poor information. I said:

@lemire too bad @timbray’s post is completely lacking in empirical data (consultants  don’t count).

To which Tim replied:

@alex77 Follow some of the links from my piece. Tons of data. Also, get any COO drunk & you’ll hear.

And I finished with:

@timbray anecdote is not the same as peer-reviewed data, sadly. Software field is light-years from even being able to have a “ClimateGate”

So what did I mean? And what did he mean?

Hacker science

There are a lot of Tims in the industry (and this is a good thing!). For example, his Wide Finder project is a classic example of ‘tinkering’ leading to innovation. I think the best way to characterize his work is engineering science. He starts with an itch, and tries to figure out the best way to scratch it. Along the way, we get new understanding of the problems (e.g., how can non-experts use Clojure effectively) and some data on what works and what doesn’t (in the ‘real world’). This is very similar to the development of the airfoil, and this stream of science is well described by Vincenti in “What Engineers Know and How They Know It”.

But this isn’t enough. Vincenti makes clear that further airfoil improvements come from a good understanding of the physics underlying lift and computer modeling, not iterative tinkering (although there are clear parallels). Similarly, while the Wide Finder project is a great way to start understanding the theories and constructs involved in concurrent programming in large-scale problems, I’m not convinced this is sufficient. What we would need is studies of many programmers working on many different problems. For example, how would the C solution work with non-expert C programmers? Do the concepts of Agents in Erlang make intuitive sense to most people?

Standard of evidence

These questions cannot be answered by individual tinkering. They require larger scale studies and more thorough investigation. But unfortunately, there is a trend in the software blogosphere to use one or two data points as solid evidence that “C sucks” or “Perl is unreadable”.

But why would we accept this standard of evidence for programming, but not for, say,  climate change? In the same way, can we really base conclusions about the state of IT on the musings of some influential bloggers, industry whitepapers, and self-serving consultant reports? Let’s imagine the same situation prevailed in climate science. Then we would have a few people commenting on the basis of anecdotal encounters (“boy, it sure is cold in Toronto this winter!”), Shell Oil writing reports on the oilsands (“only 100 hectares are polluted!”) and a climate mitigation consultant encouraging mitigation strategies. In none of these cases do you have enough information for rational action (not that this is always the goal!).

Maybe you would say these are apples and oranges, that climate science is a much bigger issue, etc. etc. I’m not sure. In both cases we are talking about billions of dollars of costs. The only real difference is that policy action will be taken by collective action (climate change) or corporate action (IT).

Improve the Data

So am I saying that Tim Bray is wrong, that corporate IT is fine, that the lessons of Twitter don’t apply? Not really. What I am saying is that you should base these sorts of conclusions on empirical data that is collected in the best traditions of open science. It would be data that is from a survey whose questions we can see. It would be data from peer-reviewed research journals.

Let’s stop making conclusions using data from Gartner, Standish, et al. These firms are in the business of selling advice, and they are not interested in objective truth. A lot of the reports are based on closed source surveys, talks with “Thought Leaders”, random observations, etc. If you are Standish, are you really interested in a report that says “Most IT projects successful”?

What is needed is a non-partisan study, much like the IPCC, that will examine the relevant scientific research on the issue. Before we can draw conclusions, especially black and white conclusions, we need to know what we don’t know (“unknown unknowns”!). This means raw data, and this means more openness on how well these IT projects are really doing. It would mean allowing researchers access to IT development teams, to perform proper case studies, to see, for example, why the FBI system failed. Too often software researchers are in the position of begging companies to release data. And even in industries where publicly accessible studies are mandated, we find many games being played to prevent negative outcomes being published.

So will there be improvements in empirical data on software? I have my doubts. But it’s necessary if we really want to know what software can do, and what software cannot do.

However, a recent paper,  A Replicated Survey of IT Software Project Failures, suggests that if we want software projects to succeed in IT, than these aspects are perhaps the best place to focus our attention.The paper is a refreshing improvement on the Standish reports.

Of the top five reasons a project was cancelled, three were management-related: Senior management not sufficiently involved (33%), too many requirements and scope changes (33%), and lack of necessary management skills (28%). The other two were project over budget (which presumably has many possible causes), and a lack of necessary technical skills (22%).

As software engineering researchers, I would argue that a lot of our work is focused on the last reason (technical skills) or less important causes (e.g. technology problems). There is a large research community dedicated to requirements research, but arguably this community is seen as less important (compare, for example, papers in ICSE 2008: of  103 papers, only 6 were requirements related).

These results also suggest what I’ve suspected: the choice of technology (C#, Java, SQLServer, AJAX, etc) is less important than getting a good team together, with properly scoped requirements and a sound leadership vision. As Karl Pilkington would say, the rest is just ‘pfaffing about’.

Thanks to Jorge and Lorin Hochstein for the pointer.